The Waymo Driver’s training regimen: How structured testing prepares our self-driving technology for the real world

Imagine a tiny city where you control everything that happens on the streets. You manage how many cars zip down the roads and how fast they are going. You dictate how many cyclists are on a roundabout or whether they follow the road rules. The “weather” around the vehicle can change multiple times a day from blue skies and sunshine one minute to heavy rain showers the next, but only if you want it that way. One may say such a city doesn’t exist, but if you drive out to the middle of Merced County in California, you’ll find it at Castle, a former Air Force Base our team uses to help build the World’s Most Experienced Driver™. 

Spanning 113 acres, Castle is a state-of-the-art, closed-course testing facility — which has been specially designed and redesigned over the years for Waymo’s unique testing needs to prepare the Waymo Driver to handle challenges on the road, evaluate new software before it’s released to our self-driving fleet, and validate our system’s performance. While our closed-course testing at Castle has always been a critical component of our development process, over these past few months of shelter-in-place, Castle has produced even more benefits. It has helped test and scale the new policies we’ve put in place to keep our team and entire community safe, while unlocking some key technology advancements for our teams working at home. Now that our vehicles are back on the road, we can accelerate the Waymo Driver’s deployment by enabling the entire development cycle -- from gathering real-world data and analyzing it for interesting encounters, to rigorously testing updates we make to the system and evaluating its performance.

Mastering the fundamentals to handle the one in a million

Our testing site within the former base is set up like an adaptable city, including everything from wide avenues and suburban driveways to a railroad crossing and roundabouts, similar to where our vehicles operate in the real world. At Castle, we stage complex and rare scenarios — such as a person walking out of a porta-potty onto a street or a pile of garbage falling out of the truck in front of us — in a safe, controlled environment and test them over and over again, changing different variables, to ensure that our self-driving technology can handle the wide range of situations it might come across.

Over the years, we’ve been able to amass a library of over 40,000 structured testing scenarios — not counting all the variations we create with each situation. These scenarios include things that we have never seen on public roads but could imagine happening and others that occur once every hundreds of millions miles. Additionally, we create some relatively mundane tests that can also be quite challenging for self-driving vehicles and humans alike. For example, driving behind a large garbage truck on a narrow street — that stops every hundred feet or so and leaves empty trash cans strewn about — may create a lot of complexity on the road.

Our teams develop these scenarios in a variety of ways. We use national databases to identify the most common traffic accident scenarios, recreate situations our trained drivers see while testing on public roads, create scenes specifically to test new driving functions developed by our engineers, and field suggestions from team members based on their driving experience. By exposing the Waymo Driver to a wide variety of scenarios and teaching our self-driving technology fundamental skills rather than only to handle individual situations, our vehicles become more equipped to handle any situation they encounter on the road, even if it hasn’t seen that specific scenario before.

How structured testing and simulation complement each other

Testing is not a one and done task, but rather a never ending feedback loop that includes structured testing, simulation, and public road operations. When we develop a scenario to test new software, we can use either one of these tools or a combination of them. For example, after executing a test on our private track, we can then create and run hundreds of variations of that scenario in simulation. Our simulation technology allows us to do this in a matter of seconds right at our developers’ desks as they work on new features for the Waymo Driver.

As much as simulation can help scale the value of structured testing, structured testing complements our simulation. While we maintain a high level of realism in simulation, structured testing allows us to evaluate both our software and hardware stack. By recreating scenarios our vehicles have already successfully completed in simulation, we can continuously verify our simulator and assess small nuances that may affect vehicle behavior. Adding this extra layer of testing and redundancy into our development cycles helps us to ensure we have done our due diligence before our vehicles drive on public roads.

Advancing our next-generation Driver

Testing Waymo’s fifth-generation hardware suite on the electric Jaguar I-PACE at Castle

Not only does structured testing help verify software updates, it also allows us to test and validate our hardware. With closed-course testing, we can evaluate our integrated system’s performance across our vehicle and sensing systems.

Over the past decade, we’ve changed the number of vehicle platforms and sensor suites we test and operate on five times — from Lexus 450 SUVs to our custom-built Firefly, from Chrysler Pacifica minivans to 18-wheeler trucks, and now to all-electric Jaguar I-PACES. All these platforms have different body builds and require different modeling for sensors, perception, motion control, and planning. Structured testing allows us to test various system capabilities, such as how our sensing and computer vision systems work together to detect and identify a speeding motorcycle at long distances. With structured testing, we can also measure how well our motion control system follows the intended path, and validate our system’s performance and reliability across more diverse operating domains through weather testing in rain tunnels and thermal chambers, across speed bumps and potholes.

Like the platforms that came before it, our custom-built 5th generation sensor suite has been rigorously tested at Castle to help ensure its safety and readiness for public roads. With each generation, we have matured our models and our methods, so that each new bring up is smoother and quicker, enabling us to get on the roads to serve our riders and Waymo Via partners sooner. And, if you are in the Bay Area, Phoenix, or Detroit area, you should see more of our next-generation Waymo Driver driving around.

As we continue to build the hardware and software to power the Waymo Driver, we’re looking for people to join our growing team. At Waymo, our teams motivate and inspire one another, see their research implemented in tangible ways, and together make real steps toward a positive impact on the world of mobility. Whether you’re an engineer, researcher, or a curious and critical thinker driven to make the roads safer for everyone, we’re looking for people to help us tackle real-world problems. Learn more at waymo.com/joinus.