Simulation City: Introducing Waymo's most advanced simulation system yet for autonomous driving

At Waymo, we use simulation to advance the Waymo Driver's abilities in several ways. We can amplify our number of real-world miles to accelerate our Driver’s learning, prepare for rare events, train new models, validate new software, and even evaluate how the Waymo Driver would have performed in actual fatal crashes that have already occurred. For simulation to be valuable as a learning tool, it has to closely match our target domain – performing rider-only trips for Waymo One and goods delivery trips for Waymo Via in the real world– in which case, there is only one question that matters. Did the Waymo Driver safely and efficiently complete the trip?

To answer this question, Waymo created a software solution internally dubbed SimulationCity to add to our industry-leading evaluation tools. SimulationCity automatically synthesizes entire journeys to assess the Waymo Driver's performance – whether it's safely completing a 20-minute Waymo One rider-only trip across San Francisco or efficiently running an 11-hour goods-only Waymo Via trip from Phoenix to Dallas. The capability is enabled by the more than 20 million autonomous miles the Waymo Driver has collected on public roads, third-party data such as NHTSA Crash Data Systems and Naturalistic Driving Study Data, Waymo's unique access to Alphabet's state-of-art technical infrastructure, and the culmination of advancements in sensor simulation and machine learning techniques in areas such as intelligent agents, object trajectory generation, and high-quality automated data labeling. With the power of SimulationCity, we can significantly accelerate the development and deployment of the Waymo Driver – an “everything engine” that allows us to safely bring up new vehicle platforms, test them in new geographies and refine our operations so we can bring our technology to more people in more places.

Let's back up and see how it has all come together.

Creating realistic simulation

For simulation to be an effective tool, it has to closely represent the environment it's emulating to provide accurate insights into a system’s performance. What happens in simulation must be predictive of what happens in the real world to ensure you are simulating the right things. This requires minimizing the differences between the simulated and real world, from the way your sensors see to the way your agents react to changes in their simulated environments — and are statistically representative of the real world.

One way we ensure statistical realism in our simulated world is by creating realistic conditions for our autonomous driving technology to experience. So, if the Waymo Driver is going through spring showers in Detroit at sunset in SimulationCity, we can recreate raindrops on our sensors and even simulate other minute details such as the dimming light and solar glare.

Just like it's not sufficient to only test in ideal conditions, it is not enough to test how autonomous driving technology reacts to just well-behaved or overly aggressive road users. We want to understand how the Waymo Driver reacts to the full distribution of behaviors it will encounter in the real world.

Suppose we simulate a tailgating scenario at an intersection. We want to understand as many various outcomes as possible and their likelihood to happen to evaluate the Waymo Driver's behavior. If we picked a random tailgating scenario, chances are the tailgater would brake in time. But it's important that we also assess how the Waymo Driver behaves if the tailgater doesn't brake in time, such as when that driver is distracted or inattentive. As we simulate more and more variations of the same scenario, we begin seeing a convergence of the distribution of outcomes between what we observe in simulation and the real world. SimulationCity also enables us to explore rare events, so as to create risky scenarios the Driver has never encountered before, but are still proven to be realistic and very useful.

To help ensure a simulated environment is representative of the real world, you need to have a lot of high quality data from the real world to base it off of and compare it to. Our simulated environments are constantly informed and refreshed with the experiences our fleet collects every day in the dozens of cities we drive in, letting us incorporate minor subtleties into our simulation regularly.

Lastly, creating high fidelity simulation for Class 8 trucks requires additional nuances, such as the variable payloads in the trucks' free-moving trailers, which all benefit from additional simulation. That's why, in addition to scripting the situations our Driver encounters in the real world, we also model various payloads we may haul. This testing is platform-agnostic – many of the functions and features we've designed for our passenger vehicle simulation transfer over to the simulations that operate our Class 8 trucks, and vice versa.

As our capabilities grow, so do our evaluation requirements

Common use of simulation involves short targeted-scenario segments of real driving data to evaluate isolated events, like whether the autonomous driving technology successfully made an unprotected left turn. While our short-form simulation allows us to optimize and refine isolated skills, such as nudging around a double-parked vehicle, it is important to evaluate how an autonomously driven system’s capabilities work together over full-length trips. As our simulation becomes more sophisticated, we can assess harder behaviors and aggregate important trip-level statistics about the Waymo Driver's performance, enabling us to advance and accelerate our technology in many ways, including:

Safely and efficiently bringing up new vehicles and hardware Last year, we introduced our fifth-generation hardware suite which will first operate our new fleet of all-electric Jaguar I-PACEs. Because SimulationCity allows us to take what we've learned from previous generations and apply it to our latest system, progress on the Waymo Driver is cumulative rather than duplicative.

Improving the Waymo Driver’s operational expensesSafely getting people and things where they are going is our priority, but doing so efficiently and comfortably is essential, too. With SimulationCity, we can continue refining the Waymo Driver by optimizing its performance on trip-level statistics, like how long it takes to complete a trip, how much energy is used, and other operating considerations that matter when you're running a service.

Enhancing the Waymo One rider experienceOur years helping Waymo One riders get where they are going safely has also provided us a deep understanding of what makes a successful pick-up and drop-off. Using our years of experience, and feedback from riders, we can even simulate pick-up and drop-offs, including how long it would take a Waymo One rider to walk from their current location to meet their vehicle and evaluate whether it would earn a good rating.

Bringing our technology to more people in more placesSimulationCity is especially beneficial as we expand our operating domains. Even if we haven't autonomously driven in a specific location, we can upload a map, insert a Waymo vehicle into the scene, and combine the millions of miles we have driven with other data to inform what other roads users could behave like in areas we currently don't have behavioral data on. This process enables us to refine our overall driving abilities and rider experience in a new area before having a single rider in a vehicle. It also means that we can put a lot more vehicles "on the road" – with no constraints on when, where or the number of vehicles we can simulate, and how quickly we can simulate them, we can rapidly speed up our learning and development.