Demonstrably Safe AI For Autonomous Driving
Autonomous driving is the ultimate challenge for AI in the physical world. At Waymo, we’re solving it by prioritizing demonstrably safe AI, where safety is central to how we engineer our models and AI ecosystem from the ground up. As a result, we’ve built an incredibly advanced AI system safely operating in the physical world at scale. With well over 100 million fully autonomous miles driven, we are making streets safer where we operate — achieving a more than ten-fold reduction in crashes with serious injuries compared to human drivers.
Now, we invite you inside the engine room. This post offers a detailed look at Waymo’s AI strategy and how it’s fueling our momentum, allowing us to safely bring our service to more riders, faster than ever before. We will unpack our holistic AI approach, centered around the Waymo Foundation Model, which powers a unified demonstrably safe AI ecosystem that, in turn, drives accelerated, continuous learning and improvement.
Waymo’s Holistic Approach to AI
Unlike other AI applications that may optimize for capability first and layer on safety later, in autonomous driving, safety cannot be an afterthought. At Waymo, it’s the non-negotiable foundation upon which we build our AI ecosystem.
Achieving demonstrably safe AI — where safety is proven, not just promised — requires a holistic approach. Beyond a smart and capable Driver, you also need a closed-loop, realistic Simulator to train and rigorously test the Driver in a myriad of challenging situations, and a sharp Critic to evaluate the Driver's performance and identify areas for improvement.
The power is in unity. Developed jointly and with safety at their core, our Driver, Simulator, and Critic are all fueled by the same underlying AI — the Waymo Foundation Model — creating a continuous virtuous cycle.
Waymo Foundation Model: Cornerstone of Waymo AI
The Waymo Foundation Model is a versatile, state-of-the-art world model powering our AI ecosystem. Its innovative architecture provides significant benefits over the pure end-to-end or modular approaches.
In particular, the model leverages the full expressibility of learned embeddings as a rich interface between model components and supports full end-to-end signal backpropagation during training. At the same time, its additional compact, materialized structured representations like objects, semantic attributes, and roadgraph elements allow for:
Powerful correctness and safety validation at inference time in the Driver
Highly efficient, physically-correct and realistic closed-loop Simulation at extremely large scale
Strong verifiable feedback signals for evaluation by the Critic and reinforcement learning during training
The Waymo Foundation Model employs a Think Fast and Think Slow (also known as System 1 and System 2) architecture with two distinct model components:
Sensor Fusion Encoder for rapid reactions. This perceptual component of the foundation model fuses camera, lidar, and radar inputs over time, producing objects, semantics, and rich embeddings for downstream tasks. These inputs help our system make fast and safe driving decisions.
Driving VLM for complex semantic reasoning. This component of our foundation model uses rich camera data and is fine-tuned on Waymo’s driving data and tasks. Trained using Gemini, it leverages Gemini’s extensive world knowledge to better understand rare, novel, and complex semantic scenarios on the road. For instance, in an extremely rare scenario where there’s a vehicle on fire on the road ahead, while the physical space and drivable lanes might be clear for passage, the VLM can contribute a semantic signal prompting the Waymo Driver to take a different route or turn around.
Both encoders feed into Waymo’s World Decoder, which uses these inputs to predict other road users behaviors, produce high-definition maps, generate trajectories for the vehicle, and signals for trajectory validation.
Waymo’s AI Ecosystem: Distilling Knowledge from Teacher to Student Models
Informed by our holistic approach, the Waymo Foundation Model powers the Driver, Simulator, and Critic. We achieve this by first adapting it to each of these three tasks, resulting in large, high-quality Teacher models that excel in their specific roles. However, these Teacher models are too big to run on vehicles for real-time decision making or in the cloud to simulate and evaluate hundreds of millions of miles, so we safely distill them into smaller Student models. Distillation is key, as it allows us to retain the superior performance of large models within their more compact and efficient versions. As a result (and mirroring similar trends in other areas of AI), by first training powerful high-capacity Teacher models and then leveraging efficient distillation techniques, we are able to achieve much better scaling laws for the resulting students.
Driver. Our Teacher Driver models are trained to generate safe, comfortable, and compliant action sequences. Through distillation we transfer their rich world understanding and reasoning capabilities to more efficient Student models, optimized for real-time onboard deployment. To maximize the benefits of distillation, our onboard architecture is designed to mirror the Waymo Foundation Model structure. Importantly, the Waymo Driver employs a separate and rigorous onboard validation layer, which then verifies the trajectories produced by the Driver’s generative ML model.
Simulation is an essential tool for closed-loop training and testing of our Driver across a range of diverse and challenging scenarios, including potential collisions, inclement weather, intricate intersections, and unusual behaviors on the road. The Simulator Teacher models are capable of creating high fidelity, multi-modal dynamic worlds to evaluate our Driver. The student models are compute-efficient versions of these larger models that are designed to run the massive scale of simulations that are needed for the robust evaluation of the Driver. The Waymo Foundation Model’s architecture allows us to seamlessly combine compact materialized world-state representations and sensor simulation, unlocking large-scale, hyper-realistic and physically correct, yet computationally efficient virtual environments.
By using text-based prompts for global scene elements, such as weather conditions and time of day, along with semantic conditioning for the dynamic elements in the scene, such as other road users and traffic lights, we can transform real-world scenes (on the left) into highly realistic simulations (camera simulation in the middle, lidar simulation on the right). Notably, in this example, the sensor data is purely synthetic and is produced by our generative sensor-simulation models from the underlying compact structured world representation.
Critic. Our world-class evaluation system is designed to stress-test the Waymo Driver, proactively identify subtle edge cases, and enable rapid, targeted improvements. The Critic Teacher models can analyze driving behavior and generate high-quality signals, used for training Student models and for automatically building rich evaluation datasets. Then the Critic Student models analyze driving logs, identify interesting or problematic scenarios, and provide nuanced feedback on driving quality.
Powered by the Waymo Foundation Model, all of these components comprise a seamless AI ecosystem and create a flywheel for ongoing learning and improvement.
Creating Flywheels for Continuous Improvement
A great Driver is not static — it’s the product of continuous learning and refinement. There are several mechanisms that inform the Waymo Driver’s evolution. Our inner learning loop, powered by the Simulator and Critic, utilizes Reinforcement Learning to train the Driver. Within this safe and controlled simulated environment, it gains experience, receiving rewards or penalties based on its actions, enabling massive-scale learning.
Our outer learning loop, informed by Waymo’s real-world driving, creates an even more powerful learning flywheel. The cycle begins with our Critic automatically flagging any suboptimal driving behavior from our vast fully autonomous experience. Next, we generate improved, alternative behaviors from these events to serve as training data for the Driver. These improvements are rigorously tested in our Simulator, with the Critic verifying the fixes. Finally, once our safety framework confirms the absence of unreasonable risk — and only then — the enhanced Driver is deployed to the real world.
This flywheel is enabled by the unprecedented amount of fully autonomous data we’ve accumulated over the years and are continuing to accumulate at an exponentially increasing rate. Historically, we relied heavily on high-quality manual driving data to train and refine the Waymo Driver. Today, our fully autonomous mileage far exceeds manual data. There is simply no substitute for this volume of real-world fully autonomous experience — no amount of simulation, manually driven data collection, or operations with a test driver can replicate the spectrum of situations and reactions the Waymo Driver encounters when it’s fully in charge. Integrating this rich, real-world fully autonomous data directly into our unique flywheel enables the Waymo Driver to learn from its own vast experience and continuously improve.
By embracing this holistic approach to AI and building learning flywheels, we are not just advancing the Waymo Driver, but also setting the standard for safe autonomous driving at scale. We're continually innovating and pushing the boundaries of what's possible, and a lot of exciting work in AI is still ahead.