Sharing our safety framework for fully autonomous operations

On October 8th, Waymo opened its fully autonomous ride-hailing service to the general public in Phoenix. Right now members of the public are hailing vehicles with no human driver controlling the car – either in the vehicle or remotely – to help them get to where they’re going as part of their everyday lives.

This milestone is not just the output of many years’ experience in developing the Waymo Driver - it’s also the result of over a decade developing our safety practices. Since we started in 2009 as the Google Self-Driving Car Project, we’ve spent years refining a comprehensive set of robust methodologies that assess safety across our technology, operations, and team culture, and ultimately guide the deployment and safe operations of the Waymo Driver.

We take our responsibilities to the communities we serve seriously. Now that we’ve opened up our fully autonomous ride-hailing service to the public, we’re also sharing in detail the safety framework that guides our progress.

We want to give everyone who rides with the Waymo Driver a deeper understanding of the incredibly high standards to which we hold ourselves in the safety practices that govern our deployments. Today, we’re publishing two papers that explain the processes we use to drive fully autonomously on public roads and validate the safety of our operations.

The first, Waymo’s Safety Methodologies and Safety Readiness Determinations, includes details of our Safety Framework – the careful and multilayered approach to safety that has helped make it possible to deploy fully autonomous vehicles on public roads.

The second, Waymo’s Public Road Safety Performance Data, analyzes the miles we’ve driven on public roads in Arizona - one of 10 states we’ve driven in the U.S. since our founding - to provide data about our safe operations in practice.

This is the first time an autonomous technology company has released a framework describing the safety of its fully autonomous commercial operations. We believe this transparency and accountability is important for demonstrating the trustworthiness of our operations, and critical to deepen the dialogue around autonomous driving safety. We look forward to ongoing engagement with experts and academia to help ensure our work can continue to evolve and grow.

Waymo’s Safety Framework: Safety Methodologies and Safety Readiness Determinations

There is currently no universally accepted approach for evaluating the safety of autonomous vehicles – despite the efforts of policymakers, researchers and companies building fully autonomous technologies. Since we began our work in 2009, we have worked to develop a robust Safety Framework – with multiple complementary methodologies – that embeds safety in all aspects of our technological and product development. This framework draws on insights from the research and automotive communities, and is based on our extensive experience in building autonomous systems.

By publishing our framework, we aim to transparently share details about our approach; show how we evaluate our safety performance; demonstrate how we understand and measure our readiness for safe autonomous operations at scale; and invite perspectives from experts as we continue to expand our operations.

As our industry progresses, we encourage other autonomous driving companies to develop and publish guiding frameworks and principles that can demonstrate how autonomous systems are safely and responsibly deployed.

Our Safety Framework, available here, maps out our approach to safety in three fundamental layers: 

1. The Hardware Layer

We call our automated driving system (ADS) the Waymo Driver. A vehicle equipped with the Waymo Driver has four main subsystems, which form the ‘hardware layer’. This includes the vehicle itself; the systems used for steering and driving; the sensor suite built into the vehicle; and the computational platform used to run our software. 

Our Safety Framework explains:

• How we verify and validate the performance of these components.

• How we select our base vehicles with safety performance in mind. 

• Our use and testing of backup steering and control actuators. 

• How we specify and measure the performance of vehicle sensors. 

• The backup computers and power sources within our computational platform. 

In combination, these steps have led to the development of a robust hardware stack that has been designed around the goal of ensuring safety in any scenario. You can read about the latest generation of our hardware here. 

 2. The ADS Behavioral Layer

There are three primary capabilities upon which we evaluate the performance of the Waymo Driver’s behavioral layer:

•  Avoiding incidents with other vehicles or users of the roads.

• Successfully completing trips in fully autonomous mode. 

• Adhering to applicable driving rules where the vehicle is operating. 

We use a series of methodologies to evaluate these capabilities, including:

• Hazard analysis techniques to design for the robustness of the Waymo Driver from the beginning of the development process.

• Scenario-based testing, both in simulation and closed-course test environments, to verify the behavior and performance of the Waymo Driver. 

• Simulated deployments to evaluate the aggregate performance of the Waymo Driver over a large number of miles and a wide variety of situations. 

3. The Operations Layer

The first two parts of our Safety Framework lead to the development of safe, capable autonomous vehicles. The final part – operations – reflects the importance of the systems, approaches and culture we have been building around our technology to help ensure it is safe. Our Safety Framework sets out the inner workings of several important operations processes at Waymo:

• Fleet Operations includes a broad range of monitoring and support functions to help ensure the safe operation of the Waymo Driver and our vehicles. For example, while the Waymo Driver handles the entire driving task, our fleet response team can confirm what our ADS is seeing and provide additional information.

• Our Risk Management Program enables us to proactively identify and resolve potential safety issues that are triggered by ongoing technological change, categorizing and prioritizing potential risks so they can be mitigated. 

• Finally, our Field Safety Program helps collect, assess and resolve potential safety concerns that occur during real-world operations. We draw from many sources, including employees, our riders, and the public.

Our governance structures are important tools that help us monitor and review safety-related decisions within our company. Our Safety Framework details this work, which includes the role of the Waymo Safety Board. This group works together to help resolve safety issues, approve new safety activities, and ensure our overall approach to safety continues to evolve and improve. 

We use these methodologies to assess and continuously improve safety of the Waymo Driver. We also use them collectively to make our assessment about whether the Waymo Driver is ready to make the next step – as we did before opening fully autonomous operations for ride-hailing to the public in Phoenix earlier this month. 

We establish rigorous performance criteria for our Driver in a particular operating environment, and drawing on all our safety methodologies to determine whether the Driver is ready to be tested or deployed in new conditions without creating any unreasonable risks to safety.

Waymo’s Public Road Safety Performance Data

As we broaden the availability of Waymo services and vehicle platforms enabled by the Waymo Driver, we want to be sure that the communities in which we operate understand the safety approach we take so they can be confident in the safety of our products.

In addition to the Safety Framework, today we are sharing a white paper on Waymo’s Public Road Safety Performance Data. There is not yet a globally or nationally recognized framework to assess the relative safety of autonomous vehicles – so we are taking this opportunity to share safety-relevant data with our peers, our stakeholders, and our communities.

The paper includes results from 6.1 million miles of automated driving with a trained vehicle operator in Arizona in 2019. It also includes an additional 65,000 miles of fully autonomous operations – operating at Level 4 autonomy, with no human driver behind the wheel and no remote operators – from 2019 and the first nine months of 2020.

This data represents over 500 years of driving for the average licensed U.S. driver – a valuable amount of driving on public roads that provides a unique glimpse into the real-world performance of Waymo’s autonomous vehicles. The data covers two types of events:

1. Every event in which a Waymo vehicle experienced any form of collison or contact while operating on public roads

2. Every instance in which a Waymo vehicle operator disengaged automated driving and took control of the vehicle, where it was determined in simulation that contact would have occurred had they not done this

The paper also includes analyses of the events by type, and provides details of the most potentially significant events, including diagrams. This data is unlike almost all driving data available to the world, as it includes every collision or contact, even those with no damage that would not be reported to insurance companies or local police departments. The majority of incidents in our report fall into this category of typically unreported minor collision or contact.

In addition to sharing data, the paper describes the simulation processes Waymo uses to understand how events would likely have played out in situations with on-board vehicle operators where they chose to disengage the Waymo Driver. These simulations use models of human behavior, and help us better understand how our vehicles and other road users interact.

The paper is available at www.waymo.com/safety. Like our Safety Framework, it is intended to inform our riders, our stakeholders, our peers, and the communities in which we drive about the safety of the Waymo Driver and our progress.

We hope that our transparency will lead to greater openness within our industry and a more meaningful conversation on autonomous driving performance and safety. Ultimately, we believe this will build trust and understanding in responsibly developed autonomous technology as they are deployed in fully autonomous operations to move people and goods safely and efficiently.