How we ensure the Waymo Driver operates safely in San Francisco

Last year, Waymo expanded its testing in San Francisco and began welcoming new riders in the city. As we continue to expand our operations providing rides to more San Franciscans, we’re sharing even more on how we evaluate the safety of our technology and operations in the City by the Bay.

Even against a backdrop of ongoing conversations among regulators, policymakers, and industry leaders, there is currently no universally accepted approach or metric for evaluating the safety of autonomous vehicles. Back in 2020, Waymo became the first autonomous technology company to publicly release its safety framework. We have encouraged others to do the same because we believe this transparency around safety is paramount for the future progress of autonomous driving. In this blog, we’ll share more details about how we use this approach in San Francisco.

Over the decade that we’ve been driving the streets of San Francisco, we’ve designed five generations of our autonomous driving system–the Waymo Driver–each even more advanced than the previous. We’ve also gained experience deploying each Waymo Driver across a range of vehicle platforms—from our first prototypes back in 2009 to the Jaguar I-PACEs that are now providing rides to hundreds of San Franciscans.

Along the way, Waymo became the first company to operate a fully autonomous, commercial ride-hailing service–Waymo One™. Before launching the service in Arizona, we used a comprehensive set of processes and methods to evaluate our Driver’s readiness. These methods provide us with the foundation of knowledge and experience to efficiently validate our performance with new generations of hardware, on new vehicle platforms, and in other locations.

We use a range of methods as part of this evaluation, including computer simulations with unprecedented accuracy, structured testing on closed courses, and then of course driving on real roads. Last year, we took all of these steps to safely introduce the fifth generation Waymo Driver on the Jaguar I-PACE vehicle platform in San Francisco. We did it with highly trained autonomous specialists behind the wheel, who could disengage the vehicle’s autonomous mode and take over driving when they felt appropriate.

After conducting our simulation and closed course testing, we drove over 2.7 million miles autonomously in the city to advance and validate our performance, as we brought up and integrated our new technology stack last year. These miles are critical to training the system, ensuring simulation realism, evaluating the Driver’s performance and continuously improving its safety. We evaluated the Waymo Driver’s software performance in increasingly challenging driving situations—day and night, in various weather conditions, from Masonic Avenue’s dense traffic to Valencia Street’s many pedestrians and road closures on a Saturday night. Finally, we began offering rides to members of the community: hundreds of people ride with Waymo One in San Francisco now, providing valuable feedback that helps us refine and advance our service, and we’re gradually extending our program to even more riders in 2022.

As we’re working towards removing the autonomous specialist from the vehicle in San Francisco, we evaluate each of these three components—hardware, software, and operations—consistent with the safety framework we first publicly unveiled in October 2020.

1) Hardware and vehicle platform

Last year, we added more vehicles to our fleet of hundreds of Jaguar I-PACEs in San Francisco. Before we deploy a vehicle in our testing fleet, we ensure robust integration of the Waymo Driver’s sensors and compute with the vehicle platform, validating how these systems work together in a wide range of real world conditions. We also rigorously evaluate the performance of our sensors and compute platform in the huge range of situations they can expect to encounter. In San Francisco, this means evaluating:

• the vehicle’s steering and speed control systems driving San Francisco’s narrow streets, turns, and hills,

• the capabilities of our compute platform that enable the Driver to react quickly in situations like when an emergency vehicle is approaching or someone runs a red light at a busy city intersection,

• the performance of our sensors in various weather conditions—including San Francisco’s notorious fog, powerful gusts of winds, and sands blowing off Ocean beach, and

• the robustness of our autonomous system as a whole, such as our Driver’s ability to come to a safe stop and the readiness of our backup systems and redundancies to handle the unexpected

2) Software

We use a comprehensive set of methodologies to evaluate the performance of our driving software, testing how it completes trips autonomously while avoiding conflicts and adhering to applicable road rules. Dense urban environments present particular challenges against which we rigorously test the Waymo Driver. These include complex intersections, narrow streets, ever changing layouts, and social interactions with other drivers, cyclists and pedestrians. The latter requires the Driver to understand other road users’ intentions and accurately predict their next moves, for example, in a situation like this:

One example is navigating tight spaces. For instance, a commonly proposed approach to help autonomous vehicles avoid conflicts on the road is to bake hard rules into the driving software—like “always stay at least 40 cm away from any other object.” But that doesn’t work in dense cities, where vehicles have to routinely manage tight turns and roadways. We’ve engineered the Waymo Driver to drive safely by adapting to the specific situation—like navigating the tight space between a double-parked truck and an oncoming car in the other lane. Last year, we gradually deployed these new capabilities, carefully testing the software updates and evaluating our Driver’s performance. Consistent with our safety framework, we comprehensively evaluate how our software handles these situations, and as a result, the Waymo Driver behaves more confidently than before, allowing us to tackle even harder driving environments.

Another example is weather. We conduct weather testing and evaluation to understand how the Waymo Driver may perform differently in diverse conditions. We look at specific weather challenges across our hardware, software, and operations and compare our Driver’s performance to the targets and metrics of our safety methodologies that evaluate safe driving from many different angles. This means our Driver can behave differently in inclement weather conditions, for example, driving more slowly on slippery roads or proceeding cautiously at intersections in the fog. For fog in particular, we made updates in our fifth generation hardware and perception system that help the Waymo Driver handle foggy conditions even more effectively.

3) Operations

Evaluating our hardware and software and how they interact with the base vehicle platform helps us deploy safe and effective autonomous vehicles. But serving our riders also requires assessing the performance of our fleets and operations—particularly in a city as complex as San Francisco.

Waymo’s Risk Management Program identifies and resolves potential safety issues before any new or updated features or software are used on public roads. We also have a Field Safety Program that complements the Risk Management Program by collecting, assessing, and resolving potential safety concerns—whether they originate from our riders, employees, or the public. The dynamic nature of both these programs helps us proactively address new potential risks associated with operating in a new or changing environment.

As we always continue to refine and advance our service, we made a number of advancements last year. For example, we’ve fine-tuned our fleet operations to account for the many rules around pick-ups and drop-offs on San Francisco’s busy streets. We’ve also refined our Waymo One service in both Arizona and California thanks to rider feedback; for example, adding features that let riders add multiple stops to their journeys and provide clearer directions to pick-up points to meet requests from people with accessibility challenges.

New survey results show our riders trust the Waymo Driver

This all translates into rider trust. According to our recent survey, 94% of our riders in San Francisco are satisfied with the Waymo riding experience and 97% expressed trust in our technology.

Many riders reported that using the Waymo One service inspired new excitement about the technology and made it easier for them to get around. In particular, riders like Rolinda often rely on Waymo for shopping and running errands, activities and leisure, as well as visiting friends and evenings out. When compared to other ride-hailing services, most of the respondents felt that Waymo is better at driving safely and providing a safe riding space where they feel at ease.

As we’ve designed our evaluation frameworks over the past 12 years, we’ve learned that assessing the technological, safety, and service readiness of autonomous driving technology requires a complex set of methodologies to get the full picture of a system’s safety. As we work to make it safer and easier for San Franciscans to get around, we’re proud of the progress we’re making building a safe Driver for everyone.

Those interested in a deep dive on Waymo’s robust safety methodologies, may want to review this summary, which provides a deeper look at the detailed analysis and validation methods Waymo uses to determine the Waymo Driver’s readiness across its hardware, software, and operational layers.