Behind the Innovation: AI & ML at Waymo

Our mission at Waymo is simple: be the world’s most trusted driver. The steps to get there, however, are considerably more complex. Getting this right requires solving some of the toughest and most impactful AI and ML challenges of our time. That wouldn’t be possible without our growing world-class AI team.

For over 15 years, we’ve been building robust systems to safely transport riders on their daily journeys – from work commutes to doctor’s appointments. From navigating complex traffic patterns to interpreting other road users’ intent, we’re developing the Waymo Driver to seamlessly and autonomously operate in the real world. 

Srikanth Thirumalai, VP of Engineering, Onboard: “The problem we’re trying to solve is how to build autonomous agents that navigate in the real world. This goes far beyond what many AI companies out there are trying to do.”

What makes our work particularly compelling – and challenging – is building state-of-the-art models that handle the full complexity of real-world driving, a social task that at scale necessarily encompasses many long-tail scenarios. From erratic behavior of fellow road users to rapidly changing weather conditions, our goal is to build a system that consistently and reliably handles these edge cases.

Chen Wu, Head of Perception: “Waymo offers a unique opportunity to apply the most advanced cutting-edge technology to achieve a product that has never been created before.”

We deploy some of the world’s most advanced AI and ML technologies, powered by our cutting-edge research, across our stack. The Waymo Driver uses a rich sensor suite (lidar, radar, cameras, and external audio receivers) to perceive the dynamic environment and road users, predict their behavior, and plan and navigate a journey from A to B in real time. 

With advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs), we’re pushing our cutting-edge capabilities and the boundaries of AI even further.  Our next generation AI models combine Waymo’s driving experience and AV-specific AI advancements with the ‘world knowledge’ and reasoning capabilities of LLMs/VLMs to create models specifically applicable to the driving context. We call this architecture the Waymo Foundation Model.

These models are significantly enhancing the capabilities of the Waymo Driver from interpreting the scene, to generating driving plans and agent trajectories. Our unrivaled compute infrastructure and advanced closed-loop simulation systems allow us to iterate rapidly, pushing the boundaries of what’s possible with embodied AI. Waymo Foundation Model is also significantly enhancing the capabilities of those closed-loop simulation systems, simulating realistic future world states and other road users’ behavior.

Drago Anguelov, Head of Research: “There is an opportunity to build a Waymo Foundation Model that marries ideas from the AV space with innovation in generative AI to obtain the most compelling embodied AI and the most trusted driver.”

Just as important is making this level of autonomy work at scale. At Waymo, we work with one of the most sophisticated real-world AI systems today that enable us to serve hundreds of thousands of riders every week across diverse driving environments.

With every mile our autonomous vehicles drive, our AI models can learn and improve to make each trip safer and more reliable, while meeting the high standards of our safety-critical domain. Robust evaluation is key to our industry-leading safety record, and we’ve spent years refining a comprehensive set of methodologies to assess safety across our technology and operations.

The most interesting work for Waymo in AI is still ahead, as we continue to scale the Waymo Driver. With world-class infrastructure, rapid iteration, and a real-world product operating at scale – there has never been a more exciting time to work on ML and AI at Waymo. If you’re passionate about solving some of AI’s biggest challenges in autonomous vehicles, robotics, and beyond – join us!