Rides
Our service
Phoenix
San Francisco
Los Angeles
Rides
Our service
Phoenix
San Francisco
Los Angeles
Technology
Technology
About
Our History
Waymo Leadership
Latest Updates
Press Resources
About
Our History
Waymo Leadership
Latest Updates
Press Resources
Safety
Safety
Community
Community
Careers
Benefits
Values
People
Open Roles
Careers
Benefits
Values
People
Open Roles
GINA-3D: Learning to Generate Implicit Neural Assets in the Wild
Bokui Shen
Xinchen Yan
Charles R. Qi
Mahyar Najibi
Boyang Deng
Leonidas Guibas
Yin Zhou
Dragomir Anguelov
Abstract
Update 8/2023: The large-scale, object-centric dataset we constructed (Object Assets - Waymo Open Dataset) is now available for download at [https://waymo.com/open/data/perception/#object-assets](https://waymo.com/open/data/perception/#object-assets).
Modeling the 3D world from sensor data for simulation is a scalable way of developing testing and validation environments for robotic learning problems such as autonomous driving. However, manually creating or re-creating real-world-like environments is difficult, expensive, and not scalable. Recent generative model techniques have shown promising progress to address such challenges by learning 3D assets using only plentiful 2D images -- but still suffer limitations as they leverage either human-curated image datasets or renderings from manually-created synthetic 3D environments. In this paper, we introduce GINA-3D, a generative model that uses real-world driving data from camera and LiDAR sensors to create realistic 3D implicit neural assets of diverse vehicles and pedestrians. Compared to the existing image datasets, the real-world driving setting poses new challenges due to occlusions, lighting-variations and long-tail distributions. GINA-3D tackles these challenges by decoupling representation learning and generative modeling into two stages with a learned tri-plane latent structure, inspired by recent advances in generative modeling of images. To evaluate our approach, we construct a large-scale object-centric dataset containing over 520K images of vehicles and pedestrians from the Waymo Open Dataset, and a new set of 80K images of long-tail instances such as construction equipment, garbage trucks, and cable cars. We compare our model with existing approaches and demonstrate that it achieves state-of-the-art performance in quality and diversity for both generated images and geometries.
Share
Links
Download PDF
ArXiv
Copy BibTeX
Copied!
Publication
CVPR 2023
Topics
2023
Perception
CVPR
Simulation