Skip to main content
Waymo Open Dataset
Challenge

Scenario Generation

Overview

**Note: The tutorial notebook demonstrates the process and steps discussed on this web page, including sample generation of scenarios, computing the metrics, and generating files to submit to the evaluation server.**

The Scenario Generation Challenge expands on the Sim Agents Challenge.  In the Sim Agents challenge, the task is as follows: Given one-second pose histories of all agents on a provided map, simulate 32 realistic joint futures for all the agents.  In the Scenario Generation Challenge, the task is extended to include generating the initial states (position, bounding box size) and full trajectories for all the expected agents in the scenario.  More specifically, the task is as follows: Given one-second pose history of just the ego agent on a provided map, and the number and types of agents expected in the scenario (vehicles, pedestrians, cyclists), generate 32 realistic instantiations of the scenario, including the initial states (position, bounding box dimensions) and full trajectories (past, current, and future) for the requested agents.

Scenario Generation diagram

Leaderboard

Submit

A validation or test set submission may be submitted to the evaluation server. Our tutorial notebook contains code for creating submission protos for a dataset split. The sample code creates one binary SimAgentsChallengeSubmission proto for each shard. Please refer to the tutorial notebook for the naming convention of these files. Once you have the 150 files, compress them as a single .tar.gz archive and upload the single file.

To be eligible to participate in the Challenge, each individual/all team members must read and agree to be bound by the WOD Challenges Official Rules.

You can only submit against the Test Set 3 times every 30 days. (Submissions that error out do not count against this total.)

Sign in to submit

You must be signed in to upload submissions. Please sign in to continue.

Sign in

Dataset

The WOD Motion Dataset contains agent positions (and other attributes) at 10 Hz. Each example in the training and validation sets contains 1 second of history data, 1 timestep for the current time, and 8 seconds of future data. This corresponds to 10 history timesteps, 1 current timestep, and 80 future timesteps, for a total of 91 timesteps per scene. The 91 timesteps in each example can be organized as:

{t - 10, t - 9, …, t - 1, t, t + 1, …, t + 79, t + 80}

where t indicates the current timestep. Note that agents are not necessarily present in all timesteps. For example, an agent may get occluded in the future, or it may be currently occluded and only appear in future data. Presence and absence of agents at particular timesteps is indicated by the valid data attributes.

In addition to the agent positions (x, y, z) and headings (bbox_yaw) over time, the dataset also contains the bounding box dimensions (length, width, height) and type of agents (vehicle, pedestrian, cyclist). Please see the tf.Example format for more information.

Task Inputs and Outputs

Here are the inputs that are available to participants:

  • The road graph (map)

  • The history of traffic light states (past and current timesteps)

  • The history of the ADV (past and current timesteps)

  • The number of agents to generate for each type

Note: both the validation and test subsets in the WOD Motion Dataset contain the history of agent tracks, which reveal their initial states as well as their current and past positions over 1 second. Participants should ignore this information and train their models to generate full agent trajectories without looking at this data. Similarly, participants should not use the camera or lidar data in the open dataset for this challenge. We will sanity check results, and submissions where there is significant copy/paste action detected may be disqualified.

The Scenario data can only be used to determine the number of agents to generate for each agent type. Our tutorial notebook contains the following utility functions:

  • num_agents_per_type(): Determines the number of agents to generate for each type. The numbers are determined based on the number of agents which are valid at the current timestep.

  • strip_logged_trajectories(): Strips the ground truth data of privileged information that should not be visible to the model during training and inference on validation or test subsets.

Given only the inputs listed above, the model is expected to generate 32 full instantiations (rollouts) of the scene containing all the required agents. In each rollout, for each required agent, the model should generate a full trajectory over 91 timesteps (spanning across past, current, and future), including agent positions (center_x, center_y, center_z), heading, and bounding box dimensions (length, width, height). The bounding box dimensions can stay constant throughout the trajectory.

These generated trajectories are stored inside a SimulatedTrajectory proto and then bundled together inside a JointScene proto holding trajectories for all agents in the scene. A ScenarioRollouts proto holds 32 rollouts for each scenario and a SimAgentsChallengeSubmission proto holds scenario generations for all scenarios in one dataset shard. Submissions are uploaded as serialized SimAgentsChallengeSubmission protos.

Metrics and Evaluation

Our evaluation metrics are a generalization of the Sim Agents evaluation metrics. We generate distributions of behaviors from the 32 rollouts for each scenario and then measure how likely the ground truth data is against these distributions by computing the negative log likelihood of the ground truth. In the Sim Agents Challenge, distributions are created and scored individually for each agent. In contrast, in the Scenario Generation Challenge we generate and score collective distributions over all the agents in the scenario. This generalization allows the order of agents in the generated scenarios to be different from the ground truth.

The metrics cover behavior measurements in three categories: agent motion, agent interactions, and map / road adherence. Agent motion attributes compute per-timestep distributions of agent linear velocity, angular velocity, linear acceleration, and angular acceleration from the generated rollouts. Agent interaction features measure per-timestep occurrences of bounding box collisions, distance to nearest object, and time to collision (vehicles only).  Finally, map adherence attributes capture per-timestep occurrences of offroad driving, traffic light violations, and distribution of distances to the road edges. Metrics from these three categories are aggregated into a single metametric according to specified weights.  Please see the Sim Agents Challenge page for more details.

Publications

Our team has published the following two papers on the scenario generation problem.

Rules Regarding Awards

See the Official Challenge Rules here.