Real-time 2D Detection

Leaderboard

To see results of the 2D Detection Challenge without Latency, select 'Show all' in the 'Latency (s)' column of the leaderboard.

Submit

To submit your entry to the leaderboard, upload your file in the serialized protobuf format specified here. If you are participating in the challenge for awards, or submitting for latency evaluation, please note that you would be required to submit a docker image to a Bucket on Google Cloud Storage or to GoogleContainer/Artifact Registry and indicate resource location using docker_image_source field of the submission proto. The details for submitting a docker image can be found here.

If there are many predictions, you can shard them across multiple files where each file contains a subset of the predictions. Then tar and gzip them into a .tar.gz file before uploading.

To be eligible to participate in the challenge, each individual/all team members must read and agree to be bound by the Official Challenge Rules.

You can only submit against the Test Set 3 times every 30 days. (Submissions that error out do not count against this total.)

Metrics

Leaderboard ranking for this challenge is by Mean Average Precision (mAP) / L2 among "ALL_NS" (all Object Types except signs), that is, the mean over the APs of Vehicles, Cyclists, and Pedestrians. Only submissions that run faster than 70 ms/frame on a Nvidia Tesla V100 GPU will be eligible to win the challenge. Only camera data can be used. And we enforce a causal system, i.e, for a frame at time step t, only sensor data up to time t can be used for its prediction.

Primary metric

Average Precision (AP): ∫p(r)dr where p(r)is the PR curve

Latency

The latency of your submission is measured on a Nvidia Tesla v100 GPU in milliseconds per frame (ms/frame).

IoU Overlap Threshold

Vehicle 0.7, Pedestrian 0.5, Cyclist 0.5

Sensor Names

C: All cameras
I: Invalid

Label Difficulty Breakdown

Each ground truth label is categorized into different difficulty levels (two levels for now):

• LEVEL_1, if not marked as LEVEL_2 in the released data.

• LEVEL_2, if marked as LEVEL_2 in the released data. When evaluating, LEVEL_2 metrics are computed by considering both LEVEL_1 and LEVEL_2 ground truth.

Metric Breakdown

The following metric breakdowns are supported:

• OBJECT_TYPE: Breakdown by object type ("ALL_NS" refers to all objects except signs: Vehicle, Cyclist, and Pedestrian)

• RANGE: Breakdown by the distance between object center and vehicle frame origin. [0, 35m), [35m, 50m), [50m, +inf)

Rules regarding awards

See the Official Challenge Rules here.