ByteDrive-Toy: From Pixels to Steering

Abstract Project summary

ByteDrive-Toy organizes autonomous-driving research into two learning stages. First, a frozen DINOv3 visual backbone supports joint 29-class semantic segmentation and metric depth estimation. Second, the learned perception representation aggregates three camera frustums, LiDAR voxels, and a rigidly aligned historical BEV into a forward-facing bird's-eye view, then jointly predicts spatial fields, road structure, traffic controls, multimodal trajectories, and behavior labels.

After training, the closed-loop runner connects the model to synchronous CARLA simulation: observation → two-frame inference → trajectory selection → pure-pursuit/PID control → next observation. It records episode terminal state, progress, diagnostics, and videos.

1 System Overview End-to-end pipeline

The repository covers the complete path from synthetic data generation to simulated vehicle execution. Its complexity comes from real dependencies between modules, not a collection of feature labels.

1.1 Heterogeneous data system Data System

A Python 3.7 worker drives CARLA 0.9.15 while a Python 3.12 collector orchestrates, encodes, and stores data. The runtimes exchange frames through a shared-memory arena. Sensors are normally recorded at 2 Hz and kinematics at 10 Hz; RGB uses H.265 and non-RGB data uses LMDB.

1.2 Perception pretraining Frozen visual backbone

The frozen DINOv3 ViT-S+/16 backbone returns complete token sequences from layers 3, 6, and 12. Their multiscale representation jointly supports 29-class semantic segmentation, symlog depth regression, and depth-range classification, then remains available to the driving model.

1.3 Geometry-aware BEV and joint planning Driving Model

Camera calibration maps patch centers and corners through sampled depths into 3D. Camera frustums, LiDAR voxels, and the preceding BEV meet in a forward 64 m × 64 m bird's-eye space. Eight learned mode tokens query intermediate BEV features while conditioned on the route target and planar ego velocity.

2 Qualitative Results Repository-generated outputs

The following images and videos are produced directly by tools in the repository. They are experimental outputs used to inspect prediction quality, spatial alignment, and closed-loop behavior.

Expand ↗

Expand ↗

2.1 CARLA closed-loop execution Closed-loop Episodes

The top row shows the left/front/right cameras and vehicle state, the middle row shows spatial fields and road structure, and the bottom row shows every candidate trajectory and the executed winner.

3 Implementation Status Current repository scope

The table distinguishes working implementation from research goals that have not been implemented.

4 Scope & Limitations Research boundary

• Limited data distribution. Current training and demonstrations focus on `Town02_Opt`, one expert style, and a fixed collection recipe.
• Two-frame temporal context only. The model uses one rigidly aligned historical BEV and has no long-term memory.
• Behavior cloning compounds error. Stopping, drifting, collision, and low route completion remain possible.
• CARLA simulation research only. Do not use this system for real roads, real vehicles, or safety-critical decisions.

5 Start from the Repository Documentation and source

The primary README documents tensor shapes, training, configuration, data formats, weights, closed-loop execution, and known limitations. This page is an introduction; the repository is the source of truth.