Manuscript · 2026

Daphne Cornelisse1 · Julian Hunt2 · Zixu Zhang3 · Waël Doulazmi4,5 · Kevin Joseph2 · Jaime Fernández Fisac3 · Eugene Vinitsky1

1NYU Tandon School of Engineering · 2NYU Courant · 3Princeton University · 4Centre for Robotics, Mines Paris · 5Valeo

Spiced self-play combines over 60 years of simulated self-play with 30 minutes of human driving data as a behavioral anchor, improving coordination with logged human trajectories without reward engineering or domain randomization.

99.4% safe task completion with human-replay proxies

2,500× less human driving data than imitation-learning baselines

20B self-play transitions, roughly 63 years of driving

15 hrs end-to-end training on one consumer GPU

Abstract

Spiced self-play adds a small behavioral anchor to scalable self-play.

Self-play reinforcement learning can substitute cheap, large-scale simulation for large human driving datasets, but policies trained only through self-play can converge to effective yet incompatible driving conventions. Prior work often addresses this through reward engineering and domain randomization.

We introduce spiced self-play: policies trained with a minimal safe-goal-reaching reward, over 60 years of self-play simulation, and 30 minutes of human driving data as a behavioral anchor. The resulting policies coordinate with logged human trajectories using 2,500× less human data than imitation-learning baselines, and the full pipeline runs in 15 hours on a single consumer-grade GPU.

Rollouts

These videos align paired held-out human-replay scenarios so the regularized policy and unregularized policy can be compared in a shared camera view. In each rollout, the learned ego agent is goal-conditioned: the blue vehicle shows the regularized policy, the yellow vehicle shows the unregularized policy, the green circle marks the goal, and the other light green vehicles replay recorded human trajectories. Both policies use the same minimal reward: +1 when the goal indicator I[goal] is true, and -1 for off-road or collision events.

Cross-traffic negotiation.
Dense queue interaction.
Waiting in traffic.
Awkward behavior avoided.
Room left around replayed traffic.
Measured progress near the goal.
Cautious timing through a busy interaction.
Patient interaction around nearby vehicles.

Summary

Spiced self-play uses a pinch of human data to steer scalable self-play toward human-compatible driving.

Self-play RL can scale through simulated experience, but policies trained only against themselves may adopt effective driving conventions that do not coordinate with human drivers. Spiced self-play keeps self-play as the main training engine and adds a small behavioral cloning anchor from human driving data.

With 30 minutes of Waymo human driving data and 20B self-play transitions, spiced policies reach 0.994 safe task completion under human replay, outperforming both unregularized self-play and SMART-tiny CLSFT while avoiding reward engineering and domain randomization.

Spiced objective

PPO training uses a minimal safe-goal-reaching reward, while the behavioral anchor keeps updates close to conventions observed in a small human dataset.

Human replay

Held-out scenes replay logged human agents while the learned policy controls the ego vehicle, exposing coordination failures that self-play evaluation can miss.

Data scaling

The experiments vary map diversity and human driving data from minutes to full-dataset references to measure when coordination emerges.

Evaluation

Human compatibility is measured by safe task completion under replay.

Diagram comparing self-play evaluation, human-replay collision events, and human-replay at-fault collision events.
Safe task completion combines task completion with at-fault collisions, distinguishing failures caused by the learned policy from collisions initiated by replayed human agents.

Results

Spiced self-play improves safe coordination with a fraction of the human data.

Plots comparing safe task completion, total training transitions, and trajectory behavior for spiced self-play, self-play, and imitation-learning baselines.
Headline result from the current manuscript. Spiced self-play reaches 0.994 safe task completion with 20B self-play transitions and roughly 30 minutes of human driving data, improving over unregularized self-play while using 2,500× less human data than imitation-learning baselines.
Charts comparing collision severity distributions for unregularized and spiced self-play policies.
Spiced policies also change the character of failures: collisions are less severe on average and concentrate at lower per-event speed changes than the unregularized self-play baseline.

Regularized
Unregularized

Failure comparison

Different collisions in the same scene

Both policies fail in replay, but the paired rollout shows regularized agents display more cautious behavior around other agents.

Regularized
Unregularized

Failure comparison

Representative replay failure

This pair illustrates a remaining failure mode under human replay, including cases where the controlled vehicle is contacted from behind.

Citation

BibTeX


          @misc{cornelisse2026humanlikeautonomy,
          title         = {Human-like autonomy emerges from self-play and a pinch of human data},
          author        = {Cornelisse, Daphne and Hunt, Julian and Zhang, Zixu and Doulazmi, Wa{\"e}l and Joseph, Kevin and {Fern{\'a}ndez Fisac}, Jaime and Vinitsky, Eugene},
          year          = {2026},
          eprint        = {2606.19370},
          archivePrefix = {arXiv},
          primaryClass  = {cs.LG},
          url           = {https://arxiv.org/abs/2606.19370}
        }