Human-robot collaboration (HRC) combines the complementary strengths of humans and robots to improve task efficiency. However, many existing collaborative systems rely on hand-engineered pipelines, limiting their scalability and flexibility for new tasks. In this work, we show that models trained end-to-end with imitation learning, specifically vision-language-action (VLA) models, can support collaborative manipulation, and characterize the key factors affecting their real-world performance.
We evaluate two state-of-the-art models and identify a failure mode of action-chunking policies in implicit HRC, where demonstration action leakage (i.e., action chunks crossing latent task transitions) can cause premature assistive behavior. We find that this issue increases with longer execution horizons and occurs in real-world collaborative VLA systems, such as when a robot attempts to hand over a tool before the person is ready. We propose an inference-time steering method to mitigate these erroneous assistive actions while preserving policy performance.
Finally, through a 16-participant user study on a long-horizon collaborative assembly task, we show that steering enables a longer execution horizon while mitigating premature assistance, leading to faster collaboration and fewer failures compared to a shorter-horizon policy.
We fine-tune VLAs on collaborative demonstrations and add inference-time interventions to improve timing and mitigate premature robot actions in implicit HRC.
Many HRC systems use hand-designed pipelines with dedicated modules for intent inference and human-aware motion planning. We ask whether a robot can instead be shown how to collaborate. In implicit collaboration, the robot has to read subtle cues, like which part the person is working on or whether they have put out their hand, and decide when and how to help. We train action-chunking VLAs on such collaborative demonstrations and find that they can learn these tasks, but they also have a characteristic timing failure.
Contributions
Experimental setup and the three collaborative assembly tasks, each shown in its initial state with the final assembly inset.
Sequential steps. Human and robot each place four blocks into a fixture. The person places red, blue or purple blocks, and the robot must answer with the matching color (either color for purple, to test multimodality).
Handovers both ways. The robot must recognize whether the person is attaching the seat or the back, hand over the matching bolts and screwdriver, and take both screwdrivers back at the end.
Long-horizon. About 5.5-minute demonstrations with two tool handovers, eight object handovers and two sequential manipulations. Used in the user study.
We compare a Diffusion Transformer policy (DiT) trained from scratch against a fine-tuned π0.5, each run at the demonstration rate and with a speed-up that consumes actions 1.5× faster. Both models predict 16-step action chunks, execute 8 steps before replanning, and are trained on 100 demonstrations at 10 Hz. To speed the robot up without jerky motion, each new action chunk is truncated to start at the point closest to the last commanded position. Each condition is evaluated over 10 rollouts with a collaborator who did not appear in the training data.
Full, uncut rollouts of each model (execution horizon 8), played side by side at the same speed. A panel dims once its rollout is finished.
| Abstract Assembly | Mini-Chair | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Policy | Progress (%) ↑ | Total time (s) ↓ | Robot time (s) ↓ | Mistakes ↓ | Failures ↓ | Progress (%) ↑ | Total time (s) ↓ | Robot time (s) ↓ | Mistakes ↓ | Failures ↓ |
| DiT | 95 ± 11 | 120 ± 34 | 109 ± 35 | 0.8 ± 0.6 | 2.1 ± 3.7 | 50 ± 53 | 318 ± 56 | 299 ± 68 | 6.2 ± 3.8 | 0.9 ± 1.1 |
| DiT + Speed | 100 ± 0 | 84 ± 15 | 73 ± 14 | 0.7 ± 0.7 | 0.5 ± 1.0 | 50 ± 53 | 275 ± 101 | 264 ± 107 | 7.6 ± 5.6 | 0.3 ± 0.5 |
| π0.5 | 100 ± 0 | 71 ± 4 | 58 ± 3 | 0.5 ± 0.5 | 0.0 ± 0.0 | 100 ± 0 | 121 ± 6 | 101 ± 5 | 0.1 ± 0.3 | 0.0 ± 0.0 |
| π0.5 + Speed | 100 ± 0 | 63 ± 3 | 51 ± 3 | 0.8 ± 1.1 | 0.0 ± 0.0 | 100 ± 0 | 125 ± 9 | 102 ± 9 | 0.1 ± 0.3 | 0.0 ± 0.0 |
Policy performance over 10 rollouts per condition (mean ± SD). Bold marks the best policy for each metric.
π0.5 outperforms DiT on both tasks. DiT often misread the person's initial intent in Mini-Chair, fumbled precise fetches of tools and bolts, and learned only one of the two assembly orders, suggesting that manipulation pretraining also helps collaboration. Speed-up helps most when the person is waiting on the robot: it shortened Abstract Assembly but gained little on Mini-Chair, where the robot usually fetches the next item while the person is still working.
While reviewing rollouts, we noticed that the robot sometimes handed over an object before the person extended their hand. The cause is action chunking. In a collaborative demonstration, a chunk whose observation still shows the robot waiting can include the motion that follows once the person signals readiness. We call this demonstration action leakage: a training chunk that spans a transition in the human's latent intent.
Two action chunks with nearly identical "waiting" observations. One contains only idle actions; the other leaks into the upcoming handover motion.
A false start at real-time speed. The collaborator's hands stay on the table, but the robot begins the handover anyway.
Because waiting observations look almost identical, the policy can sample a leaky chunk at any point while it waits. This produces a false start that drifts into out-of-distribution states and ends in a false commitment, such as a premature handover.
The problem grows with the execution horizon. Shortening the horizon reduces false starts, but it cannot eliminate them, because any chunk can cross the boundary.
Direct evidence. At a representative Mini-Chair handover, all 100 action chunks sampled from π0.5 contain a transition from waiting into future task motion. Executing only the first 4 steps (dashed line) stays clear of the leaked motion in this sample, but longer execution horizons execute more and more of it.
When we manually removed leaky chunks from the training set, only 1 of 100 sampled chunks leaked, and the robot stayed in place until the collaborator signaled readiness.
Chunks that end closer to the basin get higher reward, and the basin pull further encourages waiting.
We bias the policy toward waiting at inference time, with no retraining or architecture changes. We anchor steering on basin points, which are waiting positions found by MeanShift clustering of the training data.
FK steering. We adapt Feynman-Kac steering to sample action chunks from a tilted distribution πtilt(a | c) ∝ πθ(a | c) exp(λ r(a)). While the robot is inside a basin, a set of particles is denoised in parallel and resampled toward chunks that end near the basin without drifting away from it. With 16 particles, π0.5 inference takes 88.8 ms, compared with 67.2 ms without steering.
Basin pull. Noise can still let the robot creep out of a basin, so we also interpolate the commanded end-effector position toward the active basin, weighted by how conservative the best particle is.
Both mechanisms turn off outside the basin radius, where the policy samples from its original distribution.
Waiting to hand over with an idle collaborator. Without steering, the robot repeatedly drifts toward the person. With steering, it holds its waiting position, even though the clip covers a much longer span.
Waiting experiments. We ran the sped-up π0.5 for five minutes at a handover boundary with an idle collaborator (8 trials per condition). Longer execution horizons produced more false commitments and drift. FK steering combined with basin pull reduced both at every horizon.
On the Radio task at an execution horizon of 12, steering cut premature commitments from 7 (± 4) to 0 and waiting drift from 0.99 m to 0.06 m over five minutes. The cost is a slightly slower reaction to handover cues: 0.40 s on average over 50 handovers, compared with 0.29 s without steering.
Sixteen participants completed the Radio assembly with two configurations of the sped-up, fine-tuned π0.5, in counterbalanced order. The two conditions are the two ways of avoiding false starts described above:
Radio assembly with each condition.
User study results (n = 16, paired t-tests). Light bars: short horizon (4). Dark bars: 12+Steer.
@misc{xu2026learning,
title={Learning to Assist: Collaborative VLAs for Implicit Human-Robot Collaboration},
author={Leo Xu and Letian Li and Alex Cuellar and Michael Hagenow},
year={2026},
eprint={2606.12475},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2606.12475},
}We are grateful to Tutor Intelligence for donating the robot arm featured in this project. We thank Kerimcan Turkucu for his help collecting interaction data for each of our tasks and Johnny Hong for helping run the demonstration-budget experiments in the appendix. We also thank William Chen for helpful discussions.