HABIT, the Dataset for Interactive Action Models
The Config Team · 6 min read
Previously, we wrote about teaching robots to coexist with humans. What we showed there was one slice of a larger idea at the time: a robot working side by side with a person — sorting recyclables while a human sorts alongside it, learning to pause and yield from the data alone. In the vocabulary we use now, that was the Coworker role.
But coexistence is bigger than working side by side — because the future of work itself is bigger than automation. At Config, we don't believe every part of every job will be handed to a robot: many tasks are inherently shared, and a robot built for that future needs more than one way of being around people. Sometimes it must act together with a person on the same object — a handover, a jointly-held tray. Sometimes it works side by side, splitting the load. Sometimes it simply follows direction, with nothing more than a pointing gesture. HABIT is the expansion of that concept into the full picture of Interactive Action Model that enables human-robot collaboration: three interaction roles — Collaborator (act together), Coworker (work side by side), and Supervisor (follow human cues) — 60 tasks, and a human in every single episode.
Today, two announcements. First, the first version of HABIT is now publicly available on Hugging Face, accompanied by our paper and project page. Second, we go one step beyond the paper: we show CFG-1, Config's robot foundation model, trained on HABIT.
HABIT in brief
HABIT (Human-Aware Behavior and Interaction Training dataset) is a large-scale bimanual manipulation dataset in which a human shares the workspace in every episode: 10,563 episodes, 164 hours, 60 tasks, evenly spanning the three roles above. Task design, the reactive-interaction collection protocol, and the full evaluation framework are in the paper; the project page has the visual tour. Here is what CFG-1, trained on HABIT, looks like across six of the tasks:
The claim that matters: the value is in the data
The paper evaluates HABIT with two open-source VLAs, π0.5 [1, 2] and GR00T N1.6 [3, 4], and finds the same pattern in both: human-present training improves success on every comparable task and removes each role's characteristic interaction failure (details and per-task breakdowns in the paper).
For this post, we ran the paper's identical evaluation protocol with CFG-1.
The same comparison, qualitatively — CFG-1 under Robot-only vs. HABIT training, on each of the four comparable tasks:
With CFG-1, HABIT improves every comparable task as well — and the gains organize cleanly by role. On the two Coworker tasks, where the characteristic failure is collision, success jumps from 40% → 90% (Waste Sorting) and 20% → 75% (Box Packaging), with collisions dropping to zero in both. On the two Supervisor tasks, where the characteristic failure is gesture following, Donut Serving goes 35% → 100% and Food Storage 50% → 65%. (The two Collaborator tasks have no Robot-only baseline — they require a human partner by definition.) The trend matches the other two backbones: three models, three architectures, one consistent effect — the human-aware behavior comes from the dataset, not from any particular policy.
CFG-1 on HABIT: ceiling-level behavior
CFG-1's results on this benchmark are strong in absolute terms: 19/20 on Table Service with zero coordination violations, 20/20 on Donut Serving with zero gesture-following failures, 90% on Waste Sorting and 75% on Box Packaging with zero collisions across all forty Coworker trials. On its strongest tasks, this is at or near the ceiling of the evaluation protocol. These behaviors also hold under the paper's OOD protocol (Appendix E of [5]: unseen clothing colors, unseen body silhouettes): as with the two baselines, the remaining drops come from manipulation rather than yielding or synchronization — gesture grounding being the one appearance-sensitive case.
Get the data
- Dataset: huggingface.co/datasets/configinc/HABIT — all 10,563 episodes with five synchronized RGB streams (three robot-side, two human-side), joint- and Cartesian-space actions, and per-episode human and robot subtask annotations, under CC BY 4.0. Faces are blurred; body silhouette, clothing, and gesture are preserved. The two human-side views are untouched by the paper's experiments [5] — an open opportunity for gesture grounding, intent prediction, and human-motion-conditioned policies.
- Paper: arxiv.org/abs/2606.31682
- Project page & videos: habit-dataset.github.io
- Policy training code (for π0.5 and GR00T N1.6): github.com/HABIT-dataset/HABIT-policy-training
If you're interested in HABIT, CFG-1, or human-present data collection for your own deployment, talk to us: forms.config.inc/contact.
Follow us on X and YouTube to stay up to date with our latest work.
References
[1] Physical Intelligence, K. Black, N. Brown, et al., "π0.5: a Vision-Language-Action Model with Open-World Generalization," 2025. arXiv:2504.16054.
[2] K. Black, N. Brown, et al., "π0: A Vision-Language-Action Flow Model for General Robot Control," 2024. arXiv:2410.24164.
[3] NVIDIA, "GR00T-N1.6-3B," Hugging Face model card, 2026. huggingface.co/nvidia/GR00T-N1.6-3B.
[4] NVIDIA, J. Bjorck, et al., "GR00T N1: An Open Foundation Model for Generalist Humanoid Robots," 2025. arXiv:2503.14734.
[5] J. Song, S. Jeong, B. Jeon, S. Kim, M. Seo, H. Son, and K. Lee, "HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation," 2026. arXiv:2606.31682.