A team of researchers led by Nvidia has released DreamDojo, a new AI system designed to teach robots how to interact with the physical world by watching thousands of hours of human videos – a development that could significantly reduce the time and cost required to train the next generation of humanoid machines.
The research, published this month and involving collaborators from UC Berkeley, Stanford, the University of Texas at Austin and several other institutions, echoes what the team says "First-of-its-kind robot world model that exhibits strong generalization to diverse objects and environments after training."
At the core of DreamDojo is what researchers described "A large-scale video dataset" Involved "44K hours of diverse human egocentric videos, the largest dataset ever for world model pretraining." The dataset, called DreamDojo-HV, is a dramatic leap in scale – "15 times longer span, 96 times more skills, and 2,000 times more scenes than the previous largest dataset for world model training," As per project documentation.
Inside the two-step training system that teaches robots to see like humans
The system operates in two distinct phases. First of all, DreamDojo "Acquires comprehensive physical knowledge from large-scale human datasets by pre-training with latent functions." then it passes "After training on the target avatar with continuous robot actions" – Essentially learning general physics by observing humans, then fine-tuning that knowledge to specific robot hardware.
For enterprises considering humanoid robots, this approach addresses a stubborn obstacle. Teaching a robot to manipulate objects in unstructured environments traditionally requires enormous amounts of robot-specific performance data – costly and time-consuming to collect. DreamDojo addresses this problem by leveraging existing human videos, allowing robots to learn from observations before touching a physical object.
One of the technological breakthroughs is speed. Through the distillation process, researchers achieved "Real-time conversation at 10 fps for more than 1 minute" – A capability that enables practical applications such as live teleoperation and on-the-fly planning. The team demonstrated the system working on multiple robot platforms, including the GR-1, G1, AgiBot and YAM humanoid robots, showing what they say "Realistic action-conditional rollout" across "A wide range of environments and object interactions."
Why is Nvidia betting big on robotics as AI infrastructure spending increases?
This release comes at a key moment for Nvidia’s robotics ambitions and the broader AI industry. At the World Economic Forum in Davos last month, CEO Jensen Huang declared that AI represents robotics "once in a generation" Opportunities, especially for sectors with a strong manufacturing base. According to DigiTimes, Huang also said that what the next decade will hold "An important period of accelerated development of robotics technology."
The financial stakes are huge. Huang told CNBC "halftime report" It was reported on February 6 that the tech industry’s capital spending – potentially reaching $660 billion this year from major hyperscalers – are "Fair, reasonable and sustainable." He portrayed the present moment as "The largest infrastructure construction in human history," Companies like Meta, Amazon, Google, and Microsoft have dramatically increased their AI spending.
That infrastructure push is already reshaping the robotics landscape. Robotics startups to raise a record $26.5 billion in 2025, according to data from Dealroom. European industrial giants including Siemens, Mercedes-Benz and Volvo have announced robotics partnerships in the past year, while Tesla CEO Elon Musk has claimed that 80 percent of his company’s future value will come from its Optimus humanoid robot.
How DreamDojo can transform enterprise robot deployment and testing
For technology decision makers evaluating humanoid robots, DreamDojo’s most immediate value may lie in its simulation capabilities. Researchers highlight downstream applications that include "Reliable policy evaluation without model-based planning for real-world deployment and test-time improvement" – Capabilities that could let companies simulate robot behavior on a large scale before committing to expensive physical tests.
This matters because the gap between laboratory demonstrations and the factory floor remains significant. A robot that performs flawlessly under controlled conditions often grapples with the unpredictable variations of real-world environments – different lighting, unfamiliar objects, unexpected obstacles. By training on 44,000 hours of diverse human videos spanning thousands of scenes and nearly 100 different skills, DreamDojo aims to build the kind of general physiological intuition that makes robots adaptable rather than brittle.
Research team led by Linxi "gym" Fan, Joel Zhang, and Yuke Zhu, with Shenyuan Gao and William Liang as co-first authors, have indicated that the code will be released publicly, although no timeline was specified.
The big picture: Nvidia’s transformation from gaming giant to robotics powerhouse
It remains to be seen whether DreamDojo translates into commercial robotics products. But the research hints at where Nvidia’s ambitions are headed as the company increasingly establishes itself beyond its gaming roots. As Kyle Barr observed at Gizmodo earlier this month, Nvidia now sees "Anything related to gaming and ‘personal computers’" As "Outliers on Nvidia’s quarterly spreadsheet."
This shift reflects a calculated bet: that the future of computing is physical, not just digital. Nvidia has already invested $10 billion in Anthropic and has signaled plans to invest heavily in OpenAI’s next funding round. DreamDojo suggests the company sees humanoid robots as the next frontier where its AI expertise and chip dominance can come together.
For now, the 44,000 hours of human video at the heart of DreamDojo represent something more fundamental than a technical benchmark. They represent a theory – that robots can learn to navigate our world by watching us live in it. It turns out that machines are taking notes.
<a href