A year ago, robotics researchers were still arguing over which single kind of training data would make robots useful. Josh Gao, founder and CEO of the robot-data company Mecka AI, remembers people insisting that only one source would work: "it's only UMI data that's going to work. It's only egocentric data that's going to work." People have slowly realized, he said, that all of it is "essentially a data pot of soup," and every ingredient will matter at some point.
Gao made the case on an October 20, 2026 episode of Training Data, the podcast hosted by Sonya Huang of Sequoia Capital. Mecka sells training data to robotics companies, so Gao sees what many of them order. He described the mix they have settled on. Then he explained why he does not think simply making that mix bigger will produce a robot that can learn any job by watching.
The ingredients
Robots learn physical skills from recordings of tasks being done, and there are several ways to make those recordings. Egocentric data is first-person video and sensor data captured by people wearing cameras while they work. UMI data, named after the Universal Manipulation Interface, comes from people doing tasks with handheld grippers fitted with cameras and motion sensors, so that their movements can later be replayed by robot arms. The original UMI paper describes the gripper hardware and tests on jobs such as arranging cups, folding cloth with two arms and washing dishes. Teleoperation, or teleop, is a person remotely steering a real robot. It captures the robot's own body and surroundings, but it needs hardware, an operator and real time for every recording.
Huang asked whether the industry had converged on a set mix of internet video, egocentric data, UMI data and teleop, arranged in something like a triangle by volume. She recalled a visual of this kind from NVIDIA researcher Jim Fan at a Sequoia event. In a Sequoia interview, Fan divides robot training data into three complementary sources: internet material, simulation and recordings from real robots. Internet video is plentiful but generally lacks motor-control information, while real-robot data matches the actual machine but needs hardware, operators and time.
Gao said he does see a convergence. Today's recipe, he said, looks like "some version of" a million hours of human data across egocentric and other modalities, some tens of thousands of hours of UMI data, and a last layer of teleoperation and in-domain data, meaning recordings from the setting where the robot will actually work, amounting to "just a couple hours, even of that demonstration."
Published research suggests that combining the ingredients does not help automatically. In a July 2026 version of the EgoVerse study, researchers trained on human and robot demonstrations together across three robot platforms. Human data generally helped, but grocery bagging got worse on one robot while improving on the other two, a difference the authors associated with the robots' different manipulation strategies. Adding more varied human recordings paid off when human data matched to the specific task was also in the mix; either kind of human data alone gave little gain.
What is still missing from the pot
Gao said Mecka does not "subscribe aggressively to even one religion here," partly because some ingredients have not been added yet. Tactile data, from sensors that record touch and force, "haven't even entered the mix yet." If models will eventually need to learn "how much force it takes to apply towards the door handle," he said, that data will have to go into the pot. "Simulation will come into the data pot of soup," he added, and all of these sources will matter.
Federer and Move 37
Huang raised the objection she said she hears most often about this kind of data collection: "you could watch millions of hours of tennis data and you still wouldn't know how to beat Roger Federer."
Gao answered with a board game. "When you talk about robots playing tennis, you got to talk about robots playing Go," he said. He pointed to DeepMind's AlphaGo and its match against the Go champion Lee Sedol. AlphaGo began, Gao said, with large-scale behavior cloning, meaning it learned by imitating expert human games. Then it played against itself within the rules of the game, exploring possibilities until it arrived at "the magic of Move 37," a move he described as far outside what humans would have considered. DeepMind's account of AlphaGo confirms that sequence: training on expert human games, then reinforcement learning through self-play. AlphaGo beat Lee 4-1 in March 2016, and Move 37 came in the second game, where it helped AlphaGo win. DeepMind describes it as a move a human would have been exceptionally unlikely to play.
The difficulty for robots is that Go's rules are, in Gao's words, "the entire circumference of the world that you will operate in." A robot has to deal with the full, rich detail of the physical world, and "we ultimately don't have a world simulator that can recreate that today." Egocentric data, he said, serves two purposes in closing that gap: it grounds basic physics and the imitation stage, and it helps build the world simulator that self-play would need. Somewhere along the way, he said, robots will play tennis like Federer.
Asked whether that self-play would happen in simulation or in the real world, Gao said, "I could see a hybrid, ultimately." Simulation would not have to reconstruct the world in full detail right away, he said.
Dreaming a broken cup
That led Huang to world models, AI systems that learn to predict what happens next in a scene. She noted that the term is currently used almost interchangeably with video models. For Gao, their appeal is the chance to practice without consequences: being able "to even just dream what it means to drop a cup and have it shatter without actually doing it is incredibly valuable." He also sees a way to use the huge stock of internet and YouTube video, which these models learn from by predicting pixels. If that pixel prediction can be translated back into action trajectories, the paths a robot's body should follow, it could speed progress considerably, he said.
Training inside a learned simulator has an early demonstration in David Ha and Jürgen Schmidhuber's 2018 paper World Models. There, a small controller learned to dodge fireballs in a simple version of the video game Doom entirely inside the model's imagined version of the game. Moved back into the real game, it averaged about 1,100 survival steps over 100 trials, above the 750-step threshold for the task. The setting was a simple game, and early controllers learned to exploit errors in the imagined world, for instance by suppressing the fireballs, until the researchers made the model's predictions noisier during training.
Scaling the recipe, and what it won't reach
Huang said she no longer likes talk of a "GPT-3 moment for robotics." Her own test is simpler: "when will a robot actually fold my clothes or cook a meal for me?" She asked whether scaling up the current mix would get there.
Gao said scaling the same ratios by some order of magnitude would solve many tasks along the way, especially in constrained settings and business operations that follow specific standard operating procedures. The ideal is different: a robot that can be dropped into any situation, solve a task just by looking at it and handle real-world complications such as weather. For that robot, he said of the current recipe, "I don't think this is what it will look like, ultimately."
What it needs, Gao said, are more modalities, starting with tactile data. He likes the move toward what he called omni-models, which accept any kind of observation as input, "whether it is force, pixel, even smell in a very extreme case," and produce actions, voice or other controls as output. Huang asked whether chips that can sense smell even exist. "It's pretty early," Gao said, though he imagines that approach will eventually be unlocked.
Will the big AI labs steamroll robotics?
Huang then brought up GPT-6 Astra, which she said was generalizing "incredibly well" to robot tasks. She asked whether pure robotics companies would be "clobbered" by the labs building general AI, with the bitter lesson, the idea that general methods scaled up with more data and computing tend to beat specialized engineering, steamrolling robotics too.
One published test of the model's robot skills covers a narrower setting. NVIDIA's Astranomy report, published October 14, 2026, tested GPT-6 Astra on simulated pick-and-place tasks, five episodes per task and setup. Generating actions directly, the model fully completed 66.3% of tasks. A persistent agent that kept its interaction history reached 74.8%, and a setup in which Astra supervised a specialist NVIDIA robot policy reached 66.5%. The simulation paused while the model was thinking, with no time or retry limits. The authors note that repeated trials were limited, that differences between setups were not shown to be statistically significant, that inference was slow, and that insertion, assembly and fast-changing scenes were left out.
Gao gave the big labs credit for nerve. "It takes a lot of courage to scale up data with imperfect information," he said, and many of these labs have that in their DNA. As robotics data grows and is added to the pre-training mix of the main language and foundation models, he said, those models start to generalize across many domains. Huang summarized what she heard: the labs' robot results are not that surprising, because a fair amount of robot data has gone into their training. "It's actually quite expected when you think about it," Gao said.
He does not expect the general models to take everything. A model that can do everything will be very good at a lot of things over time, Gao said, but more targeted robotics foundation models might reach "four nines of capability and reliability," meaning 99.99%, on the particular tasks they are built for.