How Are Humanoid Robots Taught Everyday Tasks?

A humanoid does not learn to load a dishwasher or fold a towel by being told the steps. It learns from examples of hands doing the task — captured as video, broken into short action segments, and used to train a policy that maps what the robot sees to what it should do next. That training data comes from a mix of sources: egocentric human video captured from a head-worn or head-mounted viewpoint, teleoperated demonstrations on the actual robot hardware, and in some cases video of humans performing the task with no robot present at all.

The underlying pipeline is staged. Large volumes of human and human-adjacent data build a general sense of objects, hands and physics during pretraining; a smaller, more task-specific set of demonstrations aligns that general knowledge to the robot’s actual body and sensors during mid-training; and real-robot rollouts with corrections refine the policy on the exact tasks it needs to perform during post-training. /guides/robot-pretraining-data walks through how those three stages fit together across the field.

This page focuses on everyday, household-style tasks specifically. For the broader methods comparison across data sources, see /guides/humanoid-robot-data-collection-methods. For what to capture when the data source is egocentric human video, see /guides/egocentric-data-for-humanoid-robots.

Household tasks are too varied, and too dependent on subtle contact and timing, to specify by hand. A policy that tries to fold a towel needs to have seen many towels folded many ways — different fabrics, starting positions, failed grips and corrections — before it generalizes past the exact examples it was trained on. Human demonstration is the fastest way to get that variety: people already do these tasks constantly, in real homes, with real clutter and real mistakes.

Figure’s Project Go-Big is a published example of how far human-video demonstration alone can carry a humanoid skill. Figure states that training its Helix model on 100% egocentric human video, with no robot demonstration data at all, produced the company’s first zero-shot human-to-robot navigation transfer: Helix responding to commands like "walk to the kitchen table" by generating closed-loop navigation control through cluttered home environments — which Figure describes as the first time a humanoid has learned an end-to-end skill this way, with no robot-specific training (figure.ai/news/project-go-big). Figure’s own announcement scopes that result to navigation, not manipulation, which is a useful reminder that human-video-only transfer is proven for some skills and not yet for others.

A task like "make coffee" or "tidy the counter" is not one learnable unit — it is a sequence of shorter, more repeatable sub-actions: reach, grasp, lift, pour, place, release. Training data has to carry that structure as action segments, so a model can learn what a single sub-action looks like across many examples, then chain sub-actions into the full task. /glossary/task-decomposition covers this breakdown in more depth.

• Action segments mark where one sub-action starts and ends inside a longer demonstration clip
• Hand pose (2D or 3D, depending on the capture rig) records exactly how the hand approaches, closes and releases during each segment
• Object and scene labels identify what is being manipulated and where, so the same sub-action can be recognized across different objects
• Task and sub-task labels tie segments back to the overall goal, so a model can learn both the step and the sequence it belongs to

NVIDIA’s GR00T N1 describes a data pyramid for humanoid training: a broad base of web and human video data, a smaller middle layer of simulation and human-demonstration data closer to the robot’s embodiment, and a narrow top layer of real-robot teleoperation data (arxiv.org/abs/2503.14734). Each layer feeds a different stage of training, and each stage has a different tolerance for label noise and a different need for scale.

This pyramid structure is why a single everyday task usually draws on more than one data source. EgoScale’s approach (arxiv.org/abs/2602.16710, research.nvidia.com/labs/gear/egoscale) is a published example of using egocentric human video at the pretraining-to-mid-training boundary — see /guides/what-is-nvidia-egoscale for a plain-language breakdown.

Different household tasks stress different parts of a training corpus. A task with fine, precise contact (pouring, threading a cable) needs tighter hand-pose accuracy than a task that is mostly reach-and-place. A task that happens across many different objects (tidying a counter full of different items) needs more object and scene diversity than a task performed on one consistent object (closing a single type of drawer).

For how egocentric capture maps to a humanoid’s own head-and-wrist viewpoint specifically, see /guides/egocentric-data-for-humanoid-robots. For household tasks viewed from the training-data side more broadly, see /guides/training-data-for-household-robots.

Homes are not controlled environments. Lighting changes through the day, objects are cluttered and inconsistently placed, and people doing a task make mistakes and recover from them. A policy trained only on clean, successful demonstrations has no model of what a dropped object or a missed grasp looks like, and no example of how to recover — so it tends to fail the same way repeatedly instead of correcting.

• Failure and recovery segments — dropped objects, regrasps, mid-task correction — teach a model what going wrong looks like and how to respond
• Scene and lighting diversity prevents a policy from overfitting to one kitchen or one time of day
• Object diversity (the same task performed on different brands, shapes and conditions of object) is what lets a skill generalize past the exact items it was trained on

Teleoperated, real-robot data is the most precisely matched to a specific robot, but it is also the slowest and most constrained to collect — it needs the physical hardware, an operator, and time on that hardware per demonstration. Egocentric human video can be captured in parallel, at the specific household tasks and object sets a team needs, without waiting on robot availability. That is the practical reason human demonstration data is usually the base layer a team scales first, with real-robot data reserved for the final alignment and refinement stages.

A custom, commissioned collection lets a team specify exactly which everyday tasks, objects and scenes it is missing, rather than relying on whatever a public corpus happens to contain. See /custom-collection/humanoid-robots for how a custom humanoid collection is scoped and delivered, and /custom-collection/humanoid-pretraining-data for the pretraining-scale version of the same process.

• Which everyday tasks and sub-tasks matter for your deployment, broken down to the action-segment level
• Which training stage the data is for — broad pretraining tolerates more label noise than mid-training or post-training alignment data
• What object, scene and lighting diversity the task actually needs, based on where the task will be performed
• Whether you need bimanual 3D hand pose, or a simpler single-hand or 2D pose is sufficient for the task
• How many failure and recovery examples you want represented alongside successful demonstrations
• What format and sync the data needs to land in to drop into your existing training pipeline

/guides/humanoid-robot-data-collection-methods compares the data sources (egocentric human video, teleoperation, simulation) this page draws on in more detail, and /guides/how-much-data-does-a-humanoid-robot-need covers how to size a collection once the task list is set.