What Is SWIM (Structured World Models from Human Videos)?

SWIM is named in Structured World Models from Human Videos (arXiv 2308.10901) by Russell Mendonca, Shikhar Bahl and Deepak Pathak of Carnegie Mellon University, presented at Robotics: Science and Systems 2023 in Daegu. The paper expands the acronym as Structured World Models for Intentionality. The authors want robots to learn manipulation skills from only a handful of real-world interaction trajectories, and they argue that internet-scale human video is the way to get there. The obstacle they name is the morphology gap: a human arm and a robot arm send very different low-level signals, so a model of low-level joint actions does not transfer. Their answer is a high-level action space, a grasp location plus post-grasp waypoints, that both a human hand and a robot gripper can share. For the broader idea, see the site’s affordance and world model entries.

Firsthand is not affiliated with the SWIM authors and makes no claim that any model, SWIM included, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Affordance labels, contact-point labels, post-contact trajectories, world-model training, action labels, pretrained representations, embeddings, reward labels, robot trajectories, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, SWIM has three steps. First, pre-train a world model on human video. Second, fine-tune it on robot data that was collected without any task reward. Third, deploy it by planning toward a goal image. The authors call the action space structured because it is built from visual affordances rather than raw joint commands. Those affordances are a hand trajectory in image space and object locations, and from them the authors derive a grasp pixel and a post-grasp pixel. The project page describes this as a shared human-robot action space that specifies an interaction point and a post-contact trajectory, following the authors’ earlier VRB work.

The authors run the 100 Days of Hands detector on each frame to locate the hand and the object, find where the hand makes contact, and sample a later hand position as the post-grasp point. They then train an affordance model on the first frame of each clip, a UNet-style network with a ResNet18 encoder that outputs grasp keypoints and a post-grasp hand trajectory. Because a 2D image has no depth, the paper uses a robot’s depth camera to lift the chosen pixel into 3D. For human clips, where depth and gripper rotation are missing, those two values are sampled at random.

SWIM also uses a hybrid action space. A mode index chooses between executing an image-space affordance and an arbitrary Cartesian end-effector move, so the robot can follow the structure from human video and still make fine adjustments. Human video training always uses the affordance mode.

Per the paper, the human pre-training data comes from Epic-Kitchens, which the authors describe as around 50k egocentric videos of people performing manipulation tasks in kitchens. They cut it into short clips of about three seconds, and about 55K such clips train the world model. The affordance labels are therefore inferred by software from ordinary video, not recorded by a sensor, so detector errors in cluttered or occluded scenes would pass into the training signal. The paper itself notes that depth and 3D information are not available for this data.

The world model follows the Dreamer line of work, with an encoder, a latent dynamics model and decoders, and the authors swap in an NVAE encoder to handle high-dimensional images. For fine-tuning, the robot samples actions from the affordance model and executes them to build a dataset, which needs no reward or goal. At deployment the task is given as a goal image. The system gathers action proposals from the affordance model and from a Gaussian mixture fitted to the best earlier trajectories, then picks among them with the cross-entropy method through the world model.

The authors test six real-world tasks on two robots: a Franka arm in a play kitchen (open a cabinet, pick a toy vegetable, lift a knife) and a Hello Robot Stretch RE-1 in real kitchens (open a dishwasher, pull a drawer, open a garbage can). For each task they collect 25 or 50 iterations of random actions, which they say takes about 30 minutes, and they use an Intel RealSense D415 for RGBD. In their Table I, success is measured over 25 trials, with SWIM averaging about 80% with joint models and 75% with single-task models, against about 40% for the next best approaches. The project page says pre-training raises joint-model average success from 20 to 80 percent. These are the authors’ figures on their setups; Firsthand has not verified them.

The caveats are in the paper. The authors say the approach currently covers quasi-static setups, and that exploring other action parameterisations and manipulation types is future work. Their results are specific to the tasks, robots and kitchens they describe.

The first table sets SWIM beside four site entries, summarising what those entries state. In short, VRB predicts contact points and a trajectory as its output, HRP uses affordance labels to fine-tune a visual encoder, R3M trains a frozen representation, and SWIM uses affordances as the action space for a world model that is then fine-tuned on robot data. SWIM also uses an R3M-derived feature distance in baselines and for goal distance, per the paper.

Per the paper, it needs human video with hand-object interaction, a hand-object detector, a depth-equipped robot, and unsupervised robot interaction data for fine-tuning. Firsthand states real human egocentric capture, depth from a calibrated stereo rig, 3D hand pose and contact events (see the affordance entry), with one task instruction per episode. That is a note about format, not a compatibility claim: any affordance labels or world-model training would be work the buyer runs.

Epic-Kitchens is accessed under its own terms; see the site’s Epic-Kitchens entry and confirm the current licence. For real human demonstrations that public video does not cover, request on-demand custom collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

What is SWIM (Structured World Models from Human Videos)?

A published method (arXiv 2308.10901, RSS 2023) that pre-trains a world model on human video in an affordance-based action space, fine-tunes it on a small amount of robot data and plans toward a goal image. It is third-party research.

What is an affordance-space world model learned from human video?

Per the paper, a world model whose actions are visual affordances, meaning a grasp location and post-grasp waypoints found in human video, so human clips and robot deployment share one action space.

How does SWIM enable few-shot robot learning from egocentric video?

The authors pre-train on Epic-Kitchens egocentric clips, then fine-tune on robot interaction data collected without task rewards, which they report takes about 30 minutes per setup.

How does SWIM differ from VRB and HRP?

VRB predicts contact points and a trajectory, and HRP fine-tunes a visual encoder on affordance labels. SWIM uses affordances as the action space of a world model that plans toward goal images.

Was SWIM trained on Firsthand data?

No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Affordance labels, world-model training and pretrained representations are not Firsthand deliverables.

How can I request a custom egocentric dataset for world-model research?

Request a custom dataset with your tasks, environments and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.