Glossary
What Is Robot See Robot Do (Imitating Articulated Object Manipulation with 4D Reconstruction)?
Last reviewed: November 2026
Short answer
Robot See Robot Do (RSRD) is the method in “Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction” (arXiv 2409.18121). From one monocular RGB human demonstration and a multi-view object scan, it reconstructs 3D part motion with 4D Differentiable Part Models, then plans bimanual robot motions that reproduce it. Firsthand is not affiliated.
01
What is Robot See Robot Do, and what problem does it address?
RSRD is named in Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction (arXiv 2409.18121) by Justin Kerr, Chung Min Kim, Mingxuan Wu, Brent Yi, Qianqian Wang, Ken Goldberg and Angjoo Kanazawa of UC Berkeley, presented at CoRL 2024 as an oral. The authors say humans can learn to manipulate new objects by watching others, and that giving robots the same ability would be a natural interface for specifying new behaviors. The project page says RSRD, like people, imitates object motion rather than hand motion, so one video works across object orientations and the human-robot embodiment gap.
02
Where does Firsthand fit next to Robot See Robot Do?
Firsthand is not affiliated with the RSRD authors. No named model, RSRD included, was trained on Firsthand data, and Firsthand does not claim to use any hardware or software named in the sources: they name a bimanual YuMi robot and software including GARField, DINOv2, HaMeR, jaxls and cuRobo. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Robot trajectories, retargeted actions, action labels, 4D reconstructions, object part motion trajectories, 3D Gaussian or other 3D object models, grasp plans, articulated part segmentations, simulation environments, digital twins, co-training mixtures, pretrained representations, embeddings, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.
03
How does RSRD imitate articulated object manipulation from one video?
Per the paper, RSRD has two stages. First, 4D Differentiable Part Models (4D-DPM) recover 3D part motion from a monocular video using differentiable rendering. This analysis-by-synthesis approach fits part-centric feature fields in an iterative optimization, which lets geometric regularizers constrain a problem that a single video leaves under-determined. The project page adds that GARField splits the object into parts, each part gets a trainable 6D pose optimized by gradient descent, and DINO features track the parts more robustly than photometric tracking.
Second, the robot replicates the object trajectories by planning bimanual arm motions that induce the demonstrated part motion. The authors represent a demonstration as part-centric trajectories so the robot copies the intended behavior within its own morphological limits instead of copying the hand. Per the project page, hand position only helps choose which part to move, and bimanual trajectories are optimized with jaxls and checked for collisions with cuRobo.
04
What inputs does RSRD need, and what does it need from the human side?
The abstract states the inputs: a single monocular RGB human demonstration and a single static multi-view object scan. The project page says these are smartphone-captured inputs, and the official GitHub repository describes the data as a multi-view scan in nerfstudio format plus an mp4 of the demonstration, with published data to reproduce the paper’s results.
The human side is therefore one video of the object moving, not a labelled trajectory set. Firsthand states 3D hand pose at 21 joints per hand; RSRD does not use finger contacts as grasps, and part motion, reconstructions and grasp plans are the buyer’s own steps, not Firsthand deliverables.
05
What did the authors report, and what are the limits?
The authors report evaluating 4D-DPM’s 3D tracking accuracy on ground-truth annotated 3D part trajectories, and physical execution on 9 objects with 10 trials each, 90 trials in total, on a bimanual YuMi robot. They report an average 87% success rate for each phase and a 60% end-to-end success rate across the 90 trials, achieved with feature fields distilled from large pretrained vision models and without task-specific training, fine-tuning, dataset collection or annotation. These are the authors’ figures; Firsthand has not verified them.
The project page lists limits: sensitivity to demonstration viewpoint, occlusions and reconstruction quality, hyper-parameter tuning, failures from incomplete part segmentations, and trouble with feature-less parts. It also says RSRD mimics motion from one video only and cannot scale to multiple demonstrations of the same object.
06
How does RSRD compare with related entries on this site?
The comparison table sets RSRD beside four site entries. The main difference is that RSRD is a reconstruct-and-plan method for articulated objects: per the abstract it uses no task-specific training, where the other entries describe learned policies or predictors trained on human video.
07
What would an RSRD-style approach need from custom collection?
Per the sources read, it needs one monocular RGB demonstration of an articulated object being manipulated and a multi-view scan of that object, plus the buyer’s own reconstruction, grasp planning and robot execution. The authors publish data and code, and the dataset licence is Not stated in the sources read. That is a note about format, not a compatibility claim.
No off-the-shelf dataset for this is stated here, so this is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.
08
Robot See Robot Do at a glance, as stated in the paper, project page and GitHub repository
| Item | What the sources state |
|---|---|
| Paper | Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction (arXiv 2409.18121) |
| Authors | Justin Kerr, Chung Min Kim, Mingxuan Wu, Brent Yi, Qianqian Wang, Ken Goldberg, Angjoo Kanazawa |
| Institution | UC Berkeley |
| Venue | CoRL 2024 (Oral), per the project page |
| Core idea | Recover 3D part motion from one video with 4D Differentiable Part Models, then plan robot motion to reproduce it |
| Human input | One monocular RGB demonstration video |
| Object input | One static multi-view object scan |
| Robot named | Bimanual YuMi robot |
| Code | Official GitHub repository linked from the project page |
| Dataset licence | Not stated in the sources read |
| Firsthand relationship | None |
10
RSRD components, as stated in the paper and project page
| Component | What the sources state |
|---|---|
| Reconstruction | 4D Differentiable Part Models recover 3D part motion from a monocular video by differentiable rendering |
| Part model | Part-centric feature fields, with part decomposition from GARField |
| Tracking features | DINO features, with a trainable 6D pose per part |
| Priors | Differentiable priors such as temporal smoothness and rigidity |
| Part selection | Detected hand position is used as a prior; finger contacts are not used as grasp points |
| Robot planning | Bimanual trajectories optimized with jaxls, with collision checks in cuRobo |
| Training | No task-specific training, fine-tuning, dataset collection or annotation, per the abstract |
11
RSRD beside related entries on the site
| Site entry | What the entry states | How RSRD differs (sources read) |
|---|---|---|
| OKAMI | Generates a manipulation plan from a single RGB-D human video and derives a humanoid policy using object-aware retargeting | RSRD takes a monocular RGB video plus an object scan and reconstructs articulated part motion for a bimanual robot |
| Zero-shot robot manipulation from passive human videos | Predicts human hand trajectories from a scene image, learned from internet videos | RSRD reconstructs object part motion from one demonstration instead of predicting hand trajectories |
| Motion Tracks (MT-π) | Uses 2D image trajectories as actions shared by human hands and robot end-effectors | RSRD works in 3D part trajectories and uses no task-specific training |
| Embodiment gap | The mismatch between a human body and a robot’s hardware | RSRD copies part motion rather than the hand, which the project page says helps across the gap |
| Cross-embodiment learning | Training one policy on data from multiple embodiments, including human demonstrations | RSRD trains no policy; it plans robot motion from one reconstructed demonstration |
12
What RSRD consumes beside what the Firsthand site states
| Aspect | RSRD (sources read) | What the Firsthand site states |
|---|---|---|
| Human video | One monocular RGB demonstration | Egocentric capture, custom collection to a written spec |
| Depth | Not a stated input for the demonstration | Depth derived from a calibrated stereo rig |
| Hand signals | Hand position as a prior for part selection | 3D hand pose at 21 joints per hand |
| Object scan | A static multi-view scan of the object | Not stated as a Firsthand deliverable |
| Language | Not described as an input | One task instruction per episode |
| Reconstruction output | 4D part motion recovered by 4D-DPM | 4D reconstructions and part trajectories are not Firsthand deliverables |
| Robot-side data | Planned bimanual motions on the authors’ robot | Robot trajectories, retargeted actions and grasp plans are not Firsthand deliverables |
13
Frequently asked questions
- What is Robot See Robot Do?
- Per the paper (arXiv 2409.18121), Robot See Robot Do (RSRD) imitates articulated object manipulation from a single monocular RGB human demonstration and a single static multi-view object scan. It is third-party research.
- What is 4D-DPM?
- The paper proposes 4D Differentiable Part Models (4D-DPM) to recover 3D part motion from a monocular video with differentiable rendering. RSRD then plans bimanual robot motion that induces the demonstrated part motion.
- What did the authors report for RSRD?
- They report 9 objects with 10 trials each on a bimanual YuMi robot, an average 87% success rate per phase and a 60% end-to-end success rate across 90 trials. Firsthand has not verified these figures.
- Does RSRD train a policy on human video?
- The abstract says RSRD uses feature fields distilled from large pretrained vision models, without task-specific training, fine-tuning, dataset collection or annotation. The project page says it mimics motion from one video only.
- Was RSRD trained on Firsthand data?
- No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. 4D reconstructions, part trajectories, grasp plans and robot trajectories are not Firsthand deliverables.
- Does Firsthand use the YuMi robot or the RSRD software?
- No claim is made. The sources name a bimanual YuMi robot and software including GARField, DINOv2, HaMeR, jaxls and cuRobo; Firsthand does not state that it uses any of them.
- How can I request a custom egocentric dataset for articulated object manipulation?
- Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
Sources
- Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction (arXiv 2409.18121)
- Robot See Robot Do (official project page, linked from the paper)
- kerrj/rsrd (official GitHub repository, linked from the project page)
- OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation (arXiv 2410.11792)
- Zero-Shot Robot Manipulation from Passive Human Videos (arXiv 2302.02011)
- Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning (arXiv 2501.06994)