Now recruiting capture teams worldwide. Get paid to record first-person video of real work.

Apply to capture

Glossary

What Is HuDOR (Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards)?

HuDOR fine-tunes a robot hand policy online, using rewards from object points tracked in one human video. Third-party; Firsthand is not affiliated.

Last reviewed: November 2026

Short answer

HuDOR is the technique in “Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards” (arXiv 2410.23289). Given a single human video and hand pose trajectory, it replays the motion on a multi-fingered robot hand, then fine-tunes the policy online with a reward that compares tracked object points in the human and robot videos. Firsthand is not affiliated.

01

What is HuDOR, and what problem does it address?

HuDOR is named in Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards (arXiv 2410.23289, October 2024) by Irmak Guzey, Yinlong Dai, Georgy Savva, Raunaq Bhirangi and Lerrel Pinto of New York University. The project page writes the name HuDOR; the paper sets it in capitals. The authors note that learning from human video has progressed for two-fingered grippers but remains hard for multi-fingered hands, because a policy trained on human hands may not transfer to a robot hand with a different morphology, and teleoperating many-jointed hands is hard. Their answer is a reward computed from object motion rather than hand appearance. A publication venue is Not stated in the sources read. See also the site’s embodiment gap, ViViDex, DexVIP and DVD entries.

02

Where does Firsthand fit next to HuDOR?

HuDOR is third-party research. Firsthand is not affiliated with the authors, and no named model, HuDOR included, was trained on Firsthand data. Firsthand does not claim to use any hardware or software named in the paper: a Kinova JACO arm, an Allegro hand, RealSense cameras, a Meta Quest 3 headset, langSAM, GroundingDINO, Co-Tracker or DrQv2. The site states custom egocentric collection to a written spec, fast and on demand, delivering first-person human video with 3D hand pose at 21 joints per hand and a per-joint visibility flag, calibrated stereo depth and 6-DoF IMUs at 200 Hz, in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Tracked object points, object trajectories, retargeted fingertip or robot hand poses, reward functions, robot data, trained policies, trajectory replays and synthetic data are not Firsthand deliverables.

03

How does HuDOR turn one human video into a first robot attempt?

Per the paper, a human video and a hand pose trajectory are recorded in the same scene as the robot, using a VR headset for hand pose estimates and RGB-D cameras for images. ArUco markers define a shared frame, so human fingertip positions can be expressed in the robot base frame, and a custom inverse kinematics module drives the full arm and hand toward those fingertip targets. The data are aligned on timestamps and subsampled to 5 Hz.

The authors state that replaying the retargeted fingertips mostly does not solve the task, because of morphology differences and hand pose estimation error. The replay is the starting point, not the result.

04

How does the object-oriented reward work?

The first frame is segmented with a language-grounded Segment-Anything model (langSAM, which uses GroundingDINO), giving points on the object. Co-Tracker, a transformer-based point tracker, follows them through the human video and through each robot rollout. Object motion is the mean displacement of the tracked points, and the reward at each step is the negative root mean squared error between robot and human object motion. For the music box task the authors add a mean rotation term.

A residual policy is trained on top of the replay with DrQv2, exploring a chosen subset of action dimensions with Ornstein-Uhlenbeck noise. Because the reward looks at the object and not the hand, it does not depend on the human and robot hands looking alike.

05

What did the authors report, and what are the limits?

The authors report that, from one human video, their four-fingered Allegro hand learns tasks such as opening a music box in about an hour of online interaction. They test four tasks (bread picking, card sliding, music box opening and paper sliding) and report an average 4× improvement over baselines, including offline behaviour cloning, a point cloud baseline, and image and point optimal-transport rewards. These are the authors’ figures; Firsthand has not verified them.

Stated limits: the framework works only with in-scene human videos, exploration needs prior knowledge of which action dimensions to explore, and there is no retry within an episode. The authors report generalization to new objects and larger areas as partial.

06

How does HuDOR compare with related entries on this site?

The table sets HuDOR beside four site entries. HuDOR’s distinguishing choice is to take its reward from tracked object motion while the robot practises online from a single video.

07

What would a HuDOR-style approach need from custom collection?

Per the paper, it needs one in-scene human video with a hand pose trajectory per task, plus a multi-fingered robot, cameras, calibration, a point-tracking model and a reinforcement learning stack, all the buyer’s own. The paper does not describe egocentric capture and lists in-scene video as a limit, so a buyer would need to judge whether a capture viewpoint suits their setup. Firsthand covers the human video side only.

This is custom collection: request on-demand collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

08

HuDOR at a glance, as stated in the paper and project page

HuDOR at a glance, as stated in the paper and project page
ItemWhat the sources state
PaperBridging the Human to Robot Dexterity Gap through Object-Oriented Rewards (arXiv 2410.23289)
AuthorsIrmak Guzey, Yinlong Dai, Georgy Savva, Raunaq Bhirangi, Lerrel Pinto (New York University)
InputA single in-scene human video plus a hand pose trajectory
RewardNegative root mean squared error between tracked object motion in the human video and the robot rollout
Point trackinglangSAM (GroundingDINO and SAM) for the first-frame mask, Co-Tracker for tracks
Policy learningResidual policy trained with DrQv2 on top of a retargeted fingertip replay
Robot namedKinova JACO 6-DOF arm with a 16-DOF four-fingered Allegro hand
Human capture namedMeta Quest 3 VR headset for hand pose, RealSense RGB-D cameras for images
TasksBread picking, card sliding, music box opening, paper sliding
VenueNot stated in the sources read
Code and data licenceNot stated in the sources read (the abstract says code and videos are on the project website)
Firsthand relationshipNone

10

HuDOR beside related entries on the site

HuDOR beside related entries on the site
Site entryWhat the entry statesHow HuDOR differs (sources read)
ViViDexExtracts reference hand and object trajectories from human videos, trains state-based reinforcement learning policies guided by them, then distills them into a vision-based policyHuDOR fine-tunes online on the real robot from one in-scene video, with the reward taken from tracked object points
DexVIPUses 3D human hand poses from in-the-wild YouTube video as a prior in a reinforcement learning rewardHuDOR’s reward comes from object motion, not a hand pose prior, and it records its own video
DVDTrains a discriminator that judges whether two videos show the same task and uses the score as a robot rewardHuDOR trains no discriminator; it compares tracked object motion directly
Point PolicyLearns robot policies exclusively from offline human demonstration videos, with key points describing objectsHuDOR also tracks object points, but uses them for a reward and adds online fine-tuning on the robot

11

What HuDOR consumes beside what the Firsthand site states

What HuDOR consumes beside what the Firsthand site states
AspectHuDOR (sources read)What the Firsthand site states
Human videoOne in-scene video per task, with the demonstrator in the same scene as the robotEgocentric capture, custom collection to a written spec
Hand poseFingertip positions from VR headset hand pose detectors3D hand pose at 21 joints per hand with a per-joint visibility flag
DepthRealSense RGB-D cameras positioned around the tableDepth derived from a calibrated stereo rig
Object tracksComputed by the method with langSAM and Co-TrackerNot a Firsthand deliverable
Robot-side dataOnline interaction, up to about an hour per task per the authorsRobot data is not a Firsthand deliverable
LanguageText prompts to segment the objectOne task instruction per episode

12

Frequently asked questions

What is HuDOR?
Per the paper (arXiv 2410.23289), HuDOR is a technique that fine-tunes a multi-fingered robot hand policy online, with a reward computed from object points tracked in a single human video. It is third-party research.
What are object-oriented rewards?
They are rewards computed from how the manipulated object moves, not from how the hand looks. HuDOR tracks points on the object in the human video and in each robot rollout, then rewards the robot for matching that object motion.
Does HuDOR need robot teleoperation data?
Per the authors, no. It starts from a retargeted replay of the human fingertip trajectory and improves it through online reinforcement learning on the robot, from one human video per task.
What did the authors report for HuDOR?
They report learning tasks such as opening a music box in about an hour of online interaction on four tasks, with an average 4× improvement over baselines. Firsthand has not verified these figures.
Was HuDOR trained on Firsthand data?
No. Firsthand is not affiliated with the authors and no named model was trained on Firsthand data. Tracked points, reward functions, robot data and trained policies are not Firsthand deliverables.
What human video data does Firsthand deliver?
Egocentric human video collected to a written spec, with calibrated stereo depth, 3D hand pose at 21 joints per hand, 6-DoF IMUs at 200 Hz and one task instruction per episode, in RLDS, LeRobot format or WebDataset.
How can I request a fast, on-demand custom egocentric dataset for this research?
Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.