Glossary
What Is Occlusion in Egocentric Video?
Last reviewed: October 2026
Short answer
Occlusion in egocentric video is when something blocks the camera’s view of a hand or object, often the hand or arm itself during a grasp. Firsthand ships these frames uncut, marks occluded hand joints with a per-joint visibility flag, and rejects an episode if the manipulated object is out of frame for over 15 percent of it.
01
Why does hand occlusion matter for robot training data?
During a grasp the hand and arm often block the camera’s view of the contact point. A dataset that trims those moments teaches a cleaner world than a robot with a head camera will meet. One that keeps them unmarked leaves a training loop unable to tell a seen joint from a hidden one.
02
How does stereo capture and per-joint visibility handle occlusion?
Firsthand records occlusion rather than hiding it. Hand pose is 3D, 21 joints per hand, in metres in the head-camera frame, with a per-joint visibility flag (hand_visible) on every frame, so occluded joints can be masked or down-weighted. Depth comes from an active-stereo sensor. The site does not state that any stream recovers a hidden joint, and it publishes no occlusion rate.
03
What is occlusion in egocentric video?
Occlusion is when something sits between the camera and what it should record. In first-person video the blocker is usually the wearer: a hand passes behind an object mid-reach, an arm crosses the view, or a held object covers the fingers. It differs from an object leaving the frame, though the site’s rejection-criteria guide lists an object out of frame under its occlusion row.
04
How do head and wrist cameras change what gets occluded?
The head camera shares the wearer’s viewpoint, so the hand and arm frequently occlude the contact point during a grasp. A wrist camera moves with the hand and keeps the object and contact point in frame through the approach and grasp. Firsthand pairs a head-mounted 4K/60 camera with two wrist-mounted 1080p/60 cameras on one PTP clock. A method for filling a head-view gap from a wrist view is Not stated.
05
How are occluded frames flagged or rejected?
Flagging happens in the labels: hand_visible carries a per-joint visibility flag on every frame, so a policy can see when and how a grasp was occluded, and the occluded moment stays inside one action segment.
Rejection happens at ingest, where episodes are machine-checked then reviewed by a person. Two criteria concern view loss: the manipulated object out of frame for more than 15 percent of the episode, and more than 35 percent invalid depth pixels in the manipulation region. Rejected episodes go to the reject log, are recaptured at Firsthand’s cost and do not count as validated hours. A hand hidden mid-reach is flagged, not rejected, and a per-frame occlusion threshold is Not stated.
06
Is recorded visibility different from estimated hand pose?
The site states Firsthand records 3D hand pose at capture time rather than estimating it afterwards from 2D pixels. EgoScale (arXiv 2602.16710) applies off-the-shelf SLAM and hand-pose estimation pipelines to its in-the-wild video and says the estimates are noisy because collection is unconstrained. It does not discuss occlusion there, so this page attributes none.
07
How do you plan for occlusion in a skill spec?
Name the occlusion angles that matter in the skill spec before capture. Each becomes a coverage cell that is counted and checked, and counts toward validated hours. How much occlusion a program tolerates before an episode fails is Not stated by the site and belongs in the spec.
08
How do you request a custom dataset?
Custom egocentric data collection starts from a written skill spec, usually proved by a Spec Pilot before volume scales. Where coverage is deep, a signed spec can reach a validated first batch in a median of 48 hours; where contributors must be recruited first, the site gives three to four weeks. Send the task, occlusion angles, streams and reject bar, and confirm hand_visible is carried in your delivery format, since the RLDS format page does not itemise it.
09
Occlusion types in first-person capture and how the site’s stated capture and labels handle them
| Occlusion type | What the site states | Status |
|---|---|---|
| Hand passes behind an object mid-reach | Recorded uncut; hand_visible flags occluded joints per frame | Stated |
| Hand or arm blocks the contact point | Wrist cameras keep it in frame; contact_events marks contact made or broken | Stated |
| Some of a hand’s 21 joints hidden | Per-joint visibility flag; how hidden positions are filled is not described | Fill method Not stated |
| Manipulated object out of frame | Episode rejected and logged beyond 15 percent of it | Stated |
| One hand hidden in a two-handed task | Per-hand pose with visibility flags shows which hand is hidden | Stated |
| Wearer’s body out of frame | Named as a head-view limit; body visibility flag is stated for hands only | Body flag Not stated |
11
Delivered fields relevant to visibility, and their status
| Field or item | What the site states | Status |
|---|---|---|
| hand_joints | float32 [T, 2, 21, 3], metres, head-camera frame | Stated |
| hand_visible | Per-joint visibility flag in the annotations group | Stated |
| hand_visible value type | Binary or confidence | Not stated |
| hand_visible in RLDS | Not itemised; confirm on the Spec Pilot | Not stated |
| contact_events | When contact was made or broken | Stated |
| Occlusion rate, per-frame threshold | No figure given | Not stated |
| Hand pose accuracy | No figure given | Not stated |
12
Frequently asked questions
- What is occlusion in egocentric video?
- It is when something blocks the camera’s view of a hand or object, usually the wearer’s own hand or arm during a grasp.
- What are occluded frames in robot training data?
- Frames where part of a hand or object is hidden. Firsthand records them uncut and marks occluded hand joints with hand_visible.
- Does Firsthand remove occluded frames?
- No. A hand hidden mid-reach is recorded and flagged. An episode is rejected only on listed criteria, such as the object out of frame for more than 15 percent of it.
- Does stereo depth fix occlusion?
- The site does not state that it does. An episode with more than 35 percent invalid depth pixels in the manipulation region is rejected.
- Can I ask for specific occlusion cases in a custom dataset?
- Yes. Name the occlusion angles and your reject bar in the skill spec before capture.
- What does Firsthand not state about occlusion?
- No occlusion rate, hand pose accuracy figure, per-frame threshold, hand_visible value type, or method for filling hidden joints.
Related terms
Sources