Glossary
What Is Egocentric Action Recognition?
Last reviewed: October 2026
Short answer
Egocentric action recognition is the task of classifying what a camera wearer is doing from first-person video, usually assigning one action label to a whole clip. Firsthand does not ship recognition models. It delivers the labeled data they train on: timestamped action segments with a verb and noun, collected fast, on demand, to a written spec.
01
What is egocentric action recognition?
Egocentric action recognition is a video-understanding task: given first-person footage, decide which action the camera wearer is performing. In the common setup the whole clip gets one label, so a model learns what a clip is about and nothing about where one sub-action ends and the next begins. The Ego4D paper (arXiv 2110.07058) lists activity recognition among the challenges egocentric video offers, alongside anticipation, hand detection and human-object interaction. Firsthand trains and ships no recognition models and states no benchmark scores: Not stated. It supplies labeled first-person data that a team can use to build a recognizer or a policy.
02
How are action segments captured and labeled in egocentric video?
Firsthand delivers action_segments inside each episode as {start_ns, end_ns, verb, noun} entries in time order, on nanosecond timestamps aligned to the same PTP clock as video, depth and hand pose. The site describes the vocabulary as 97 verbs and 512 nouns, with start and end to the frame, human-reviewed. A segment such as “grasp mug” is one verb plus one noun. Because every segment has its own boundaries, one clip can hold several separately timed actions. Collection is fast, on demand and to a written spec, delivered in RLDS, LeRobot, WebDataset or HDF5.
03
How does first-person action recognition differ from third-person?
Third-person recognition works on footage from a camera that watches the scene. The Ego4D paper contrasts its first-person data with Internet datasets that capture “brief, isolated moments in time” from a third-person view, and says pre-training egocentric video models with third-person data suffers from a sizeable domain mismatch.
First-person video puts the wearer’s hands entering the frame from below, occlusion during a grasp and motion blur during a reach into the data. That is the view a robot with a head camera sees. Firsthand captures first-person video from a head-mounted global-shutter camera plus two wrist cameras. Whether it supplies any third-person footage is Not stated.
04
What labels does an action recognition dataset need?
A recognition dataset needs a fixed set of action classes and a label for each clip or span. Firsthand does not state a class list, a clip-level class field, train and test splits or class balance. Those are Not stated and are set per program in the spec.
What the site does state is the segment label: a start and end timestamp with a verb and a noun, human-reviewed. A buyer can add its own clip-level tag on top. The first table lists the stated fields. An episode is accepted only if RMS reprojection stays under 0.28 px.
05
How does action recognition relate to robot learning?
Recognition tells a model what happened. A robot policy also has to learn how. The site says action recognition typically needs only a clip-level label, while robot and embodied AI training also needs synchronized hand pose and action-segment boundaries, so a policy can learn the motion and not just classify the clip.
So the same episode carries hand_joints, contact_events and action_segments together. The Ego4D paper makes a related point about its Hands and Objects benchmark: it is similar to action recognition, but one state change, such as cutting, can appear in many object, tool and action combinations. Robot results from Firsthand data are Not stated.
06
How do you request a custom egocentric action dataset?
Write down the actions you need recognised, the setting, the verb and noun vocabulary, and a delivery format: RLDS, LeRobot, WebDataset, HDF5 or Rerun RRD. A custom collection starts from a written skill spec, typically proven on a smaller Spec Pilot, with a first validated batch in a median of 48 hours where coverage is deep. Anything marked Not stated below is set per program in the spec.
07
Action-related fields Firsthand states, with delivery names and formats
| Field | What it encodes | Delivery field | Stated shape |
|---|---|---|---|
| Action segments | One labeled sub-action per entry, in time order | annotations/action_segments | {start_ns, end_ns, verb, noun}; 97 verbs, 512 nouns |
| Hand joints | 3D hand pose across the sequence | annotations/hand_joints | float32 [T, 2, 21, 3], metres, head-camera frame |
| Joint visibility | Whether each joint is visible | annotations/hand_visible | Per joint; shape Not stated |
| Contact events | Contact and release moments | annotations/contact_events | Shape Not stated |
| Timestamps | Per-stream time on one clock | *_timestamp_ns | int64 nanoseconds, PTP domain |
| Clip-level class label | One action label for a whole clip | Not stated | Not stated |
09
Action recognition compared with related terms
| Term | What it is | Granularity | Firsthand relationship |
|---|---|---|---|
| Action recognition | Classifying what the wearer is doing, typically one label per clip | Whole clip | No model or score delivered; Firsthand supplies the labeled data |
| Action segment | Labeled interval of one sub-action | Interval inside an episode | Delivered as {start_ns, end_ns, verb, noun} |
| Action labels | Data saying what action was taken, such as hand pose or a segment | The label set | Captured at recording, not inferred afterward |
| Narration or caption | Free-text description of what happened | Free text | Not the delivered label; segments use a fixed verb-noun vocabulary |
10
What the site states and what is Not stated
| Detail | Stated on the site | Status |
|---|---|---|
| Class list or taxonomy for recognition | No | Not stated; set per program |
| Clip-level class label field | No | Not stated |
| Train and test splits | No | Not stated |
| Recognition model or accuracy | No | Not stated; Firsthand trains and ships no recognition models |
| Segments or hours per action | No | Not stated |
| Robot results from this data | No | Not stated |
11
Frequently asked questions
- What is egocentric action recognition?
- Classifying what a camera wearer is doing from first-person video, usually with one action label for a whole clip.
- Is action recognition the same as action segmentation?
- No. Recognition classifies an entire clip with one label. Segmentation marks the start and end timestamp of each sub-action, so one clip can hold several separately timed segments.
- What labels does Firsthand deliver for egocentric action data?
- Action segments with start and end timestamps, a verb and a noun, alongside 3D hand pose, joint visibility and contact events, all on one clock inside each episode.
- Does Firsthand sell an action recognition model?
- No. Firsthand delivers labeled first-person data, not models. Recognition accuracy and benchmark scores are Not stated.
- How many action classes does the dataset have?
- The site describes a segment vocabulary of 97 verbs and 512 nouns. A class list for a recognition task is Not stated and is set per program in the spec.
- Why does a robot need more than a clip-level label?
- A policy has to learn the motion, not just name the clip. It needs synchronized hand pose and action-segment boundaries, which a single clip-level label does not carry.
- How can I order a custom egocentric action dataset quickly?
- Request a custom dataset with your actions, vocabulary and delivery format. Collection is on demand against a written spec, with a first validated batch in a median of 48 hours where coverage is deep.