Glossary
What Is an Affordance in Robot Learning?
Last reviewed: October 2026
Short answer
An affordance is an action an object makes possible for an agent, such as a handle that can be pulled or a mug that can be grasped. In robot learning it is often learned from egocentric human video as where a hand makes contact and how it moves next. Firsthand captures hand-object signals, not affordance labels.
01
What is an affordance in robot learning?
The idea comes from the psychologist J.J. Gibson, whom the VRB paper (arXiv 2304.08488) quotes: the meaning or value of a thing consists of what it affords. In robot learning an affordance is the actionable part of that idea: where on an object a robot should interact, and how to move once it has. The paper argues that naming objects is not enough, because a robot also needs to know where and how to manipulate them. It adds that an affordance only means something if an actor can execute it.
02
How do robots learn affordances from egocentric video?
In VRB, a visual affordance model is trained on videos of human interactions to estimate where and how in the scene a person is likely to interact. The paper represents an affordance as contact points plus a post-contact trajectory. To get training targets, it uses an off-the-shelf hand-object detector to find the frames where contact begins, then follows the hand afterward and compensates for head movement. Firsthand does not train or ship affordance models, maps or labels: Not stated. What it delivers are the signals this style of method starts from, recorded during capture rather than inferred afterward from footage: contact events, 3D hand pose and depth on one clock. Collection is fast, on demand and to a written skill spec.
03
What is an object affordance?
An object affordance is an affordance attached to a specific object or part of one: the handle of a lid, the front of a drawer, the edge of a pan. The VRB paper’s examples: an oven opens by pulling its handle down, a drawer is pulled outwards. Each has a place to make contact and a motion to make afterward.
The affordance depends on who acts. The paper notes that human morphology is very different from a robot’s, so it models a contact point and a trajectory rather than copying human movement, which keeps the representation independent of the robot body. Data from people also carries no robot actions, so the embodiment gap still has to be bridged.
04
Which captured signals relate to affordances?
Firsthand states no affordance field. It does state the signals that describe where a hand touched an object and how it moved around it: contact events, 3D hand pose with a per-joint visibility flag, action segments and, where a spec calls for it, tracked instance masks. The first table lists them with the formats the site gives, and marks affordance labels as Not stated.
All are recorded in one session from a head-mounted global-shutter camera and two wrist cameras on one PTP clock domain, so a contact event lines up with the hand joints and action segment around it.
05
How do affordances relate to grasping and contact events?
A grasp is something a hand does. An affordance is something an object offers. In VRB the training targets come from the first moment of contact and the hand motion after it, so a recorded contact event is the closest signal Firsthand states. That is a relationship, not an equivalence: a contact event marks one moment, an affordance a possibility.
The site does not define separate contact event types for grasps, placements or hand-offs, and grasp type is named on one collection page but is not a field in the episode schema. A buyer builds any such reading from contact_events and the surrounding action segment. The second table compares the terms.
06
How do you request custom egocentric hand-object data?
Name the objects, the grasps or interactions, and the environments your policy must handle, then list the signals you need. If your method needs affordance-style targets, say so: labels are Not stated, so extra annotation is scoped in the skill spec.
A small Spec Pilot tests the spec before the program scales. Where coverage is deep a first validated batch lands in a median of 48 hours from signed spec; where contributors must be recruited first, three to four weeks is realistic. Billing is per validated hour, delivery is RLDS, LeRobot, WebDataset or HDF5, and the license is buyer-owned.
07
Affordance-related fields Firsthand states, with formats
| Field | What it encodes | What the site states |
|---|---|---|
| Contact events | When and where a hand made or broke contact with an object | annotations/contact_events; shape Not stated |
| Hand joints | 3D hand pose across the sequence | annotations/hand_joints; float32 [T, 2, 21, 3], metres, head-camera frame; per-joint visibility in hand_visible |
| Action segments | The sub-action a contact belongs to | annotations/action_segments; {start_ns, end_ns, verb, noun} |
| Instance masks | Tracked object ids, where a spec calls for it | annotations/instance_masks; linked to an objects list and to contact events |
| Depth | Approach direction and distance | Active stereo, 848×480/30, metric, per-frame confidence map |
| Affordance labels | Where an object can be grasped or pulled | Not stated |
09
Affordance compared with hand-object interaction, contact event and grasp
| Term | What it is | Time scale | Firsthand relationship |
|---|---|---|---|
| Affordance | What an object offers an agent: where to interact and how to move after | A property of an object in a scene | No labels, maps or models stated |
| Hand-object interaction | What a hand does to an object: approach, grasp, manipulate, release | A whole episode | Recorded as hand pose, contact, grasp events and action segments |
| Contact event | A labeled moment a hand made or broke contact | Moments: onset and release | Delivered as contact_events |
| Grasp | The hand closing around an object | A span inside an action segment | Grasp events listed with contact onsets; grasp type is not a schema field |
10
What the site states and what is Not stated
| Detail | Status |
|---|---|
| Affordance labels, maps or models | Not stated; Firsthand ships none |
| Object pose (6-DoF) | Not stated |
| Force or tactile data | Not stated; contact events are labels, not force readings |
| Benchmark scores or accuracy for affordance learning | Not stated |
| Robot results trained on this data | Not stated |
11
Frequently asked questions
- What is an affordance in robot learning?
- What an object lets an agent do, such as where to grasp it or which way to pull. Robots can learn affordances from human video as contact points and post-contact motion.
- Can robots learn affordances from egocentric video?
- Yes. The VRB paper (arXiv 2304.08488) trains an affordance model on videos of human interactions, using contact points and post-contact trajectories.
- Does Firsthand deliver affordance labels?
- No. Affordance labels, maps and models are Not stated. Firsthand delivers contact events, 3D hand pose, action segments and depth.
- How is an affordance different from a contact event?
- A contact event is a labeled moment a hand made or broke contact. An affordance describes what an object offers, including where to touch and how to move.
- Do human videos give a robot affordances directly?
- Not on their own. Human data carries no robot actions, so the embodiment gap still has to be bridged.
- How can I order custom egocentric hand-object data quickly?
- Request a custom dataset with your objects, interactions and signals. Collection is on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms