Glossary
What Is an Episode in Robot Learning?
Last reviewed: 2026-09-28
Short answer
An episode is one recorded demonstration of a skill, packaged as a single unit with four groups: meta (episode_id, skill_id, duration, license), streams (head/wrist video, depth, IMU), annotations (hand pose, action segments, masks) and calibration (intrinsics, extrinsics). Every robot-learning delivery format — RLDS, LeRobot, WebDataset, HDF5 — is a different projection of the same episode.
01
Why it matters
Policies train on sequences of steps within a demonstration, so the episode — not the raw video file — is the unit a dataloader actually consumes. A clean episode boundary with synchronized streams and labeled action segments is what makes a demonstration usable for imitation learning.
02
How Firsthand handles it
Firsthand structures every recording as an episode with the canonical schema (meta.json, streams/, annotations/, calibration/), delivered as the same episode content projected into RLDS, LeRobot format, WebDataset or HDF5.
03
One episode’s four groups, and how each format represents them
| Episode group | Contents | RLDS / LeRobot / WebDataset / HDF5 |
|---|---|---|
| meta | episode_id, skill_id, site_id, duration_ns, license | Episode-level metadata fields in each format |
| streams | head_rgb, wrist_l/r, depth_mm, imu_*, timestamps | Per-step observation tensors, aligned by timestamp |
| annotations | hand_joints, action_segments, instance_masks | Per-step or per-segment labels alongside observations |
| calibration | intrinsics, extrinsics (SE3), rms_reproj_px | Episode-level calibration metadata, not per-step |
05
Frequently asked questions
- What is an episode in robot learning?
- One recorded demonstration of a skill, structured as a self-contained unit with metadata, sensor streams, annotations and calibration — the basic unit that robot-learning datasets are organized into.
- What is inside a Firsthand episode?
- Four groups: meta.json (episode_id, skill_id, site_id, duration_ns, license), streams/ (head and wrist video, depth, IMU, each with nanosecond timestamps), annotations/ (hand joints, action segments, instance masks) and calibration/ (intrinsics, extrinsics, reprojection error).
- Is an episode the same thing across RLDS, LeRobot, WebDataset and HDF5?
- Yes — the episode content is the same; only the container differs. Firsthand’s canonical episode schema is the layout every delivery format projects from, so the same meta, streams, annotations and calibration groups appear in each format.
- How is an episode different from a raw video clip?
- A raw clip is just pixels. An episode bundles synchronized multimodal streams, structured annotations like action segments and hand pose, and calibration metadata around that video — everything a training pipeline needs, not just footage.
- What are action segments inside an episode?
- Labeled time intervals marking one sub-action within an episode — a start and end timestamp with a verb and noun, such as “grasp mug” — that turn a continuous episode into supervised, temporally-localized units.