Glossary

Egocentric AI & embodied data glossary

Plain-language definitions of the terms that come up when you evaluate egocentric video for embodied AI. Each entry leads with a one-paragraph answer, then explains why it matters and how we measure it.

3D Diffuser Actor3D Diffuser Actor is a published policy that denoises end-effector trajectories from 3D scene tokens and language. Firsthand is not affiliated.
3D-VLA3D-VLA is a published 3D generative world model that predicts goal images and point clouds to plan robot actions. Firsthand is not part of it.
Act3DAct3D is a published 3D feature field transformer for multi-task robot manipulation that computes 3D action maps. Firsthand is not affiliated.
Action chunkingA policy design where a robot model predicts a block of consecutive future actions at once instead of one action per step, as in GR00T N1 and EgoScale.
Action labelsData attached to a video or trajectory that says what action was taken, such as a hand pose, wrist trajectory, or action segment.
Action representationThe way a policy’s actions are encoded — e.g. a wrist-level pose from human video, or a latent action learned without one.
Action segmentA labeled time interval of one sub-action, with start/end timestamps and a verb-noun.
Action tokenizationAction tokenization turns continuous robot actions into discrete tokens a VLA can predict. How RT-2, OpenVLA and FAST do it, and which VLAs skip tokens.
Action-space alignmentAdapting a model’s human-derived action representation with a smaller amount of robot action data so human and robot actions become comparable.
AffordanceAn affordance is what an object lets an agent do, such as where to grasp or pull it. Robots can learn affordances from egocentric human video.
AgiBot World datasetAgiBot World is an open robot manipulation platform: 1M+ trajectories of dual-arm humanoid teleoperation across five real-world domains.
ALOHAALOHA is a published low-cost, open-source bimanual teleoperation system, paired with the ACT policy. Firsthand is not part of it.
Annotation guidelinesAnnotation guidelines are the written rules behind robot-training labels. See what Firsthand states about its spec and review, and what is Not stated.
AnonymizationMaking a person no longer identifiable from data by any reasonably likely means.
Any-point Trajectory ModelingATM is a published method that learns point tracks from action-free video to guide robot policies. Third-party research; Firsthand is not affiliated.
ARCTIC datasetARCTIC is a published motion-capture dataset of two hands manipulating articulated objects, with 3D hand and object meshes and contact information.
Aria Digital Twin datasetADT is a published egocentric dataset recorded with Aria glasses, with 3D object, eye-gaze and human-pose ground truth. Firsthand is not part of it.
Aria Everyday Activities datasetAEA is a published egocentric dataset recorded with Project Aria glasses. Firsthand is not part of it and does not use it.
Assembly101Assembly101 is a published multi-view video dataset of people assembling and disassembling toy vehicles, with action segments and 3D hand poses.
Batch deliveryBatch delivery ships a custom dataset as a series of validated batches, each checked against the spec, rather than as one final hand-off.
Behavioral cloningLearning a policy that maps observations to actions by directly copying demonstrations.
Being-H0Being-H0 is a published vision-language-action model pretrained on human videos by explicitly modeling hand motion, then adapted to robots.
Bimanual manipulationTasks using both hands together, often with coordinated, asymmetric roles.
BitVLABitVLA is a published 1-bit vision-language-action model with ternary weights, built for memory-constrained robots. Firsthand is not affiliated with it.
Body pose estimationRecovering 3D positions of body joints over time, alongside hand pose, for whole-body motion.
Bounding boxThe rectangle of pixel coordinates that marks where a detected object is.
BridgeData V2BridgeData V2 is an open robot manipulation dataset of 60,096 trajectories across 24 environments. Firsthand is not part of it.
BridgeVLABridgeVLA is a published 3D VLA model that projects 3D inputs to 2D images and predicts actions as 2D heatmaps. Firsthand is not affiliated.
CALVIN benchmarkCALVIN is a simulated benchmark for language-conditioned, long-horizon robot manipulation: four environments, 34 tasks, about 24 hours of play data.
Camera calibrationSolving a camera’s intrinsic and extrinsic parameters — lens distortion and 3D position — so pixels can be turned into accurate geometry.
Camera intrinsics and extrinsicsIntrinsics describe one camera’s lens and sensor; extrinsics give its pose relative to another frame. Stereo depth needs both.
Capture frame rateCapture frame rate is how many frames per second a camera records. Firsthand states 60 fps for head and wrist cameras and 30 fps for depth.
Capture sessionA capture session is one supervised recording block: a calibrated rig, a named skill, a real room, and episodes checked against the spec.
Charades-Ego datasetCharades-Ego is a published dataset of paired third- and first-person videos of scripted indoor activities, recorded by crowd workers. Firsthand is not part of it.
ChatVLAChatVLA is a published VLA that unifies multimodal understanding and robot control. Firsthand is not affiliated.
Co-trainingTraining one policy on multiple data sources at once — e.g. human video and robot demonstrations — rather than one after another.
Code as PoliciesCode as Policies is a published method where a code-writing language model writes robot policy programs. Third-party; Firsthand is not affiliated.
CogACT VLA modelCogACT is a VLA model whose diffusion transformer action module generates robot action sequences from a vision-language model’s cognition feature.
CogVLACogVLA is a published efficient VLA model that routes and prunes visual tokens by instruction. Third-party research; Firsthand is not affiliated.
ConRFTConRFT is a published reinforced fine-tuning method for VLA models using a consistency policy. Third-party; Firsthand is not affiliated.
Consent chainThe documented trail proving every participant granted the rights you need.
Contact eventA contact event is a labeled moment a hand or tool makes or breaks contact with an object in egocentric video, recorded with hand pose and depth.
CoT-VLACoT-VLA is a VLA that reasons with subgoal images before it acts. It is third-party research, not Firsthand’s.
CronusVLACronusVLA is a published framework that extends single-frame vision-language-action models to use multi-frame history. Firsthand is not affiliated.
Cross-embodiment learningTraining one robot policy or foundation model on data from multiple embodiments — different robots and human demonstrations — so skills transfer across bodies.
Cross-stream sync errorThe timing offset between simultaneously captured streams like video, depth and IMU.
CrossFormerCrossFormer is a published policy that controls arms, wheeled robots, quadcopters and quadrupeds with one set of weights. Firsthand is not affiliated.
Custom data collectionCommissioning new training data captured to your own written spec, instead of buying a dataset someone else already built.
Data cardA structured record of how a dataset was collected, consented and licensed.
Data collection as a serviceA managed service where one provider runs spec, contributors, capture, QA, consent, and delivery, and you receive a finished, owned dataset.
Data collection turnaround timeThe elapsed time from a signed spec to delivered, QA’d data, and how it differs from first-batch time, pilot timing and delivery cadence.
Data contributorA screened, consented, paid person who performs tasks on a capture rig against a written skill spec.
Data curationData curation for robot pretraining is selecting, filtering, validating, balancing and documenting data before it enters a training mix.
Data flywheelA loop where deployment reveals failures, failures define new data, and new data improves the model.
Data lineageData lineage is the traceable record of where AI training data came from and what happened to it. What Firsthand states, and what it does not.
Data mixtureThe blend of data sources and their proportions that go into a robot pretraining run.
Data moatA data moat is a lasting AI advantage from training data rivals cannot easily get or copy, such as exclusive, buyer-owned custom data.
Data processing agreement (DPA)A data processing agreement (DPA) is the contract governing how a processor handles personal data for a controller, such as people in training footage.
Data pyramidA way of organizing robot training data by volume and embodiment closeness — broad data at the base, a small real-robot layer at the top.
Data recollectionData recollection is re-capturing episodes that fail QA. See what Firsthand states about rejected episodes and which figures are Not stated.
Data residencyData residency is where a dataset is physically stored and processed. What Firsthand states about storage and transfer, and what it does not.
Data retention policyA data retention policy states what data a provider keeps, for how long, and when it is deleted or returned. What Firsthand states, and does not.
Data scaling lawA reported relationship between training data volume and model performance or loss.
Dataset acceptance criteriaAcceptance criteria are the written tests a custom dataset must pass before delivery. See which Firsthand states and which are Not stated.
Dataset coverageWhether a dataset fills every scene, task, object and condition a spec targets — not just a total hour count.
De-identificationIrreversibly removing personal identifiers like faces, plates and screens.
DeeR-VLADeeR-VLA is a published dynamic early-exit framework that lets a robot VLA run only as much of its multimodal LLM as each situation needs. Firsthand is not affiliated.
Depth dataPer-pixel distance from the camera, delivered as a metric-millimetre depth map alongside the video.
DexCapDexCap is a published portable hand-motion capture system and imitation method for dexterous robots. Firsthand is not part of it and does not use it.
DexGraspVLADexGraspVLA is a published hierarchical VLA framework for language-guided dexterous grasping. Firsthand is not affiliated with its authors.
DexMV (Imitation Learning for Dexterous Manipulation from Human Videos)DexMV turns human hand-object video into multi-finger robot-hand demonstrations for imitation learning. Third-party; Firsthand is not affiliated.
Dexterous manipulationFine-grained, multi-finger manipulation — regrasping, in-hand reorientation, precision pinches �� as opposed to a single wrist or gripper motion.
DexVLADexVLA is a VLA with a plug-in diffusion action expert for general robot control. It is third-party research, not Firsthand’s.
DexWild (Dexterous Human Interactions for In-the-Wild Robot Policies)DexWild is a published system that collects human hand data in the wild and co-trains robot policies on it. Firsthand is not affiliated.
DexYCB datasetDexYCB is a published benchmark dataset for capturing hand grasping of objects, with 3D hand pose and 6D object pose labels.
Diffusion policyA robot policy that generates actions by denoising random noise, conditioned on observations. The diffusion action head in GR00T N1 is one example.
Diffusion-VLADiffusion-VLA is a published VLA that pairs autoregressive reasoning with a diffusion action head. Firsthand is not affiliated with its authors.
Dita diffusion transformer policyDita is a published diffusion transformer policy that denoises robot action sequences with in-context conditioning. Firsthand is not part of it.
DPIA for AI training dataA DPIA is the GDPR Article 35 assessment a controller carries out before high-risk processing. What it means for AI training data that shows people.
DreamitateDreamitate is a published method: fine-tune a video model on human tool demos, generate a task video, track the tool, run it on a robot. Third-party.
DreamVLADreamVLA is a published vision-language-action model that forecasts compact world knowledge before acting. Firsthand is not affiliated with its authors.
DROID datasetDROID is an open robot manipulation dataset of 76k teleoperated trajectories across 564 scenes. Firsthand is not part of it.
Ego-Exo4DEgo-Exo4D is a multi-view dataset of skilled human activity filmed from first- and third-person cameras at once. Firsthand is not part of it.
Ego4DEgo4D is a large public egocentric video dataset and benchmark suite, accessed under a licence agreement. Firsthand is not part of it.
EgoBody datasetEgoBody is a published head-mounted dataset of 3D body shape and motion in social interactions. Firsthand is not part of it.
Egocentric action recognitionClassifying what a camera wearer is doing from first-person video, usually one label per clip; Firsthand delivers the labeled segment data, not a model.
Egocentric data collectionCapturing first-person video and synchronized streams from the wearer’s point of view for AI and robot training.
Egocentric pretrainingPretraining a robot foundation model on large-scale first-person human video with action labels, before mid-training and post-training on robot data.
Egocentric SLAMSLAM recovers camera motion from egocentric video. EgoScale pairs it with hand-pose estimation to derive action labels without a purpose-built rig.
Egocentric videoFirst-person video recorded from a camera worn by the person doing the task.
EgoDex datasetEgoDex is a published large-scale dataset of egocentric dexterous manipulation video with 3D hand and finger tracking. Firsthand is not part of it.
EgoLife datasetEgoLife is a published week-long egocentric daily-life dataset, benchmark and assistant system. Firsthand is not affiliated with its authors.
EgoMimicEgoMimic is a published framework that co-trains robot manipulation policies on egocentric human video with 3D hand tracking and on robot data.
EgoSchema benchmarkEgoSchema is a published benchmark of five-option questions on three-minute egocentric clips from Ego4D. Firsthand is not affiliated with its authors.
EgoVLAEgoVLA is a published vision-language-action model pretrained on egocentric human video, then fine-tuned on a few robot demonstrations.
EgoZero (Robot Learning from Smart Glasses)EgoZero is a published system that learns robot policies from smart-glasses human demonstrations and zero robot data. Firsthand is not affiliated.
EGTEA Gaze+EGTEA Gaze+ is a public first-person cooking dataset with gaze tracking, action labels and hand masks. Firsthand is not affiliated with it.
Embodied AIAI that perceives and acts through a physical or simulated body.
Embodied chain-of-thoughtEmbodied chain-of-thought (ECoT) trains a VLA to reason in text about plans, sub-tasks and object locations before each robot action.
Embodiment gapThe mismatch between a human body and a robot’s hardware that keeps human demonstration data from transferring to a robot policy directly.
End effectorAn end effector is the gripper, hand or tool at the end of a robot arm; its pose is position plus orientation. Firsthand captures human hands, not robots.
EO-1EO-1 is a published open embodied foundation model for robot control, pretrained on interleaved vision-text-action data. Firsthand is not affiliated.
EPIC-KITCHENSEPIC-KITCHENS is a public egocentric video dataset of unscripted kitchen activity with timed verb and noun labels. Firsthand is not part of it.
Epipolar geometryThe geometry linking two calibrated cameras: a point in one image must lie on a matching epipolar line in the other, which narrows stereo matching.
EpisodeOne recorded demonstration of a skill — meta, streams, annotations and calibration bundled together.
Face detectionLocating faces in an image and returning a box and confidence for each.
Failure-case coverageDeliberately including the hard cases: transit, occlusion, low light, recovery.
FAST action tokenizerFAST is a frequency-space action tokenizer for VLA models: a DCT plus byte-pair encoding turns action chunks into compact tokens, as used by pi0-FAST.
Fast-in-Slow (FiS-VLA)Fast-in-Slow (FiS-VLA) is a published dual-system VLA that embeds a fast execution module inside a slow VLM. Firsthand is not affiliated with its authors.
Field of viewField of view is the angle of scene a camera sees. See what it means for egocentric video and which angles Firsthand states for its rig.
First-person videoVideo recorded from the performer’s own viewpoint, usually a head-mounted camera — the plain-language term for egocentric video.
Flow matchingThe training objective behind the action head in GR00T N1, EgoScale and EgoVerse: it turns random noise into an action chunk in a few steps.
FLOWER VLAFLOWER is a published 950M-parameter vision-language-action flow policy built for efficiency. Firsthand is not affiliated.
FPHA datasetFPHA is a published first-person RGB-D dataset of hand actions with 3D hand pose annotations. Firsthand is not part of it.
Frontier AI labA frontier AI lab trains general-purpose AI models at the leading edge of capability; in robotics it commissions custom first-person data.
Gato (A Generalist Agent)Gato is a published generalist agent: one transformer for games, text and robot control. Third-party; Firsthand is not affiliated.
Gaze trackingGaze tracking records where a wearer was looking in first-person video. Firsthand lists a gaze_target field; eye-tracker hardware is set per spec.
Gemini RoboticsGemini Robotics is Google DeepMind’s Gemini 2.0-based robotics family: Gemini Robotics-ER for embodied reasoning and Gemini Robotics, a VLA model.
Gen2ActGen2Act is a published method that generates a human video of a task and conditions a robot policy on it. Third-party; Firsthand is not affiliated.
General FlowGeneral Flow is a published affordance: predicted future 3D trajectories of points on an object, learned from RGBD human video. Third-party research; Firsthand is not affiliated.
GeoVLAGeoVLA is a published VLA framework that adds point clouds from depth maps to image-language inputs. Third-party; Firsthand is not affiliated.
GigaHands datasetGigaHands is a published dataset of bimanual hand activities with 3D hand and object annotations and text. Firsthand is not part of it.
Global-shutter cameraA camera that exposes every pixel simultaneously, avoiding rolling-shutter skew.
GR-1GR-1 is a published GPT-style model that learns robot manipulation after video generative pretraining. Firsthand is not part of it.
GR-2GR-2 is a published generative video-language-action model for robot manipulation, pretrained on Internet video. Firsthand is not part of it.
GR-3GR-3 is a published large-scale vision-language-action model for generalist robot policies. Firsthand is not part of it.
GR00T N1GR00T N1 is NVIDIA’s open foundation model for humanoid robots, a vision-language-action model. Firsthand is not part of it.
GRAPEGRAPE is a published method that aligns VLA robot policies to preferences at the trajectory level. Third-party; Firsthand is not affiliated.
Grasp typeA grasp type is a named hand configuration for holding an object. See the GRASP Taxonomy and what Firsthand does and does not deliver.
GraspVLAGraspVLA is a published grasping VLA model pretrained on the SynGrasp-1B synthetic dataset. Firsthand is not affiliated with its authors.
H2O datasetH2O is a published egocentric dataset of two hands manipulating objects, with 3D hand and 6D object poses, multi-view RGB-D and action labels.
HaMeR hand mesh recoveryHaMeR is a published transformer model that reconstructs a 3D hand mesh from a single image, with an accompanying 2D-keypoint dataset called HInt.
HAMSTERHAMSTER is a published hierarchical VLA: a fine-tuned VLM predicts a 2D path that guides a low-level 3D policy. Firsthand is not affiliated.
Hand pose estimationRecovering 3D positions of hand and finger joints over time.
Hand-eye calibrationSolving the fixed transform between a mounted camera and the frame it rides on, and how head and wrist extrinsics play that role in egocentric data.
Hand-object interaction (HOI)Hand-object interaction (HOI) data records how hands approach, grasp, manipulate and release objects: hand pose, contact and object state over time.
Hardware triggeringHardware triggering uses a physical signal to start every camera’s exposure at once, so multi-camera and stereo streams capture the same instant.
HD-EPICHD-EPIC is a published highly detailed egocentric kitchen video dataset with interconnected labels grounded in 3D. Firsthand is not part of it.
Head-mounted cameraA camera worn on the head that records a task from the wearer’s own point of view, giving robot learning data the viewpoint a robot with a head camera actually sees from.
Hi RobotHi Robot is a hierarchical VLA: a high-level model turns complex prompts into commands for a low-level policy. Third-party research.
HiRTHiRT is a published hierarchical robot policy: a slow VLM guides a fast low-level policy. Third-party; Firsthand is not affiliated.
HOI4DHOI4D is a published 4D egocentric dataset for category-level human-object interaction. Firsthand is not part of it and does not use it.
HoloAssist datasetHoloAssist is a published two-person egocentric dataset for interactive AI assistants. Firsthand is not part of it and does not use it.
HOT3DHOT3D is a published egocentric dataset for 3D hand and object tracking. Firsthand is not part of it and does not use it.
HPT robot modelHPT pre-trains one shared transformer trunk across robot embodiments, aligning each one’s proprioception and vision to tokens before control heads.
HRP (Human Affordances for Robotic Pre-Training)HRP fine-tunes an existing visual encoder to predict hand, object and contact affordances mined from human video. Firsthand is not affiliated.
Human demonstration dataRecordings of people performing tasks directly, used to train a policy by imitation.
Humanoid foundation modelA generalist robot model built for a human-shaped body, pretrained on web, human, synthetic and robot data, then adapted to a specific humanoid.
Humanoid Policy ~ Human Policy (HAT)Humanoid Policy ~ Human Policy pairs the PH2D dataset with the HAT policy to co-train humanoids on egocentric human data. Firsthand is not affiliated.
Humanoid robotA robot with a human-like head, torso, two arms and hands, and usually two legs, built to work in spaces and with tools designed for people.
HumanPlusHumanPlus is a published system for humanoid shadowing and imitation from human data. Firsthand is not part of it.
HumeHume is a published dual-system VLA: System 2 picks the best of several sampled actions by value, System 1 denoises it. Firsthand is not affiliated.
HybridVLAHybridVLA is a published VLA that combines diffusion and autoregressive action prediction in one LLM. Firsthand is not affiliated with its authors.
Im2Flow2ActIm2Flow2Act is a published framework that uses object flow as the interface between human video and robot actions. Third-party research; Firsthand is not affiliated.
Imitation learningTraining a policy to reproduce demonstrated behavior from example trajectories.
IMU dataAccelerometer and gyroscope motion data, hardware-timestamped on the same clock as video.
IndustReal datasetIndustReal is a published egocentric dataset of people assembling a toy car, with steps and errors labeled. Firsthand is not affiliated.
Inner MonologueInner Monologue is a published method where an LLM robot planner reads closed-loop language feedback to replan. Third-party; Firsthand is not affiliated.
Instance segmentationPer-pixel masks that identify and separate individual objects over time.
InstructVLAInstructVLA is a published VLA trained with vision-language-action instruction tuning to keep VLM reasoning while acting. Firsthand is not affiliated.
Inter-annotator agreementInter-annotator agreement measures how consistently annotators label the same item. See the label review Firsthand states and what is Not stated.
Interleave-VLAInterleave-VLA is a published VLA framework that follows interleaved image-text instructions to output robot actions. Firsthand is not affiliated.
Inverse dynamics modelA model that infers the action most likely to have caused an observed state change, used to pseudo-label video that carries no explicit action ground truth.
Knowledge insulating VLAA knowledge insulating VLA stops action-expert gradients reaching its VLM backbone, which trains on FAST tokens. Firsthand is not affiliated.
Language-conditioned policyA language-conditioned robot policy picks actions from observations plus a language instruction. How RT-2, OpenVLA, Octo, π0 and GR00T N1 take language.
LAPA (latent action pretraining)LAPA is a published method that pretrains vision-language-action models on video by learning latent actions, with no robot action labels.
Latent actionsA learned, compressed representation of what changed between two video frames, inferred without an explicit robot action label.
Lens distortionHow a real lens bends straight lines away from the pinhole model, radially and tangentially, and the coefficients calibration estimates to undo it.
LeRobot formatAn episodic dataset format used by the LeRobot robot-learning ecosystem.
LIBERO benchmarkLIBERO is a published benchmark for lifelong robot learning: 130 manipulation tasks in four task suites, with human-teleoperated demonstrations.
LIV (Language-Image Representations and Rewards)LIV is a published method that learns a vision-language representation and reward from text-annotated video, pre-trained on EPIC-KITCHENS. Third-party.
LLaRALLaRA is a published framework that turns a vision-language model into a robot policy using instruction-tuning data. Firsthand is not affiliated.
LLARVALLARVA is a published LMM instruction-tuned on structured prompts to predict robot actions and 2-D visual traces. Firsthand is not affiliated.
Loco-manipulationA robot moving its base or body through space while it manipulates objects, rather than manipulating from a fixed position; common for humanoids.
LoHoVLALoHoVLA is a published unified VLA that plans sub-tasks and predicts actions in one model. Third-party; Firsthand is not affiliated.
Long-horizon taskA long-horizon task is a multi-step job whose sub-actions must happen in order, captured as one episode with ordered action segments for robot training.
Long-VLALong-VLA is a published long-horizon VLA that uses phase-aware input masking to split each subtask into moving and interaction phases. Firsthand is not affiliated.
LoRA fine-tuning for VLALoRA fine-tuning adapts a pretrained vision-language-action model by training small low-rank matrices instead of every weight. Firsthand trains no models.
Magma foundation modelMagma is a published foundation model for multimodal AI agents, pretrained with Set-of-Mark and Trace-of-Mark labels. Firsthand is not part of it.
Masked Visual Pre-training (MVP)MVP is a published method that pre-trains a visual encoder by masked modeling of real-world images, freezes it, and trains control policies on top. Third-party.
MCAP file formatMCAP is an open container format for timestamped robotics logs. Firsthand offers MCAP delivery and is not affiliated with Foxglove.
MECCANO datasetMECCANO is a published multimodal egocentric dataset of people assembling a toy motorbike, with RGB, depth and gaze. Firsthand is not affiliated.
MemoryVLAMemoryVLA is a published framework that adds a perceptual-cognitive memory bank to VLA models for long-horizon manipulation. Firsthand is not affiliated.
Mid-trainingA smaller, embodiment-matched dataset that bridges broad pretraining to a target robot before fine-tuning.
MimicGenMimicGen is an NVIDIA and UT Austin system that synthesizes large robot demonstration datasets from a small number of human demonstrations.
MimicPlayMimicPlay is a published method that learns latent plans from human play video to guide a robot controller trained on few demos. Third-party.
Mobile ALOHAMobile ALOHA is a published low-cost whole-body teleoperation system for bimanual mobile manipulation. Firsthand is not part of it.
MOKAMOKA uses a VLM and marked images to predict keypoint affordances for robot manipulation. Third-party research; Firsthand is not affiliated.
MolmoActMolmoAct is a published action reasoning model that predicts depth and a 2D path before acting. Firsthand is not affiliated with its authors.
Motion blurMotion blur is smearing from movement during one exposure. See what Firsthand states about shutter, frame rate and blur, and what it leaves Not stated.
Motion retargetingMotion retargeting maps captured human hand and wrist motion onto a robot with different kinematics, one way to bridge the embodiment gap.
Motion Tracks (MT-π) for Human-Robot TransferMotion Tracks (MT-π) is a published method using 2D image trajectories as actions for human hands and robots. Firsthand is not affiliated.
Moto (Latent Motion Token)Moto is a published method that turns video into latent motion tokens to pretrain a robot policy. Third-party; Firsthand is not affiliated.
Motor priorGeneral knowledge of how to move and manipulate, learned from broad pretraining data and reused as a starting point when adapting a model to a specific robot.
Multimodal training dataTwo or more aligned data streams, time-synchronized so a model can learn how they relate.
NaVILANaVILA is a published two-level VLA for legged-robot navigation: mid-level language actions drive a locomotion RL policy. Firsthand is not affiliated.
Neural trajectoriesSynthetic robot-action video from a fine-tuned video model, used by GR00T N1 to multiply a small teleoperation set into a much larger training set.
Non-exclusive data licenseA grant to train on a dataset that does not stop the licensor from licensing the same data to others.
NORA (small open-source VLA)NORA is a published 3B-parameter, open-source vision-language-action model for robot control. Firsthand is not part of it and makes no claim about it.
Nymeria datasetNymeria is a published dataset of multimodal egocentric daily motion in the wild, with full-body motion and motion-language descriptions.
OakInk2 datasetOakInk2 is a published dataset of bimanual hand-object manipulation in complex daily tasks, with multi-view video and 3D hand, body and object poses.
Object pose estimation6D object pose is an object’s 3D rotation and translation relative to a camera. Firsthand’s schema lists hand pose and contacts; object pose is per spec.
ObjectVLAObjectVLA is a published VLA method that co-finetunes robot data with image-text data for novel objects. Firsthand is not affiliated.
OcclusionOcclusion is when something blocks the camera’s view of a hand or object. See how Firsthand flags occluded hand joints and rejects lost-view episodes.
Octo modelOcto is a published open-source generalist robot policy: a transformer with a diffusion action head. Firsthand is not part of it.
OKAMI (Humanoid Manipulation from Single Video Imitation)OKAMI is a published method that teaches a humanoid a manipulation skill from a single RGB-D human video. Firsthand is not affiliated.
On-demand data collectionCustom training data captured against your spec when you need it, from an already-warm pipeline, instead of a fixed catalog or a program you build and run yourself.
OneTwoVLAOneTwoVLA is a published VLA that switches between reasoning and acting in one model. Firsthand is not part of it.
Open X-EmbodimentOpen X-Embodiment is an open dataset of 1M+ real robot trajectories from 22 embodiments, released with RT-X models. Firsthand is not part of it.
OpenHelixOpenHelix is a published open-source dual-system VLA model and short survey. Firsthand is not affiliated with its authors.
OpenVLAOpenVLA is a published open-source vision-language-action model for robot control. Firsthand is not part of it and makes no claim about it.
OpenVLA-OFTOpenVLA-OFT is OpenVLA fine-tuned with the Optimized Fine-Tuning recipe: parallel decoding, action chunking, continuous actions and L1 regression.
OTTEROTTER is a published robot policy that uses a frozen CLIP to pick task-relevant visual features. Third-party research; Firsthand is not affiliated.
PaLM-EPaLM-E is a published embodied multimodal language model that turns images and robot state into text plans. Third-party; Firsthand is not affiliated.
Participant diversityVariation across the people who perform demonstrations �� hands, technique, pace, style and handedness.
Participant release formA participant release form is the signed grant of rights to use a contributor’s recordings for AI training. Firsthand collects it before capture.
PD-VLAPD-VLA is a published parallel decoding method that speeds up VLA models using action chunking. Third-party research; Firsthand is not affiliated.
PerAct (Perceiver-Actor)PerAct is a published language-conditioned Transformer that detects the next best voxel action in RGB-D grids. Firsthand is not affiliated.
Phantom (Training Robots Without Robots Using Only Human Videos)Phantom trains robot policies from human video only, by inpainting the arm and overlaying a rendered robot. Third-party; Firsthand is not affiliated.
Physical AIAI systems that perceive and act in the physical world through a body, such as a robot arm or humanoid.
Pilot data collectionA small, fast first batch that proves a spec and accept criteria before you commit to full volume.
Point cloudA point cloud is a set of 3D points. See how one is computed from calibrated stereo depth, and which depth fields Firsthand states.
PointVLAPointVLA is a published framework that adds point cloud inputs to pre-trained VLA models without retraining. Firsthand is not affiliated.
Post-trainingThe final stage of a robot foundation model training recipe, specializing a model to one deployment task, robot, or environment.
PretrainingTraining a robot foundation model on broad, large-scale data before it is adapted to a specific task.
Pretraining corpusThe large-scale dataset of egocentric human video, robot demonstrations, or both used for a robot foundation model’s pretraining stage.
Production data pipelineA repeatable process that delivers robotics training data against a spec, with measured quality every batch.
ProprioceptionA robot’s sense of its own body, such as joint and gripper state, fed to a policy as observations. Human video has none; some human signals stand in.
Pseudo-labelsLabels a model produces for data, rather than labels recorded at capture or verified by a human.
PTP clock syncPrecision Time Protocol �� sub-microsecond clock alignment across sensors.
Quality gateA quality gate is a pass-or-fail check data must clear before delivery. See which gates Firsthand states and which thresholds are not stated.
QUAR-VLAQUAR-VLA is a published VLA paradigm for quadruped robots: an image and an instruction become commands. Third-party; Firsthand is not affiliated.
R3MR3M is a published visual representation pre-trained on Ego4D human video and reused as a frozen perception module for robot manipulation. Third-party.
RDT-1BRDT-1B is an open diffusion transformer policy for bimanual robot manipulation, conditioned on language and camera images.
ReconVLAReconVLA is a published reconstructive VLA model that reconstructs gaze regions to focus visual attention. Firsthand is not affiliated.
RedactionPermanently obscuring specific regions of an image, video, or document.
Rejection rateA rejection rate is the share of episodes that fail QA. See the reject criteria Firsthand states and why no rejection rate figure is stated.
ReKepReKep turns language into 3D keypoint constraints that a solver optimizes into robot actions. Third-party research; Firsthand is not affiliated.
Reprojection errorThe pixel gap between detected calibration points and where the solved camera model projects them, and how it is read in mono and stereo calibration.
RH20T datasetRH20T is a published robot manipulation dataset: over 110,000 contact-rich sequences, each with visual, force, audio and action data.
RIPT-VLARIPT-VLA is a published reinforcement post-training method that fine-tunes a pretrained VLA model with only sparse binary success rewards. Third-party; Firsthand is not affiliated.
RLDSReinforcement Learning Datasets — an episodic format for robot learning data.
RoboBrainRoboBrain is a published model that plans manipulation tasks, then predicts affordances and trajectories. Firsthand is not affiliated.
RoboCasaRoboCasa is a published large-scale kitchen simulation framework for training and benchmarking generalist robots, with LLM-guided tasks.
RoboCatRoboCat is a published self-improving generalist agent for robotic manipulation from Google DeepMind. Third-party; Firsthand is not affiliated.
RoboDualRoboDual is a published dual-system robot policy: a slow VLA generalist guides a fast diffusion specialist. Third-party; Firsthand is not affiliated.
RoboFlamingoRoboFlamingo adapts the OpenFlamingo vision-language model into a robot policy with a history-aware policy head. Third-party research.
RoboMambaRoboMamba is a published VLA that pairs a CLIP vision encoder with the Mamba state space model. Firsthand is not affiliated with its authors.
RoboMIND datasetRoboMIND is a published robot manipulation dataset: 107k teleoperated trajectories on four robot embodiments under one collection standard.
RoboMonkeyRoboMonkey is a published test-time scaling framework for VLAs: sample actions, then a VLM verifier picks one. Firsthand is not affiliated.
RoboPointRoboPoint is a published VLM that predicts 2D keypoint affordances from language to guide robot actions. Third-party; Firsthand is not affiliated.
Robot foundation modelA large policy pretrained on broad robot data, then adapted to specific tasks or embodiments.
Robot learningTraining a robot policy from data instead of hand-coding its behavior.
RoboUniViewRoboUniView is a published vision-language model for robotic manipulation that predicts actions from a unified view representation of multi-camera images. Third-party; Firsthand is not affiliated.
RoboVLMsRoboVLMs is a published study and open framework on how to build vision-language-action models from VLM backbones. Firsthand is not part of it.
Rolling shutterA camera that exposes a frame row by row, causing skew and wobble during fast motion.
RT-1RT-1 is a published Robotics Transformer for real-world robot control. Firsthand is not part of it and makes no claim about it.
RT-2RT-2 is a published vision-language-action model that writes robot actions as text tokens. Firsthand is not part of it and makes no claim about it.
RT-HRT-H is a published robot policy that predicts a language motion, then the action. Third-party research; Firsthand is not affiliated.
RT-TrajectoryRT-Trajectory is a published robot policy conditioned on hindsight trajectory sketches. Third-party; Firsthand is not affiliated.
RVT (Robotic View Transformer)RVT is a published multi-view transformer that re-renders RGB-D scenes from virtual views to predict robot gripper poses. Firsthand is not affiliated.
SafeVLASafeVLA is a published safety alignment method for VLA models that uses constrained learning. Third-party; Firsthand is not affiliated.
SayCanSayCan is a published method where a language model picks robot skills, weighted by value functions. Third-party; Firsthand is not affiliated.
Seer (PIDM)Seer is a published end-to-end predictive inverse dynamics model: it forecasts future visual states, then infers actions. Firsthand is not affiliated.
Sensor fusionSensor fusion combines stereo video, depth and IMU streams into one time-aligned record. Firsthand states one PTP clock and a 2.0 ms sync ceiling.
Sim-to-real gapThe mismatch between a simulator’s approximations and real-world physics, lighting, and material behavior.
SimplerEnvSimplerEnv is open-source simulated evaluation of real-world robot manipulation policies, built so simulated results track real-world performance.
SimpleVLA-RLSimpleVLA-RL is a published reinforcement learning framework for robot VLA models, built on veRL. Firsthand is not affiliated.
Skill specA written definition of the exact task, environments and conditions to capture.
SmolVLASmolVLA is a published small, open vision-language-action model for affordable robotics. Firsthand is not part of it and makes no claim about it.
SpatialVLA modelSpatialVLA is a 3D-aware VLA model that adds Ego3D Position Encoding and Adaptive Action Grids to spatially enhance robot manipulation policies.
Statement of workThe document that defines the task, geometry, conditions, volume and format of a custom dataset, and how it works in a fast, on-demand collection.
Stereo baselineThe fixed distance between the two lenses in a calibrated stereo camera pair, measured every session and used to recover depth from disparity.
Stereo disparityThe horizontal pixel shift of one point between left and right stereo images. Larger for nearer points, it is the basis of stereo depth.
Stereo matchingThe search that finds which pixel in one stereo image shows the same scene point as a pixel in the other. Also called stereo correspondence.
Stereo rectificationWarping the left and right images of a calibrated stereo pair so matching points share an image row, which makes disparity search one-dimensional.
Stereoscopic videoVideo captured by two horizontally offset cameras at once, giving each frame recoverable depth.
SWIM (Structured World Models from Human Videos)SWIM pre-trains a world model on human video using affordance actions, then fine-tunes it on robot data. Firsthand is not affiliated.
SwitchVLASwitchVLA is a published VLA framework for switching tasks when a new instruction arrives mid-execution. Third-party; Firsthand is not affiliated.
Synthetic training dataRobot training data produced by software, from a physics simulator or a neural generation model, instead of recorded from the real world.
TACO datasetTACO is a published bimanual hand-object dataset of tool-action-object compositions with egocentric and third-person views, 3D meshes and action labels.
Tactile dataTactile data is touch-sensor readings of contact, such as pressure and slip. Contact events are labels; Firsthand’s standard capture has none.
Task decompositionBreaking a human job into short, repeatable task units a robot policy can be trained on one at a time.
Task taxonomyA structured hierarchy (sector, environment, task, skill) that organizes what a robot-training dataset covers, so gaps can be planned and audited.
Teleoperation vs egocentric dataRobot-collected demonstrations versus human-worn first-person capture.
ThinkActThinkAct is a published VLA framework: a multimodal LLM reasons, then a visual plan latent conditions an action model. Firsthand is not affiliated.
TinyVLATinyVLA is a published family of compact, fast VLA models that need no robot pretraining. Firsthand is not part of it.
TraceVLA (visual trace prompting)TraceVLA is a published VLA model that overlays point-tracked visual traces on robot images. Firsthand is not part of it and makes no claim about it.
Track2ActTrack2Act is a published method that predicts point tracks from web video to plan robot manipulation. Third-party research; Firsthand is not affiliated.
Training data indemnityTraining data indemnity explained: how it differs from a warranty and a licence, and what Firsthand states about its consent-chain warranty.
Training data licenseA training data license is the contract that sets what a buyer may do with a dataset to train and commercialize AI models: scope, term and exclusivity.
Trajectory dataTrajectory data is the recorded path of a wrist or hand over time, delivered as time-stamped streams beside video to train robot and humanoid policies.
TriVLATriVLA is a published triple-system VLA that pairs a VLM and a video diffusion model with a flow-matching policy. Firsthand is not affiliated.
UniActUniAct is a published framework that learns a universal action space shared across robot embodiments. Firsthand is not affiliated with its authors.
Universal Manipulation InterfaceUMI is an open-source handheld-gripper framework for collecting robot demonstrations without a robot. Firsthand is not part of it and does not use it.
UniVLAUniVLA is a published framework that learns cross-embodiment vision-language-action policies from video using task-centric latent actions.
UP-VLAUP-VLA is a published VLA model trained on understanding and future prediction. Firsthand is not affiliated with its authors.
Validated hourAn hour of footage that has passed QA against the spec — not an hour of raw recording.
Validation lossA held-out error metric used during robot pretraining to track how well a model generalizes.
VC-1 (Artificial Visual Cortex)VC-1 is the largest pre-trained visual encoder from a published study of visual representations for embodied AI, evaluated on CortexBench. Third-party.
Vid2RobotVid2Robot is a published video-conditioned policy that watches a human demo and outputs robot actions. Third-party; Firsthand is not affiliated.
Video Prediction Policy (VPP)VPP is a published robot policy that reads predictive visual representations from a video diffusion model. Third-party; Firsthand is not affiliated.
VideoDex (Learning Dexterity from Internet Videos)VideoDex is a published method that pretrains robot-hand policies on retargeted human video. Third-party; Firsthand is not affiliated.
villa-Xvilla-X is a published VLA framework that grounds latent actions in robot dynamics. Firsthand is not affiliated with its authors.
VIMA (multimodal prompts)VIMA is a published robot agent that follows multimodal prompts of text and images, tested in the simulated VIMA-Bench. Firsthand is not part of it.
VIP (Value-Implicit Pre-training)VIP is a published visual representation pre-trained on Ego4D human video; embedding distance to a goal image gives a robot reward. Third-party.
Vision-language model (VLM)A model that takes images and text together and learns how they relate, trained on frames paired with language that is true about them.
Vision-language-action modelA model that maps a visual scene and a language instruction to an action.
Visuomotor policyA visuomotor policy maps camera images to robot actions. See the cited sources and what Firsthand captures as training data.
VLA-0VLA-0 is a published VLA design where an unmodified vision-language model writes robot actions as text. Third-party; Firsthand is not affiliated.
VLA-AdapterVLA-Adapter is a published tiny-scale VLA paradigm that bridges vision-language features to actions. Firsthand is not part of it.
VLA-CacheVLA-Cache is a published training-free method that reuses cached visual tokens across frames to speed up VLA inference. Firsthand is not affiliated.
VLA-OSVLA-OS is a published study that compares how vision-language-action models plan before they act. Firsthand is not affiliated.
VLA-RLVLA-RL is a published online reinforcement learning framework for pretrained autoregressive VLAs. Firsthand is not affiliated with its authors.
VLASVLAS is a published vision-language-action model that follows spoken instructions directly. Firsthand is not affiliated.
VoltronVoltron is a published framework that learns robot visual representations from human video and captions. Third-party; not trained on Firsthand data.
VoxPoserVoxPoser is a published method that has a language model compose 3D value maps for zero-shot robot trajectories. Third-party; Firsthand is not affiliated.
VQ-VLAVQ-VLA is a published vector-quantized action tokenizer scaled on large trajectory data to improve VLA models. Firsthand is not affiliated.
VRB (Vision-Robotics Bridge)VRB is a published method that learns contact points and post-contact trajectories from human video. Third-party research; Firsthand is not affiliated.
WALL-OSSWALL-OSS is a published open-source embodied foundation model that adapts a vision-language model to robot control. Firsthand is not affiliated.
WebDatasetA tar-based format for streaming large datasets efficiently during training.
WHIRL (Human-to-Robot Imitation in the Wild)WHIRL is a published method that turns one human video into a robot policy, improved by real-world trials. Third-party; Firsthand is not affiliated.
Workflow captureRecording a real enterprise workflow as first-person video, structured into a skill spec and labeled steps for AI training.
World modelA model that learns to predict how a scene changes in response to an action.
WorldVLAWorldVLA is a published autoregressive action world model unifying a VLA and a world model. Firsthand is not affiliated with its authors.
Wrist cameraA camera mounted at the wrist that keeps the grasp point in frame, giving robot learning data a close view of contact a head camera alone cannot provide.
X-VLAX-VLA is a published flow-matching VLA that uses embodiment-specific soft prompts for cross-embodiment learning. Firsthand is not affiliated.
Zero-Shot Robot Manipulation from Passive Human VideosA published framework that predicts human hand trajectories from a scene image, learned from passive human video. Firsthand is not affiliated.
ZeroMimic (Distilling Robotic Manipulation Skills from Web Videos)ZeroMimic distills robot manipulation skills from egocentric web video, with no robot data. Third-party research; Firsthand is not affiliated.
π0 (pi-zero)π0 is a published vision-language-action flow model for robot control. Firsthand is not part of it and makes no claim about it.
π0.5 (pi0.5)π0.5 is a published vision-language-action model built on π0 for open-world generalization. Firsthand is not part of it and makes no claim about it.