What Is LAPA (Latent Action Pretraining)?

LAPA is introduced in Latent Action Pretraining from Videos (arXiv 2410.11758) by Seonghyeon Ye, Joel Jang and co-authors. The abstract starts from a data limit: existing VLAs need action labels, typically collected by human teleoperators, which restricts data sources and scale. LAPA instead learns from internet-scale videos that have no robot action labels. Read: the paper, project page and README; Hugging Face pages were not.

LAPA is third-party research. Firsthand is not affiliated with its authors and makes no claim that any named model was trained on, used with or is compatible with Firsthand data. Firsthand captures first-person human video, fast, on demand and to a written spec, not robot data: 3D hand pose at 21 keypoints per hand with a per-joint visibility flag, wrist trajectories and action segments. The only language field is one task instruction per episode.

Per the paper, a latent action quantization model is an encoder-decoder. The encoder takes the current frame and a future frame and outputs a latent action; the decoder takes it and the current frame and reconstructs the future frame. A VQ-VAE-based objective makes the latent action discrete tokens, like tokenizing atomic actions without priors such as end-effector or joint positions.

There are three. Latent action quantization trains the model above. Latent pretraining then uses its encoder as an inverse dynamics model to label frames, and trains a pretrained VLM to predict those latent actions from the image and instruction through a separate latent action head. Action finetuning swaps that head for a new one and trains on a small set of trajectories with ground-truth delta end-effector actions.

The paper sets LAPA against VLAs that depend on labeled action data and against other ways to use video, such as video generation, visual traces, hand-pose retargeting and inverse dynamics models. Being-H0 and EgoVLA, in their own entries, model hand motion explicitly. Whether those papers compare with LAPA is Not stated in the sources read.

The paper lists limits: LAPA underperforms action pretraining on fine-grained motion such as grasping, has inference latency, and is unexplored beyond manipulation videos. For human video it names no action labels and a distribution unlike robot embodiments. Its appendix says latent actions there capture camera movement too.

LAPA is a method, not a data source. When a program needs first-person human video of particular tasks, environments or people, write them into a spec and confirm it on a Spec Pilot, a small fast first batch. The median from signed spec to first validated batch is 48 hours where coverage is deep and three to four weeks where it is thin. Billing is per validated hour; delivery is RLDS, LeRobot, WebDataset or HDF5.

What is LAPA?

A method from a 2024 paper that pretrains vision-language-action models on video by learning latent actions, without ground-truth robot action labels.

Does LAPA need robot action labels?

Not for pretraining. Finetuning uses a small set of trajectories with ground-truth delta end-effector actions to map latent actions to robot actions.

What data does LAPA learn from?

The paper pretrains on robot datasets without using their actions, and on Something-Something V2, 220K human manipulation videos.

Is LAPA open source?

The README says code and model weights are under the MIT License. Read each licence yourself; this is not legal advice.

What results does the paper report?

The abstract reports gains over other video-based methods. Numbers are not reproduced here.

Does Firsthand use LAPA, and can I request a custom dataset?

Firsthand is not affiliated with the LAPA authors and makes no claim of use, training or compatibility. You can request a custom egocentric dataset: write a spec, confirm it on a Spec Pilot, pay per validated hour.