What Is Masked Visual Pre-training (MVP) for Motor Control?

The paper is by Tete Xiao, Ilija Radosavovic, Trevor Darrell and Jitendra Malik of the University of California, Berkeley, with the first two marked as equal contributors. arXiv records it as submitted on 11 March 2022. The authors observe that learning representations with large neural networks has driven progress in vision, language and audio, but that motor control remains a notable exception. Their proposal is to learn the visual part of a control policy from real-world images instead of from environment-specific experience. They say this sidesteps two problems of end-to-end reinforcement learning: training is computationally expensive with poor sample complexity, and the learned solutions typically overfit to the setting at hand. To the best of their knowledge, they state, it is the first self-supervised model to exploit real-world images at scale for motor control.

Firsthand is not affiliated with the MVP authors and makes no claim that any model, MVP included, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Pretrained encoders, embeddings, simulators, benchmarks, robot trajectories, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

The first stage uses a masked autoencoder (MAE). Per the paper, it masks random patches of an input image and reconstructs the missing pixels with a Vision Transformer. During training only the unmasked patches are fed to the encoder, which the authors say makes training more efficient. They describe it as critical to use a high masking ratio, for example masking 75% of patches, and a heavy encoder with a light decoder. They also note that it works well with minimal augmentation, namely center crop and color. No human labels are used.

After pre-training, the authors freeze the image encoder and never fine-tune it per task; the same visual representations serve every task. All image patches pass through the encoder to give a fixed-size feature vector. Proprioceptive information, joint positions and velocities, goes into a second vector, and the two are concatenated. A small multi-layer perceptron policy is then trained on that embedding with proximal policy optimization (PPO), alongside a critic of the same architecture that does not share weights. The authors give two reasons for freezing: it keeps the encoder from overfitting to one setting, and it saves memory and run time because nothing is back-propagated through the encoder.

The paper uses images from three public sources: the egocentric Epic Kitchens dataset, the YouTube 100 Days of Hands dataset and the crowd-sourced Something-Something dataset. Combined, they yield about 700K images, which the authors call the Human-Object Interaction (HOI) dataset. They state that they do not use any human labels or temporal information, even though it would be possible. ImageNet is also used for controlled comparisons with supervised baselines. One reported finding is that pre-training on in-the-wild images, for example from YouTube or egocentric video, works better for manipulation tasks than ImageNet images. That is the authors’ claim; Firsthand has not verified it.

PixMC (Pixel Motor Control) is the benchmark suite the authors contribute: hand-designed tasks varying in movements, scenes and robots, built on NVIDIA’s IsaacGym GPU simulator, with reward functions, baselines and a multi-GPU implementation of learning from pixels. It covers a Franka arm with a parallel gripper and a Kuka arm with a multi-finger hand, and its figures show 8 tasks. The abstract reports that, without labels, state estimation or expert demonstrations, MVP outperforms supervised encoders by up to 80% absolute success rate, sometimes matching oracle state performance. The project page adds that it beats the supervised baseline on 7 tasks and closely matches the oracle on 5 at convergence. These are simulation results from the authors; no scores are repeated here.

The second table sets MVP beside four site entries. The columns for the other entries summarise what those entries state, not new claims. The VC-1 entry notes that the VC-1 authors evaluate MVP (ViT-L) as an existing pre-trained visual representation. For the wider stage this belongs to, see the pretraining entry.

Per the sources read, it needs a large collection of in-the-wild images to pre-train the encoder, plus a simulator and reward functions to train controllers. Firsthand states real human egocentric capture with stereo depth, 3D hand pose, IMUs and one task instruction per episode. That is a note about format, not a claim of compatibility. Pre-training an encoder or running a simulated benchmark is outside Firsthand’s scope.

When a program needs real human demonstrations that public images and video do not cover, specified field by field. Firsthand does not build or train models, or supply robot data. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is masked visual pre-training (MVP)?

MVP is the method from arXiv 2203.06173: pre-train a vision transformer encoder with masked image modeling on real-world images, freeze it, and train reinforcement-learning controllers on top for motor control from pixels. It is third-party research.

What is PixMC?

The authors’ benchmark suite of hand-designed motor control tasks varying in movements, scenes and robots, built on the IsaacGym simulator, for studying learning from pixels.

Does MVP beat supervised pre-training?

The authors report that it outperforms supervised encoders by up to 80% absolute success rate on their simulated tasks, and sometimes matches oracle state performance. These are their claims; Firsthand has not verified them.

Can I use the MVP code and models?

The authors release the benchmark suite, pre-trained models and training code, and the GitHub repository is ir413/mvp. The repository summary read does not state a license, so confirm the current terms yourself before relying on it.

Does Firsthand use MVP, and can I request a custom dataset?

Firsthand is not affiliated with the MVP authors and claims no use, training or compatibility. Pretrained encoders and simulated benchmarks are not Firsthand deliverables. You can request a custom dataset.