What Is VC-1 (Artificial Visual Cortex)?

The paper is by Arjun Majumdar, Karmesh Yadav, Sergio Arnaud and 12 co-authors. The project page lists Georgia Institute of Technology, Meta AI, the University of Pennsylvania, Stanford University and UC Berkeley, and the arXiv record shows it was submitted on 31 March 2023 and last revised on 1 February 2024. The authors describe it as the largest and most comprehensive empirical study of pre-trained visual representations (PVRs), or visual “foundation models”, for embodied AI. Their question is whether one visual encoder can serve many robot, navigation and manipulation tasks. They also state that the paper presents no new techniques, but a systematic evaluation, a set of findings about PVRs, and open-sourced code and models.

Firsthand is not affiliated with the VC-1 authors and makes no claim that any model, VC-1 included, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Pretrained representations, embeddings, video-language annotations, narrations, robot trajectories, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the abstract, the authors curate CortexBench, a suite of 17 tasks spanning locomotion, navigation, dexterous manipulation and mobile manipulation. The project page says it is assembled from 7 benchmarks, and names Adroit, MetaWorld, DMControl, Trifinger, ImageNav, Mobile Pick and ObjectNav in its results text. It describes candidate models being evaluated with either reinforcement learning or imitation learning. The authors argue that, for a visual representation to be called foundational, it must be broadly applicable, which is why they build a benchmark rather than test on one domain.

Per the abstract, the authors combine over 4,000 hours of egocentric video from 7 different sources and ImageNet, then train vision transformers of different sizes with Masked Auto-Encoding (MAE) on slices of that data. The project page names two backbone sizes, ViT-B and ViT-L, and VC-1 is the largest model. The project page labels its top-ranked variant “MAE Ego4D+MNI”, so Ego4D is among the sources, but the full list of sources and the expansion of “MNI” are Not stated in the pages read here. The abstract’s image count differs between arXiv versions, so it is not repeated. The authors report that training the models took over 10,000 GPU-hours.

Three findings are stated in the abstract. First, among existing PVRs, none is universally dominant; the project page says R3M did best on Adroit, MetaWorld and DMControl, MVP (ViT-L) on Trifinger, ImageNav and Mobile Pick, and CLIP on ObjectNav. Second, scaling dataset size and diversity does not improve performance universally, but does so on average. Third, VC-1 outperforms all prior PVRs on average but does not universally dominate either. The abstract also reports real-world hardware experiments in which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. These are the authors’ claims; Firsthand has not verified them, and no scores are repeated here.

The project page describes two strategies: end-to-end fine-tuning with a task-specific loss, and MAE adaptation using task-specific data. It gives two reasons for adapting. MAE features are generally useful for reconstructing images, so adaptation can specialize the encoder for tasks such as object rearrangement, and it can reduce the domain gap between human-video pretraining and the simulated tasks used for evaluation. Per the abstract, VC-1 (adapted) achieves competitive or superior performance to the best known results on all of the CortexBench benchmarks.

The second table sets VC-1 beside four site entries. The columns for the other entries summarise what those entries state, not new claims. R3M is the one entry the VC-1 sources also discuss: the authors evaluate it as an existing PVR. For the wider stage this belongs to, see the pretraining entry.

Per the sources read, it needs large and diverse egocentric video to pre-train an image encoder, and then task-specific data if the encoder is adapted. Firsthand states real human egocentric capture with stereo depth, 3D hand pose, IMUs and one task instruction per episode. That is a note about format, not a claim of compatibility. Training or adapting a visual representation is outside Firsthand’s scope.

When a program needs real human demonstrations that public video does not cover, specified field by field. Firsthand does not build or train models, or supply robot data. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is VC-1?

VC-1 (Visual Cortex-1) is the largest pre-trained visual encoder from a published study of visual representations for embodied AI (arXiv 2303.18240), trained with masked auto-encoding on egocentric video and ImageNet. It is third-party research.

What is CortexBench?

The authors’ benchmark of 17 embodied AI tasks spanning locomotion, navigation, dexterous manipulation and mobile manipulation, used to evaluate pre-trained visual representations.

Does VC-1 beat every other visual representation?

Not on every task. The authors report that VC-1 outperforms all prior PVRs on average but does not universally dominate, and that adapting it leads to substantial gains. These are their claims; Firsthand has not verified them.

Can I use the VC-1 code and models?

The authors say the code and models are open-sourced, and the GitHub repository facebookresearch/eai-vc lists its license as Other. Confirm the current license terms yourself before relying on it.

Does Firsthand use VC-1, and can I request a custom dataset?

Firsthand is not affiliated with the VC-1 authors and claims no use, training or compatibility. Pretrained representations and embeddings are not Firsthand deliverables. You can request a custom dataset.