What Is VRB (Vision-Robotics Bridge)?

VRB is named in Affordances from Human Videos as a Versatile Representation for Robotics (arXiv 2304.08488), submitted on 17 April 2023 by Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain and Deepak Pathak, from Carnegie Mellon University and Meta AI. The arXiv comments and project page name CVPR 2023. The project page expands the name as Vision-Robotics Bridge, because the authors aim to integrate computer vision techniques with robotic manipulation. They argue that a robot needs to know where and how to manipulate an object, not only what it is, and that videos of humans interacting offer more than a pretrained visual encoder. For the broader idea, see the site’s affordance entry.

Firsthand is not affiliated with the VRB authors and makes no claim that any model, VRB included, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. Affordance labels, contact-point labels, post-contact trajectories, contact heatmaps, pretrained representations, embeddings, reward labels, robot trajectories, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, VRB adopts contact points and post-contact trajectories as a simple, actionable representation of visual affordance, both in pixel space. A contact point says where to interact, such as the handle of a lid, and the trajectory says how to move after contact, such as moving the lid upwards. The authors argue that copying human movement would be human-centric and would not generalize well because human morphology is starkly different from a robot’s, so this abstraction is agnostic to the morphology of the robot.

The authors train on EPIC-KITCHENS-100, using each sub-action clip, such as open door, as a sequence, and report around 54K image-trajectory-contact point tuples. A hand-object detector, with annotations released alongside the dataset, gives a hand box and a contact variable per frame. The first contact frame is found, the hand location afterwards becomes the post-contact trajectory, and skin-colour segmentation around the hand gives contact points, which a Gaussian mixture model turns into several likely targets. The project page describes the trajectory as the tracked wrist.

A homography between frames compensates for head movement, and because no human appears at robot test time, the authors map the affordances back to the first frame without the human. The labels are inferred by this pipeline, not recorded by a sensor. The project page also names Ego4D as a source of large-scale human video; the paper’s data section describes EPIC-KITCHENS-100.

Per the paper, a ResNet visual encoder produces a spatial latent, deconvolutional layers turn it into several heatmaps, and a spatial softmax gives contact-point estimates. Several heatmaps are used because the authors say the problem is multi-modal. A transformer with self-attention predicts the trajectory as relative shifts, and training uses local crops around contact points. The README says the output can be used directly at inference with sparse 3D information, such as depth, and robot kinematics. The paper projects the pixel output into 3D with a calibrated robot-camera system.

The authors integrate the model with four robot learning paradigms: affordance-driven data collection for offline imitation learning, reward-free exploration, goal-conditioned learning, and affordance as an action space for reinforcement learning. The abstract reports 4 real-world environments, over 10 tasks and 2 robotic platforms. The paper names a Franka Emika Panda arm and a Hello Robot Stretch mobile manipulator, and also tests on a simulation benchmark modelled on the Franka Kitchen environment from D4RL.

In the offline imitation test, the authors report that VRB beat prior approaches on 7 of 8 tasks, with an average success rate of 57% against 25% for the runner-up, Hotspots. They also report that its visual encoder beat R3M on three simulated Franka tasks after fine-tuning. These are the authors’ figures on their setup; Firsthand has not verified them. The conclusion lists multi-stage tasks and force and tactile information as future directions.

The first table sets VRB beside four site entries, summarising what those entries state. The General Flow entry notes that its paper lists Bahl et al. (VRB) as the only open-source work with a similar setup.

Per the paper and README, it needs egocentric human video with hand-object contact, a hand-object detector, a way to estimate camera motion, and camera calibration plus depth to turn pixel outputs into robot actions. Firsthand states real human egocentric capture, depth from a calibrated stereo rig, 3D hand pose, and contact events (see the affordance entry), with one task instruction per episode. That is a note about format, not a compatibility claim: any contact points or trajectories would come from tools the buyer runs.

This site’s EPIC-KITCHENS entry names that dataset’s video and annotation licences as non-commercial. For real human demonstrations that public video does not cover, request on-demand custom collection and confirm it on a Spec Pilot, with a median of 48 hours to a first validated batch where coverage is deep and three to four weeks where it is thin.

What is VRB (Vision-Robotics Bridge)?

A published method (arXiv 2304.08488, CVPR 2023) that learns affordances from human videos as contact points and post-contact trajectories, for use across robot learning paradigms. It is third-party research.

What does VRB predict from an egocentric video frame?

Given a scene image, the model predicts where a person is likely to make contact, as heatmaps over contact points, and how the hand would move afterwards, as a trajectory, both in pixel space.

What data trains VRB?

Per the paper, EPIC-KITCHENS-100 video, with labels inferred by a hand-object detector and a segmentation and tracking pipeline. The authors report around 54K training tuples.

Is the VRB code available?

The official GitHub repository, shikharbahl/vrb, lists an MIT License. Check the licences of any dataset you train on.

Was VRB trained on Firsthand data?

No. Firsthand is not affiliated with the authors and claims no use, training or compatibility. Contact-point labels, trajectories and heatmaps are not Firsthand deliverables.

How can I request a custom egocentric dataset for affordance learning?

Request a custom dataset with your objects, interactions and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.