What Is General Flow?

General Flow is named in General Flow as Foundation Affordance for Scalable Robot Learning (arXiv 2401.11439) by Chengbo Yuan, Chuan Wen, Tong Zhang and Yang Gao, from the Institute for Interdisciplinary Information Sciences at Tsinghua University, Shanghai Qi Zhi Institute and Shanghai Artificial Intelligence Laboratory, first submitted on 21 January 2024 and last revised on 23 September 2024. The PDF carries a CoRL 2024 header. The authors argue that robot learning needs a prediction target that can use large-scale data, and propose 3D flow as that target, learned from human video because collecting real robot data is hard to scale.

Firsthand is not affiliated with the General Flow authors and makes no claim that any model, General Flow included, was trained on Firsthand data. Firsthand captures real human egocentric video, custom collection, fast and on demand, to a written spec, with depth derived from a calibrated stereo rig, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, delivered in RLDS, LeRobot format or WebDataset under a buyer-owned commercial license, with one task instruction per episode. 3D flow labels, point tracks, optical flow, object flow, 2D or 3D trajectory labels, robot trajectories, teleoperation data, paired human-robot video, synthetic data and language annotations beyond that instruction are not Firsthand deliverables.

Per the paper, an affordance concentrates on the actions associated with an object and stays neutral to the manipulator, rooted in Gibson’s theory. The authors say affordance usually consists of a functional grasp and the subsequent motion, and that their focus is post-grasp motion. In the paper’s open Safe example, general flow is the future positions of points on the door, so a robot can follow that flow to open it. For the broader term, see the site’s affordance entry.

General Flow is not flow matching, the generative-modelling training objective; the two share only the word flow. See the flow matching entry for that technique.

Given a perception observation from any embodiment and a task instruction, plus a set of 3D query points, the general flow is the trajectories of those points over a number of future timestamps. In the paper the observation is a point cloud from an RGBD camera stream, and the instruction is natural language. The authors call it general because it applies across embodiments, from any embodiment to any robot, and across object categories: rigid, articulated and soft bodies.

Per the paper, rigid and articulated objects come from the HOI4D dataset, which the authors describe as 16 categories, 800 objects and 44.4 hours of RGBD recording with 3D labels such as active object segmentation, 3D pose and camera parameters. For soft objects they collected RGBD video of a fold clothes task with a RealSense D455 camera: 6 types of clothes, 30 trajectories and 605 extracted clips. These are the authors’ figures, and Firsthand has not verified them.

The labels are extracted by the authors’ pipeline, not recorded by the camera. For HOI4D they sample points on the active object and compute future positions from ground-truth pose and camera parameters. For their own videos they run hand-object-interaction segmentation, track points in 2D with a Tracking Any Point model, and back-project to 3D. Multiple filtering steps reduce noise.

The authors encode the instruction with CLIP, concatenate it with scene point-cloud features, process scene and query points with a PointNeXt backbone, and use a conditional VAE to capture that humans may act differently for the same task. They predict relative displacements, and rescale trajectories with Total Length Normalization because points on a door move much farther than points on a safe body. They name the resulting model ScaleFlow.

Two augmentations target zero-shot robustness: hand mask augmentation drops out hand points to build resilience to embodiment occlusion, and query point sampling varies the query point distribution.

On their held-out test set, the authors report that ScaleFlow-L reaches an average displacement error of 3.55 cm and a final displacement error of 4.70 cm with 17.1 million parameters, against 3.83 cm and 5.16 cm for PointNeXt-L. For deployment, a heuristic policy based on closed-loop flow prediction ran on a Franka-Emika robot with a single model and no in-domain finetuning. They report an average 81% success rate (146 of 180 trials) over 18 tasks in 6 scenes, 10 trials per task. Per-task rates in their Table 2 range from 3/10 for opening a drawer by grasping the handle to 10/10 for several tasks. These are the authors’ figures on their setup; Firsthand has not verified or reproduced them.

Per the paper, the diversity and volume of their data are limited, the conditional VAE may lack expressive power on larger datasets, the arm was positioned manually for task initiation, and the ICP-based heuristic policy may limit contact-rich tasks. Their failure analysis covers pushing a toy car, opening a drawer and opening a laptop, where a thin lid produced poor point clouds. They also say the label pipeline, code and model weights will be released in the future; the project page says the same, and this entry links no code repository.

The first table sets General Flow beside three site entries that also predict motion from video. The columns for the other methods summarise what the site entries state, not new claims. The paper itself names ATM and Track2Act as directions for future work, and lists Bahl et al. (VRB) as the only open-source work with a similar setup.

Per the paper, the approach needs RGBD human video, a task instruction, and 3D flow labels produced by a pipeline of segmentation, point tracking and back-projection or ground-truth pose. The paper does not state whether egocentric capture would work as the video source, so that is Not stated.

Firsthand states real human egocentric capture with one task instruction per episode and depth derived from a calibrated stereo rig. That is a note about format, not a claim of compatibility, and any flow labels would come from tools the buyer runs, not from Firsthand.

When a program needs real human demonstrations that public or synthetic data does not cover, specified field by field. Firsthand does not build or train models. Confirm the fields on a Spec Pilot, a small fast first batch; the median from signed spec to first validated batch is 48 hours where coverage is deep, three to four weeks where thin.

What is General Flow?

A published affordance representation (arXiv 2401.11439): the future trajectories of 3D points on an object of interest, predicted from a point cloud and a language instruction by a model trained on RGBD human video. It is third-party research.

Is General Flow the same as flow matching?

No. General Flow is a prediction target for robot manipulation; flow matching is a generative-modelling training objective.

What results do the authors report?

An average 81% success rate (146 of 180 trials) in zero-shot human-to-robot skill transfer, covering 18 tasks in 6 scenes, per the authors. These are the authors’ figures; Firsthand has not verified them.

What data trains a General Flow model?

Per the paper, RGBD human video: the HOI4D dataset for rigid and articulated objects and the authors’ own fold clothes recordings for soft objects, with 3D flow labels extracted by a pipeline.

Does Firsthand use General Flow, and can I request a custom dataset?

Firsthand is not affiliated with the General Flow authors and claims no use, training or compatibility. 3D flow labels, point tracks, optical flow, trajectory labels, robot trajectories and teleoperation data are not Firsthand deliverables. You can request a custom dataset.