What Is a Point Cloud in Robot Training Data?

A point cloud is a collection of points in 3D space. The Point Cloud Library describes the usual case: each point holds the X, Y and Z coordinates of a sampled surface, and colour can be added. The PointNet paper (arXiv 1612.00593) states what that means for a model: a point cloud is an unordered set of points, not a pixel grid, so there is no row-and-column order and no fixed neighbours. Open3D's PointCloud class likewise stores point coordinates and, optionally, colours and normals. A point cloud is therefore a representation, not a sensor. PCL notes that clouds can come from stereo cameras, 3D scanners or time-of-flight cameras, or be generated synthetically. The same kind of cloud can come from very different hardware, so for training data what matters is how it was derived and how well it is calibrated.

Stereo video gives two views of one scene. Matching pixels between the views gives disparity, and with a measured baseline and focal length, disparity becomes depth (see stereo disparity and stereo matching). Firsthand derives depth this way, from a head-mounted calibrated stereo rig rather than a standalone depth sensor, and delivers it in metric millimetres. A point cloud is one more step. Open3D documents it in create_from_depth_image: for a depth value d at pixel (u, v), z = d / depth_scale, x = (u − cx) · z / fx and y = (v − cy) · z / fy, with an optional extrinsic transform. Its default depth_scale is 1000, which suits millimetre input and returns metres. The inputs on the Firsthand side are depth_mm plus the intrinsics and extrinsics shipped in each episode's calibration group. Which intrinsics apply to the 848 × 480 depth grid is Not stated, so confirm it in your spec. Firsthand point-cloud file delivery is Not stated; a buyer can compute a cloud from the delivered depth and calibration.

A depth map is an image: one distance per pixel on a regular grid, so every pixel's neighbours are known. A point cloud drops the grid. Each valid pixel becomes a 3D point, pixels with no valid depth fall out (Open3D's project_valid_depth_only option, on by default, does this), and order stops mattering. The cloud can then be cropped, merged with other views or moved with a 4 × 4 transform, as Open3D's crop and transform methods do.

The trade is information. A cloud no longer says which pixel a point came from unless you keep the index, and Firsthand's per-pixel depth_conf confidence channel lives on the image grid. If you want to filter by confidence, do it on the depth map before converting. A disparity map sits one step earlier, as pixel shifts rather than distances; a mesh sits one step later, with surfaces built from points. The second table compares all four.

Point clouds feed policies that reason in 3D. The 3D Diffusion Policy paper (arXiv 2403.03954) is one example. It converts depth images to point clouds using camera extrinsics and intrinsics, crops away points such as the table and ground, downsamples with farthest point sampling, and encodes the result with a small MLP into a compact feature for a diffusion policy. In its ablation on six tasks it compares point clouds with image, depth, RGB-D and voxel inputs and reports point clouds ahead. That result belongs to those tasks and that setup, not to any dataset; Firsthand states no benchmark for point-cloud learning.

The same pipeline shows what a dataset must supply: depth that is metric, calibration that is current, and one coordinate frame. Open3D's tutorial covers the usual clean-up steps, including voxel downsampling, normal estimation, cropping and plane segmentation, and all of them assume the geometry is right to begin with.

The site states depth, calibration and one point-cloud entity; the first table lists each with its format. Depth is 848 × 480 at 30 Hz in metric millimetres, derived from the stereo rig, registered to the head-camera frame with the solved extrinsics and on the same clock as the video. A per-pixel confidence map ships beside it.

The one explicit point-cloud mention is the Rerun .rrd recording, whose entities are world/head_cam, world/hands, world/points and plots/sync_error. It is for inspecting an episode before you write a loader, not for training. The site does not state how world/points is built or at what density, and it is not an itemised stream in the episode schema. Anything further about point clouds is agreed in the spec.

Collection is custom, on demand and fast to specify: write down the tasks, environments and objects, and Firsthand collects to a written skill spec. If your pipeline needs point clouds, say so in the request. State whether depth plus calibration is enough, whether you want the Rerun view, or whether you need a specific point-cloud file format, and name the density, range and accuracy you need, because the site states none of them.

Delivery formats the site states are HDF5, RLDS, LeRobot-style and WebDataset. Use the request button below to start a custom dataset.

What is a point cloud in robot training data?

A set of 3D points, each with X, Y and Z coordinates, sampling the surfaces a camera saw. It can be computed from a depth map plus camera calibration.

Does Firsthand deliver point clouds?

Point-cloud file delivery is Not stated. The site states depth_mm plus calibration, and a world/points entity in the Rerun .rrd recording whose derivation and density are Not stated.

Does Firsthand use a depth sensor to get 3D?

No. Depth is derived from a head-mounted calibrated stereo rig, not from a standalone depth sensor, and it is delivered in metric millimetres.

How do I turn depth_mm into a point cloud?

Apply the pinhole formula Open3D documents: z = d / depth_scale, x = (u − cx) · z / fx, y = (v − cy) · z / fy. With millimetre depth, a depth_scale of 1000 gives metres. Read the intrinsics from the episode calibration.

Is a point cloud better than a depth map for training?

It depends on the model. The 3D Diffusion Policy paper reports point clouds ahead of depth and RGB-D inputs on its tasks, but other methods consume depth images. Firsthand states no comparison.

What is the difference between a point cloud and a mesh?

A point cloud is an unordered set of points. A mesh is a connected surface, and PCL lists surface reconstruction from point clouds as one of its algorithm families.

What point density or accuracy does the data have?

Point density, depth range and depth accuracy are Not stated. Name the figures you need in the spec before volume is committed.

How can I order custom stereo egocentric data quickly?

Request a custom dataset with your tasks, environments and the depth or 3D outputs you need. Collection is on demand and to a written skill spec.