Stereo Egocentric Data for Construction Robotics

It is first-person video from two calibrated cameras, recorded from a worker’s own viewpoint while they do real jobsite and trades work, with hand pose and sensor streams alongside. “Stereo” describes the capture: two cameras at a fixed spacing, so depth is recoverable from every frame pair. “Egocentric” describes the viewpoint. The data is meant for models that have to see a stud bay, a drywall seam, a taped edge or a machine control station the way the person doing the work sees it.

This guide is the hub for the construction cluster on this site. It ties together the construction crews page and the task-family pages for machinery, finishing, marking and taping, floor care and inspection, adds the MCAP delivery page and glossary entry, and shows where the off-the-shelf industrial catalog and custom collection each fit. Every figure comes from a page already on the site. Where the site does not state something, this guide says “Not stated” instead of filling the gap.

Firsthand is not affiliated with Foxglove or the MCAP maintainers. Facts about the MCAP container come only from its own documentation, linked in the sources below.

The cluster has one crews page, six task-family pages and two supporting pages. Each task page covers the stereo and container side of one family of work: what a calibrated pair adds, what the catalog lists, what the site does not state, and how to order it. The table maps them. Some rows, such as drywall taping, appear on more than one page, so do not add the row counts together.

The site states that a stereo pair at human-eye spacing makes depth recoverable from every frame pair, and that stereo keeps working outdoors where active depth sensors struggle. On a construction site the geometry is the point: a stud bay, a panel edge, a joint or a tile line is a three-dimensional thing, and the same frame pair that shows the hands working it also carries its shape. The full comparison is in stereo vs monocular video for robot learning, and the depth side is in depth data for embodied AI.

Depth is not the only reason. Construction work is hands-on in a way that hides the evidence: a hand covers the spot being marked, a taping knife covers the joint, a handle covers the fingers. The next table lists the failure modes the site names for this kind of capture and how each is handled.

The site describes two stereo setups and does not say which one a construction order uses. The off-the-shelf stereo head rig, dataset 06, is a dual-fisheye rig. The hard-environments stack is a head camera with wrist cameras, a depth sensor and IMUs. The catalog lists the capture tier on every construction row as a head-mounted phone or multiview rig, so a stereo pair is a capture choice agreed in the skill spec and confirmed on a Spec Pilot, not something already running on those rows.

For rig setup, calibration and sync in depth, see stereo camera rig setup for robotics data and how we measure sync error.

The construction crews page covers the 15 task rows listed under the Active Construction Site environment: carpentry, woodworking, general skilled trades, debris cleanup, insulation, drywall, electrical installation, painting and priming, plumbing, lumber cutting and tiling. Four rows are marked Ready to deploy and eleven are Active collection. By difficulty, six are Hard, seven are Medium and two are Easy. The environment itself is described on the Active Construction Site page.

Capture is planned around the site’s own layout and schedule rather than asking the work to pause. Paid, consented contributors do the actual task with other trades and equipment present. The spec names the task once and several contributors perform it, so real variation in technique is represented. Failure and recovery cases, such as a misaligned fastener or a redo of a measuring step, sit in the same labeled structure and can be required for a batch. The related construction and skilled trades data page covers the same trades without the stereo framing.

Machinery work is the one task family here that is recorded from a fixed operator station rather than from a moving body. The site lists one catalog row, Operating Heavy Machinery Controls: Heavy Industry & Fabrication, rated Hard and available on request. The heavy machinery operation page describes first-person controls video from a calibrated pair, with depth and hand pose alongside, so the operator’s hands on levers and panels are recorded in three dimensions.

The same limits apply as elsewhere in the cluster. The catalog lists a phone or multiview rig as the capture setup, a stereo pair is agreed in the skill spec, and MCAP topic names, schemas and rates are Not stated and agreed per order. The page does not claim stereo is already running on that row.

Finishing is a run of ordered passes over a surface that changes as you go. The surface finishing page covers nine catalog rows across sanding, drywall and painting. Here the surface is the subject, and that is where depth and its failure modes matter most. The site says depth degrades on wet and reflective surfaces and handles that with per-frame confidence maps that mask low-confidence regions. Fresh compound, paint and stain are wet surfaces, so the mask is part of the delivery, not an afterthought.

The catalog describes the steps, not a finish-quality standard. What counts as an acceptable pass is written into the accept criteria of the skill spec, and episodes that fail QA go to the reject log and are recaptured rather than shipped with a caveat.

Layout work is a run of small, ordered moves that fix everything that follows: hook a tape and read the number, mark the line, lay out positions from a plan, tape a drywall joint, mask an edge before paint. The layout marking and taping page quotes eleven catalog rows: eight measuring, marking and layout rows, two drywall taping rows and one masking row.

One hand often holds the reference or the tape while the other marks or knifes the joint, which is the hand-hiding case from the failure-mode table. The visibility flag turns that into a labeled fact. The site gives no depth accuracy figure for these rows, so a buyer who needs one should write it into the spec and check it on the Spec Pilot.

Both are adjacent to construction rather than part of the Active Construction Site rows, and the site covers them on their own pages. The floor mopping page quotes nine mopping, floor-care and janitorial rows, which is floor-care work rather than a construction trade. The surface quality inspection page quotes nine inspection rows, including weld inspection and room turnover and inspection, and covers a person inspecting parts and items against a standard.

The site gives no depth accuracy figure or pass threshold for these rows, so the standard is written into the skill spec. For how rejection works generally, see robotics training data rejection criteria.

Firsthand offers MCAP delivery. The off-the-shelf stereo head rig ships as MCAP, and a custom order is packaged in the format you name. According to its own documentation, MCAP is an open source container for multimodal log data with multiple channels of timestamped, pre-serialized data, so a stereo pair, hand pose and IMU can share one file on one log-time axis. The MCAP delivery page covers the format in full and the MCAP glossary entry defines it.

What the site does not state is just as important for a reader. MCAP topic names, message schemas, encodings, rates and files per episode are Not stated and are agreed in the skill spec, along with which streams, such as depth, wrist cameras or action segments, go in the file. If your training loop reads RLDS, LeRobot or WebDataset, name that beside MCAP: the same episodes can ship in more than one format. See pretraining data delivery formats and the formats hub.

Off-the-shelf covers what is already in stock. The industrial and skilled-trade egocentric video category lists datasets 02, 03 and 04, covering work such as machinery, auto repair, carpentry, construction, factory floors and inspection. The stereo and hand-tracking category lists the stereo head rig. Custom collection covers the gaps: custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs. The data supply page explains how the two sources hold one spec and one QA bar across batches.

The bottleneck the site states for construction is vetting and scheduling operators with the right site access, not demand for the task, which is why it is quoted up front as the longer timeline. The page makes no claim about delivery cadence or volume per week or month.

For how to write the accept criteria that the QA gate checks against, see how to write a skill spec and the skill spec glossary entry.

Most orders begin with a Spec Pilot. Before you request one, settle the items below. The skill spec fixes them, and the pilot confirms them on real footage.

• The task rows you need, from the crews, machinery, finishing, marking and taping, floor care or inspection pages
• The setting and any site-access constraints, which drive the first-batch timeline
• The rig and streams: stereo pair, wrist views, hand pose, IMU and anything else you want in the file
• The delivery format, including MCAP if your reader expects it, and which streams go in the file
• The accept criteria for what a finished pass, a marked line or a passed inspection looks like
• Whether you want off-the-shelf industrial video for immediate volume alongside the custom batches
• The licence terms, including whether you want exclusivity