Stereo Lab Pipetting and Sample Handling Data

It is first-person video of a person doing bench work, recorded with two calibrated cameras, with hand pose and sensor streams alongside. People reach it through several phrasings: lab automation robot training data, stereo pipetting video, egocentric liquid handling dataset, or custom laboratory data collection at scale. They describe one request from different angles. Stereo describes the capture, and MCAP describes the container. They are separate choices.

The site does not state that stereo is already running on these rows. The catalog lists a head-mounted phone or a multiview rig as the capture setup for the Pharmacy & Laboratory rows, so a stereo pair is a capture choice agreed in the skill spec and confirmed on a Spec Pilot. The data is human capture by paid, consented contributors, not robot or teleoperation data, so using it for a specific robot still needs your own retargeting step. Firsthand is not affiliated with Foxglove or the MCAP maintainers.

Seven, listed in lib/commercial-catalog-data.ts. Three sit in Pharmacy & Laboratory, in the Healthcare & Life Sciences sector: "Pipetting" and "Liquid handling", both rated Medium, and "Lab protocols (pipetting, titration, sample handling)", rated Hard. Four sit in Wet Lab & Clinical Bench and cover sample tubes: "Fill + cap sample tubes", "Cap, vortex + rack tubes" and "Load + balance centrifuge", rated Medium, and "Rack + label tubes", rated Easy. All seven are listed as Available on request.

The site defines Available on request as work that a spec starts, rather than a running capture stream a spec can join. That is a statement about the catalog row, not a promise of a delivery date. The site states no delivered hours and no delivered stereo footage for any of these rows, so this page claims none. Other lab rows, such as "Laboratory Work", "Weigh reagent", "Prepare a slide", "Swab + streak plate" and "Titrate to endpoint", are real catalog rows but are outside this page. The first table below reproduces the seven rows as the catalog states them.

This page is the stereo and MCAP treatment of lab pipetting, liquid handling and sample handling. The egocentric lab pipetting and sample handling page covers fourteen catalog rows, five from Pharmacy & Laboratory and nine from Wet Lab & Clinical Bench, as first-person video, and stereo is not its subject. It is the non-stereo page for the same bench work, and it also reproduces rows this page leaves out, such as weighing a reagent and titrating to an endpoint.

The egocentric medication sorting and dispensing page covers three medication rows across Care Room, Nursing Home & Assisted Living and Pharmacy & Laboratory, all Available on request. That is pharmacy work with blister packs, charts and prescriptions, not liquid transfer at a bench. The second table below sets the pages side by side with each status as the site lists it.

The catalog lists "Phone (head-mounted) / multiview rig" for the three Pharmacy & Laboratory rows and "Multiview rig + handheld grippers" for the four sample-tube rows. The site does not state a stereo pair for any of them. Stereo applies through the site’s general stereo offer: a stereo pair is a capture choice agreed in the skill spec and confirmed on a Spec Pilot. The site also states that Firsthand does not capture at a customer’s own facility, so the setting is agreed in the spec.

The site describes two stereo setups and does not say which one an order for these rows uses. The hard-environments stack is a global-shutter 4K/60 head camera with a 120° lens, two 1080p/60 wrist cameras, a depth sensor at 848×480 and 30 fps, and 200 Hz 6-DoF IMUs. The off-the-shelf stereo head rig, dataset 06, is dual fisheye cameras at 1080p per eye, a 62 mm baseline, per-unit calibration, a 9-axis IMU and hand-tracking output. For the sample-tube rows, the handheld grippers named in the catalog are part of the stated setup; whether a stereo pair is worn, mounted or combined with them is Not stated.

Bench work is about where a tip meets a tube: how the pipette is held, where the plunger sits under the thumb, how a tube is steadied while it is filled or capped. A stereo pair recovers depth from the calibrated frame pair rather than from a standalone depth sensor, so the geometry sits in the same capture that shows the hands working. The site gives no depth accuracy figure for these rows, so what a buyer does with that depth is their pipeline’s question.

A stereo pair adds depth, not a verdict. The site lists no volume, liquid level, sample identity or result label for these rows, and none is claimed. The egocentric lab pipetting page already supplies the first-person framing; stereo is the addition for a buyer who wants the depth and the MCAP file.

The rows give a natural cut. The pipetting row draws liquid into the pipette, dispenses exact volumes into tubes and changes tips. The liquid handling row measures and transfers liquids between vessels with pipettes, cylinders and dispensers. The tube rows fill to a mark, cap, vortex-mix, rack in order, label and load tubes balanced in a centrifuge. A spec that says "do the protocol" gives one long action segment. A spec that names the steps gives a sequence you can slice by step.

Whether drawing and dispensing are one segment or two, and whether capping and racking are one segment or two, is a spec decision, not a default. The site does not set an episode length, so the spec says whether one episode is one transfer, one tube, one rack or a whole protocol run. Contact events can mark when a hand meets an item, where the spec asks for them, and the spec decides what counts as the item, including whether a pipette tip or a tube cap qualifies.

A pipette, a thumb and a tube can all sit between the lens and the work. The site states that the hands occlude the work and that fingers disappear behind objects. That is why each hand joint carries a visibility flag, why contact events mark when the hand meets the item, and why instance masks are recorded per frame. It also states that reach and grasp frames are where blur and occlusion show up, and the head camera in the hard-environments stack is global-shutter, which avoids rolling-shutter skew.

The site does not state how depth behaves on small, shiny or transparent items, and the catalog row text does not describe the tube or vessel material. It states that depth degrades on wet and reflective surfaces, with confidence maps masking low-confidence regions. The site’s tool use page says grip changes and tool hand-off live in finger-level detail that RGB video alone cannot resolve, and that metric depth on sub-centimetre parts is flagged rather than interpolated. That page is about driving and fastening tools, not pipettes, so this page does not claim the same findings for lab work. If your policy depends on glass labware, liquid in a vessel or a transparent tube, say so in the spec and settle it on a Spec Pilot.

The site states that glare-heavy and low-texture scenes make single-frame depth worse, and that depth degrades on wet and reflective surfaces. A benchtop, a metal rack and a glass vessel are plausible cases, but the site does not say how often they occur or how depth behaves on them. A lighting setup for the lab rows is Not stated.

Where a program needs a specific bench layout, lighting or background, that goes into the spec and is checked on a Spec Pilot. An episode is rejected when more than 35 percent of depth pixels in the manipulation region are invalid, and for this job the spec decides what the manipulation region is at each step. Equipment, consumables, assays and protocols beyond the pipettes, cylinders, dispensers, tubes, centrifuge and rows named in the catalog are Not stated.

The catalog notes that the Pharmacy & Laboratory work requires regulatory compliance, and a separate catalog row lists putting on gloves and eyewear, handling chemicals under the hood and disposing of waste correctly. This page makes no GLP, GMP, biosafety, chemical safety, sterility or certification claim of its own: all of those are Not stated. The site states no patient-data handling for these environments, and this page covers no patient-facing task.

For every delivered episode the site states that faces and screens are blurred irreversibly, that a reviewer confirms no un-consented person is identifiable, and that footage that cannot be de-identified is discarded. The anonymization record ships with the data, an undocumented bystander is a stated reject criterion, and every episode links to a signed contributor release. Tube labels, sample labels and container markings are not listed as blur categories, so this page does not claim they are blurred. If label text in frame matters, raise it in the spec. This is not legal advice, and the executed agreement controls.

MCAP is an open source container file format for multimodal log data, supporting multiple channels of timestamped pre-serialized data. It is serialization-agnostic, so messages can be Protobuf, DDS (CDR), ROS, JSON and more. Its design is row-oriented and append-only, it stores schemas alongside the data, and it offers an optional index plus optional LZ4 or Zstandard chunk compression.

Reader and writer libraries exist for C++, Go, Python, Rust, Swift and TypeScript, and the mcap CLI can inspect, merge and split files. The foxglove/mcap repository is MIT-licensed; that is the MCAP project licence, not the licence of a Firsthand delivery. The MCAP hub page and the glossary entry cover the format in full.

Firsthand offers MCAP delivery, and dataset 06 is stated as shipping as MCAP. For these rows the site states hand pose, a stereo-derived depth stream with confidence maps, IMU, contact events, action segments, per-frame instance masks and one task instruction per episode elsewhere. What it does not state is which of those are channels in an MCAP file, how they are named, what encoding they use or at what rate.

No topic name, message schema or rate is stated on this page, and none should be assumed. Name the streams that matter in the skill spec and confirm them on a Spec Pilot batch. The fourth table below sets each stream beside what the site states about it.

Hand occlusion by the pipette and tube is the first risk, under the visibility flag above. Weak depth on glare-heavy, wet or reflective surfaces is the second, and the behaviour on small transparent items is Not stated. Calibration is measured at the start and end of every session under a 0.28 px RMS gate, and an episode is rejected for a sync breach beyond the ceiling for more than three consecutive frames, for the manipulated object out of frame for more than 15 percent of the episode, for an undocumented bystander or for a mismatch against the named skill spec.

A reject log ships with every batch, and rejected items are recaptured at the provider’s cost. The fifth table below lists the failure modes beside what the site states.

The October 2026 off-the-shelf catalog lists ~324,800 hours across 18 datasets, of which stereo is ~64,800 hours across 9 datasets. The site names no lab pipetting or sample handling dataset among them, so this work is custom collection, not an off-the-shelf purchase. The off-the-shelf stereo and hand-tracking video can still show the format before you write a spec.

It is custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs. Where coverage is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, turnaround runs three to four weeks. The site does not say which applies to the lab environments, so the timeline is quoted against your spec. Collections are billed per validated hour, and rejected episodes and recaptures are never billed.

Describe the steps and where each starts and ends: drawing, dispensing, changing the tip, filling, capping, racking and labeling. Say whether an episode covers one transfer, one tube, one rack or a whole protocol run, what counts as the object and what counts as a finished step, the privacy rule for the space, and name the variation your pipeline is missing. Then name the streams you want in the file, the topic naming and message encoding your reader expects, whether calibration ships inside the file or beside it, and whether MCAP is the only delivery or sits beside RLDS, LeRobot or WebDataset.

Most programs begin with a Spec Pilot, a small first batch that proves the spec and accept criteria on real footage before you scale. The sixth table below shows what each spec element decides.