Stereo Pharmacy Medication Dispensing and Labeling Data

It is first-person video of a person doing counter work in a pharmacy, recorded with two calibrated cameras and with hand pose and sensor streams alongside. The tasks are the two rows the site’s catalog lists in Pharmacy & Laboratory that name this work: reading the prescription, counting or measuring the medication, and packaging and labeling it; and printing and applying labels to tubes, bottles and dispensed medication and checking them. Stereo describes the capture, and MCAP describes the container. They are separate choices.

The site does not state that stereo is already running on these rows. The catalog lists a head-mounted phone or a multiview rig as the capture setup. Stereo applies only through the site’s general stereo offer: a stereo pair is a capture choice agreed in the skill spec and confirmed on a Spec Pilot. The data is human capture by paid, consented contributors, not robot or teleoperation data. Firsthand is not affiliated with Foxglove or the MCAP maintainers.

Two, both in Pharmacy & Laboratory under Healthcare & Life Sciences, in the catalog data file lib/commercial-catalog-data.ts. "Medication dispensing" is described as "Read the prescription, count or measure the medication, package and label it.", is rated Medium and is listed as Available on request. "Labeling" is described as "Print and apply labels to tubes, bottles and dispensed medication and check them.", is rated Easy and is listed as Available on request. Both list a head-mounted phone or a multiview rig as the capture setup.

The site says nothing further about what Available on request means in volume, hours or delivered stereo footage, so this page claims none. The first table below reproduces the rows as the catalog states them. The catalog names no medication, no container size, no label format and no counting or measuring tool, so this page names none either.

The site defines Available on request as scoped and recruited to spec when a buyer asks for it, so recording is not already running for these rows. That differs from Active collection, which the site defines as recording that is running now, so a spec can join an existing capture stream. Neither row carries that label, and neither is listed as Ready to deploy.

In practice a stereo pharmacy order starts from your written spec rather than from an existing stream. The site states no delivered hours and no delivered stereo footage for either row, so this page makes no such claim. What a given order delivers, and when, is quoted against your spec.

One row sits next to this work in the same environment. "Safety protocol adherence" is described as "Put on gloves and eyewear, handle chemicals under the hood and dispose of waste correctly.", is rated Medium and is listed as Available on request, with the same head-mounted phone or multiview rig setup. It is context only: this page builds nothing on it, and makes no safety claim.

The same environment also lists pipetting, liquid handling, lab protocols and laboratory work. Those are laboratory bench rows, and the stereo lab pipetting page treats the stereo side of them. Keeping them out of this page keeps its status and capture statements true to the two rows named above.

Two egocentric pages already cover these rows by task. The medication sorting and dispensing page spans three medication rows in three environments, adds the Labeling row as an adjacent row, and treats capture as egocentric. The pharmacy and lab bench page spans twenty-one rows in two environments, from dispensing to pipetting, and also treats capture as egocentric. Neither centres stereo capture of the two counter rows, and neither goes into the MCAP container.

Two stereo pages sit nearby. The stereo lab pipetting and sample handling page shares this page’s stereo and MCAP treatment but covers bench rows for pipetting and sample tubes. The stereo product labeling page covers a different "Labeling" row, in Plastic & Light-Assembly Manufacturer, which the catalog lists as Ready to deploy. This page spans only the two pharmacy rows. The comparison table below sets them side by side.

Counter work is about where the hand, the container and the label sit relative to one another: how a vial is held against a label, where a label lands on a bottle, how a package sits in the hand. A stereo pair recovers depth from the calibrated frame pair rather than from a standalone depth sensor, so the container, the label and the hands share a frame.

The site gives no depth accuracy figure and no label for a label placed correctly, a count reached or a package closed, so whether those outcomes are resolvable in depth is Not stated. Capture at your own facility is listed as not offered.

The catalog states one capture setup for both rows: a head-mounted phone or a multiview rig. The site describes two stereo setups and does not say which one a pharmacy order uses. The hard-environments stack is a global-shutter 4K/60 head camera with a 120° lens, two 1080p/60 wrist cameras, a depth sensor at 848×480 and 30 fps, and 200 Hz 6-DoF IMUs. The off-the-shelf stereo head rig, dataset 06, is dual fisheye cameras at 1080p per eye, a 62 mm baseline, per-unit calibration, a 9-axis IMU and hand-tracking output.

Which rig, resolution and streams a dispensing or labeling order uses is Not stated on the site, and the third table below sets what is stated beside what is not. Intrinsics and extrinsics are re-solved at the start and end of every session, and streams are hardware-triggered onto one clock with a 2.0 ms cross-stream ceiling. The catalog states that a multiview rig adds synchronized third-person angles when the spec asks for them, and wrist cameras give a closer view of the hands. The site does not state a pharmacy lighting setup, so lighting is Not stated and is agreed in the skill spec.

MCAP is an open source container file format for multimodal log data, supporting multiple channels of timestamped pre-serialized data. It is serialization-agnostic, so messages can be Protobuf, DDS (CDR), ROS, JSON and more. Its design is row-oriented and append-only, it stores schemas alongside the data, and it offers an optional index plus optional LZ4 or Zstandard chunk compression.

Reader and writer libraries exist for C++, Go, Python, Rust, Swift and TypeScript, and the mcap CLI can inspect, merge and split files. The foxglove/mcap repository is MIT-licensed; that is the MCAP project’s licence, not the licence of a Firsthand delivery. The MCAP hub page and the glossary entry cover the format in full.

Firsthand offers MCAP delivery, and dataset 06 is stated as shipping as MCAP. For counter work the site states hand pose, depth recovered from the stereo pair, IMU, contact events, action segments and per-frame instance masks elsewhere. What it does not state is which of those are channels in an MCAP file, how they are named, what encoding they use or at what rate. The only language data tied to Firsthand capture is one task instruction per episode.

No topic name, message schema or rate is stated on this page, and none should be assumed. Name the streams that matter in the skill spec and confirm them on a Spec Pilot batch. The fourth table below sets each stream beside what the site states about it.

Hand occlusion of small containers and labels is the first risk. The site states that fingers disappear behind objects, and a hand closing around a vial or a bottle while a label is pressed on is partly hidden from the head camera. The visibility flag on each joint is the answer the site gives for hidden fingers, contact events mark when a hand meets an item, and a spec can ask for wrist views so a grip the head camera loses is still recorded. A reject rule applies as well: an episode is rejected when the manipulated object is out of frame for more than 15 percent of it.

Small, similar-looking items, fine label placement and reflective or transparent packaging are the second. The site states no label for one item among several alike, none for where a label sits, and no depth figure at label scale, so what counts as one counted item or one correctly placed label is written into the spec. The site names reflective surfaces and clutter as cases where a single camera struggles in hard environments, and notes that wet and reflective surfaces degrade active-stereo depth in its kitchen dataset. Whether glossy labels or transparent packaging behave the same on a stereo pair is Not stated, and so is the effect of counter lighting. A cluttered counter is a clutter case in the same sense.

Calibration is the third, because stereo depth is only as good as the rectification behind it: calibration is measured at the start and end of every session under a 0.28 px RMS gate. A reject log ships with every batch. The site lists no count or measure tolerance and no force or tactile stream, so neither is claimed here.

Counter work chains steps that look alike on camera: counting one item and then the next, packaging one order and then another, applying one label and then checking it. The "Medication dispensing" description blends several verbs into one row, read, count or measure, package and label, so action segments, which carry start, end, verb and noun, are where the spec splits them and decides which object is the noun.

The site states no episode length, no segment count and no per-item label for these rows, and the catalog does not split either row into finer steps. Whether an episode is one dispense, one label, one order or a longer counter sequence is set per program. Put that boundary and the verbs you expect into the skill spec, and check them on the Spec Pilot before you scale.

For every delivered episode the site states that faces, plates and screens are detected and irreversibly blurred at ingest, that a reviewer confirms the result before anything ships, that any identifiable un-consented person goes to human review, and that footage that cannot be de-identified is discarded. Every episode links to a signed contributor release through a documented consent chain, and the consent record and the anonymization record are delivered with the data.

The rows involve reading a prescription and applying labels, so text is in frame. The site states blurring for faces, plates and screens; whether printed label or prescription text is blurred is not stated, and it is agreed in the spec. The medication sorting page states that healthcare capture uses simulated patients only; who or what is in frame for these two rows is agreed in the spec. This page makes no medication-safety, clinical, regulatory or compliance claim, because the site states none for these rows, and no rule for data stored on a device is stated. Pharmacy, client and brand names, medication names, task times, throughput and error rates are Not stated.

Off-the-shelf covers what is already in stock: the stereo category lists the stereo datasets and hand-tracking video, and only dataset 06 is stated as MCAP. This page does not claim an off-the-shelf dataset of medication dispensing or labeling, so this is custom collection: custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs.

Where coverage is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, turnaround runs three to four weeks. The site does not say which speed applies to a stereo pharmacy order, so ask when you request the quote. Collections are billed per validated hour, rejected items are recaptured at our cost, and on-demand top-ups run against the same spec and accept criteria after each training run.

Describe the counter sequence and its order, the containers and labels involved, what counts as a finished dispense or a correctly applied label, who or what is in frame, and the variation your pipeline is missing. Then name the streams you want in the file, the topic naming and message encoding your reader expects, whether calibration ships inside the file or beside it, and whether MCAP is the only delivery or sits beside RLDS, LeRobot or WebDataset.

Most programs begin with a Spec Pilot, a small first batch that proves the spec and accept criteria on real footage before you scale. The fifth table below shows what each spec element decides.