Egocentric Audio and Video Data Collection

Egocentric audio and video data collection is the recording of first-person task footage, from a head-mounted camera, together with the sound of the same moment. The video says what the wearer saw and did; the audio adds what could be heard while they did it.

The site is specific about how audio is captured and quiet about the instrument. Collection pages state that audio is hardware-triggered together with depth and IMU, and that head-mounted and wrist video, depth, IMU and audio are hardware-synchronized. They do not name a microphone, a channel count, a sample rate or a codec, so this page makes none of those claims.

This page is for robot learning, VLA, humanoid and multimodal teams that want to specify audio and order it collected, fast and on demand, as a custom collection. It is separate from the standalone speech service, which records speakers for ASR and voice models rather than audio beside first-person task video.

Many tasks make sound that carries information the picture does not: a tool starting, a container closing, a pour. The site’s glossary defines multimodal training data as two or more aligned streams, naming video, depth, audio, IMU and pose, captured from the same event and time-synchronized so a model can learn how they relate rather than finding them side by side in a folder.

The site also notes that streams which drift against each other, even slightly, break the cross-modal signal a model is meant to learn, and that hardware-triggering every stream onto one clock is what keeps a dataset genuinely multimodal.

The site does not state a measured benefit from adding audio to a training mix, so none is quoted here. Whether audio is worth the added review for your policy is a decision to test on a small first batch.

Paid, consented contributors wear the rig and perform the real task against your spec. The site states that the cameras hardware-trigger together with depth, IMU and audio, so every stream lines up to the frame, and that the same holds for stereo capture, where left and right frames trigger on one clock alongside depth, IMU and audio.

Large-scale collection pages list the streams as head and wrist RGB, depth, IMU and audio, with audio marked optional. That is the clearest statement of scope on the site: audio is available, and it is something you ask for in the spec rather than something every delivery carries.

The site does not state where the microphone sits, how many channels are recorded, or at what sample rate. Those details are set per program in the skill spec and confirmed on the first batch.

The stream table lists what travels with audio, using the rates the capture and episode specs state. The audio row is deliberately thin: the site states that audio is triggered with the other streams, and does not itemise its data type, rate or delivery key.

Because the streams share one timeline, an audio event can be read against the head camera at 60 fps, the 200 Hz head IMU and depth at 30 fps for the same moment.

The site’s audio and speech solution records people speaking: conversational and scripted speech, multilingual and accented speech, wake words, and ambient or event audio, from screened speakers in the languages and conditions you specify. That service is about the voice.

Egocentric audio is about the task. The audio arrives as one stream of a first-person episode, triggered with the camera, depth and IMU, from a contributor doing the work. The comparison table sets the two side by side. If you need speech models, scripted prompts or speaker-level metadata, the standalone service is the right page; if you need audio that lines up with first-person video, this is.

Every sensor runs in one PTP clock domain and exposures are hardware-triggered. For the rig, the site states a 1.12 ms median offset, 1.94 ms at the 99th percentile and a 2.0 ms ceiling, measured as the absolute timestamp offset between the head camera and every other stream.

The streams run at different rates: cameras at 60 fps, depth at 30 fps and the IMUs at 200 Hz. The site does not itemise a separate audio-to-video offset or an audio sampling rate, so how audio samples are aligned to a video frame is set per program in the skill spec.

A head-mounted rig hears whoever is nearby, so audio is treated the way video is. The site states that speech is scrubbed of names, addresses and card numbers while ambient audio is retained, and that a human reviewer confirms that no un-consented person is identifiable.

For the delivered files, the site describes spoken identifiers as scrubbed from the transcript and muted in the waveform, and lists muted or scrubbed waveform segments in the delivered audio track. In the same pass, faces, licence plates and screens in frame are detected and irreversibly blurred at ingest. Any clip where a bystander cannot be de-identified is discarded rather than shipped.

The site does not state whether a transcript is delivered as a separate file for egocentric episodes. That is set per program.

All formats are exported from the same QA-passed episodes as projections of the canonical episode schema. The site’s format pages describe how video, depth, IMU, hand joints and actions are carried in RLDS, LeRobot, WebDataset and HDF5 or zarr, but they do not state an audio key or an audio file layout.

Where audio sits in each layout, and whether it is embedded in the episode or delivered as separate files, is therefore set per program in the skill spec. The site’s standalone speech service lists WAV or FLAC with per-file metadata; the site does not say the same applies to egocentric episodes, so confirm the audio container in the spec.

Paid, consented contributors perform the real task against your spec while wearing the rig. We do not capture robot or teleoperation data, and we do not capture at a customer’s own facility. Every episode is tied to a signed contributor release.

The dataset ships under a perpetual, irrevocable, worldwide commercial licence that is non-exclusive by default, with the buyer owning the resulting dataset and exclusivity available. Where coverage is deep, the median time from signed spec to first validated batch is a median of 48 hours; where it is thin, turnaround runs three to four weeks. Billing is per validated hour, not per raw hour recorded, and there is no list price: you are quoted against your spec.

Write down the tasks, environments and objects your model needs, say whether audio is in scope and what you need from it, and name the delivery format. The site recommends that an RFP name the sensor streams a task requires, including audio, and the synchronization tolerance they must meet.

Most orders begin with a Spec Pilot: a small, fast first batch that proves the spec and accept criteria hold on real footage before you scale.