What Frame Rate Is Used for Egocentric Robot Training Video?

Frame rate sets the time between the samples a model sees. Reach, grasp and release happen quickly, and a lower rate leaves longer gaps in which a hand can move a long way. A higher rate samples that motion more finely and gives labels, such as hand pose or contact, more frames to attach to, but it also means more data to store, move and decode. Frame rate is only one setting: shutter type decides whether each frame is clean during fast motion, and clock synchronization decides whether frames from different cameras describe the same instant.

Firsthand states a head camera at 3840 × 2160 and 60 fps, two wrist cameras at 1920 × 1080 and 60 fps, depth at 848 × 480 and 30 fps, and 6-DoF IMUs at 200 Hz. Each stream keeps its own native rate and is delivered with its own int64 nanosecond timestamps on the PTP clock domain, so any resampling between rates happens in the loader, not in the delivery. The frame rate of the hand pose and body pose labels is Not stated, as are frame-drop rates and the rates of any other camera or rig.

Capture frame rate is how often a camera records a frame, usually written as frames per second (fps) or hertz (Hz). A camera running at 60 fps records sixty frames each second, one every sixtieth of a second.

It is easy to confuse with nearby ideas. Playback frame rate is how fast a player shows frames. Sampling rate is the general term for non-camera sensors, such as an inertial sensor, written in hertz. Annotation rate is how often a label is attached. Ask for the rate of each stream separately.

There is no single correct answer, and this page does not name a standard one. The right rate depends on how fast the task is, what the model must read from each frame, and how much data a team can store and process. Fast reach, grasp and hand-over motion argues for a higher rate; slow, deliberate work tolerates a lower one.

Treat the rate as a collection requirement: write the target for each stream into the request before capture, then check the delivered timestamps against it. What Firsthand states for its standard rig is in the table below; anything outside it is Not stated.

Frame rate only describes how often frames are taken. A global-shutter camera exposes every pixel at the same instant, while a rolling shutter reads the frame row by row and skews fast motion. Firsthand states a global shutter for its head camera, so each of its 60 frames per second stays geometrically consistent during fast motion. The shutter type of the wrist cameras and the depth sensor is Not stated.

Synchronization decides whether frames from different cameras line up. Firsthand states that its streams are hardware-triggered onto one PTP clock domain, with int64 nanosecond timestamps per stream. Two cameras at the same nominal rate can still disagree about when a frame was taken without a shared clock. The same applies to stereoscopic capture: the two views are only comparable if they were exposed together.

Each stream runs at its native rate. Firsthand states the head camera at 60 Hz and the IMU at 200 Hz, so an episode has more IMU samples than video frames. In the HDF5 layout the video arrays are shaped [T, ...] and the IMU arrays [N, 6]; T and N differ because the streams run at different native rates, and the loader, not the delivery, resamples them.

The timestamps are the join key: use the per-stream nanosecond timestamps to line up a 200 Hz IMU sample, a 30 fps depth frame and a 60 fps video frame, instead of assuming the nth sample of each stream belongs together.

Firsthand collects custom egocentric datasets on demand, and frame rate is one specification to settle at the start of a request. The table below gives the standard rig; rates for other cameras, resolutions or rigs are Not stated, so ask for them directly. The site also states that its face and plate blurring takes video from SD through 4K at any frame rate.

What frame rate does Firsthand capture egocentric video at?

The site states 60 fps for the head camera (3840 × 2160, global shutter) and for both wrist cameras (1920 × 1080), 30 fps for depth (848 × 480), and 200 Hz for the 6-DoF IMUs.

Is a higher frame rate always better for robot training video?

Not by itself. A higher rate samples fast motion more finely but increases storage and processing, and does nothing for a frame skewed by a rolling shutter or misaligned with another camera.

Why do IMU and video data have different numbers of samples?

Each stream runs at its native rate. The site states the IMU at 200 Hz and the head camera at 60 Hz, so there are more IMU samples than video frames. The loader, not the delivery, resamples them using per-stream nanosecond timestamps.

Can a custom dataset use a different frame rate?

The site states the rates of its standard rig only. Any other frame rate is Not stated, so put the rate you need in your custom dataset request.

Does face and plate blurring work at any frame rate?

The site states that face and plate blurring takes video from SD through 4K at any frame rate, with blur strength scaling with face or plate size.