Glossary
WebDataset
A tar-based format for streaming large datasets efficiently during training.
Definition
WebDataset stores samples as sharded tar archives so large datasets stream efficiently from disk or object storage during training, without a random-access database. It suits the multi-gigabyte scale of egocentric video with many streams per episode.
01
Why it matters
At egocentric-video scale, I/O becomes the training bottleneck. Sharded streaming keeps GPUs fed and makes petabyte-scale corpora practical on commodity storage.
02
How Firsthand handles it
Firsthand offers WebDataset exports for teams whose pipelines stream sharded tars, preserving per-stream timestamps and metadata.
Related terms