Glossary
WebDataset
A tar-based format for streaming large datasets efficiently during training.
Short answer
WebDataset stores samples as sharded tar archives so large datasets stream efficiently from disk or object storage during training, without a random-access database. It suits the multi-gigabyte scale of egocentric video with many streams per episode.
01
Why it matters
At egocentric-video scale, I/O becomes the training bottleneck. Sharded streaming keeps GPUs fed and makes petabyte-scale corpora practical on commodity storage.
02
How Firsthand handles it
Firsthand offers WebDataset exports for teams whose pipelines stream sharded tars, preserving per-stream timestamps and metadata.
WebDataset delivery formatOrder a custom WebDataset video datasetWebDataset format referenceRequest a custom dataset
Related terms