Egocentric Farm and Field Task Video Data

Egocentric farm and field data is first-person video of a person doing growing, irrigation, livestock or equipment work, recorded from their own point of view with synchronized sensor streams alongside. The camera sits on the head, close to what a robot’s onboard camera would see, so the frames show what the hands are doing. The same idea sits behind any agriculture robot training data or farm task video dataset request, applied here as a custom collection scoped to your tasks.

On this site the data is human capture: a paid, consented contributor wears the rig and performs a task from a written skill spec. It is not robot or teleoperation data. The produce packing, plant care and landscaping pages each have their own scope.

The task catalog lists the Farm & Field Operations environment under the Agriculture & Grounds sector. It has 21 rows, and its example line reads "Farm fieldwork, greenhouse, orchard, vineyard -- harvesting, planting, pruning, irrigation, equipment operation". The first table below lists 16 of them: harvesting, crate hauling, planting, irrigation, livestock care, egg collection and equipment operation. Each is Active collection or Ready to deploy.

The remaining five rows are crop sorting and grading, greenhouse and nursery tending, landscaping and groundskeeping, hedge trimming and pruning. They are listed in the second table for orientation. Weeding and milking are not named in any row, so if your program needs them they are written into the skill spec rather than assumed.

The site states no equipment models, crop or livestock types, farm types or sizes, seasons or regions, or cycle times.

Each attempt is delivered as one episode. On this site the standard episode carries action segments, 3D hand pose for both hands with a per-joint visibility flag, contact events that mark when a hand meets an item, an objects list, and per-frame instance masks stored at the depth resolution of 848×480. The capture setup table below lists the stated streams and sync figures.

The catalog lists one capture mode for every farm and field row: a head-mounted phone with a multiview rig. Force and tactile readings are not among the listed streams.

Action segments divide an episode into start- and end-timestamped spans, each carrying start_ns, end_ns, verb and noun. For farm work, a segment might be one fruit picked, one seedling set or one fitting connected, as written in the skill spec. The site lists no label for the outcome of a step, such as whether produce was ripe or a line stopped leaking.

Validation runs against signed accept criteria written into the spec before capture, and a reject log ships with each batch. A human reviewer checks every episode. Only footage that clears the criteria is billed as a validated hour.

Delivery is in the format your stack already reads: RLDS, LeRobot, WebDataset, HDF5, zarr or Rerun .rrd, or raw streams with metadata, with readable converter source alongside. The license is perpetual, irrevocable and worldwide, buyer-owned and non-exclusive by default, with exclusivity available as a separate term. The executed agreement controls.

The catalog describes the work in repetitive, physical steps: picking, digging, connecting lines, carrying feed, walking the pens or fields. Plants occlude the fruit and the hands, and the hands occlude the work. That is why hand pose carries a per-joint visibility flag, why contact events mark when the hand meets the item, and why masks are recorded per frame. The challenge table below maps each difficulty to the stream or label this site states.

Light, weather, terrain and animal behavior are separate problems. The site lists an IMU and body pose as streams but states nothing specific to outdoor lighting, weather or uneven ground, and nothing about animal handling, so the table says so rather than implying a fix.

A robot foundation model paper on arXiv puts human videos at the base of its data pyramid and says they have no explicit action labels, so it learns latent actions. That is robot-model research, not Firsthand data, and Firsthand has no relationship with its authors.

The catalog describes the head-mounted phone as recording the first-person view of real work. The site also states that Firsthand does not capture at a customer’s own facility. Which farm or orchard a program uses is agreed in the skill spec.

Every contributor signs a written release, and each delivered episode links to it through consent artefact IDs. At ingest, automated detection finds faces, plates and screens and blurs them irreversibly, and speech is scrubbed of names, addresses and card numbers. A human reviewer confirms that no un-consented person is identifiable in any frame, and the anonymization record ships with the data.

Safety certifications, protective equipment, machinery safety, food safety and animal welfare are not stated for this environment, and this page makes no such claim. Who may be in frame and what the footage may show are agreed in the spec.

From signed spec to first validated batch, the published timeline is a median of 48 hours where coverage is deep and three to four weeks where it is thin and contributors need to be recruited first. The catalog defines Active collection as recording already running, so a spec can join it, and Ready to deploy as contributors and site access already set up. Which range applies is confirmed when the spec is quoted rather than assumed.

Describe the field or livestock tasks and their order, the tools and containers involved, what counts as a finished step, and the failure or variation cases your pipeline is missing. That becomes a written skill spec, quoted for reach, timeline and price before any capture begins. Our skill spec guide walks through how to write one.

Most programs begin with a Spec Pilot, a small, fast first batch that proves the spec and the accept criteria on real footage before volume scales. Collections are billed per validated hour. Counts of sites, episodes and hours are set in the skill spec.