Glossary
What Are Time-Contrastive Networks (TCN)?
Last reviewed: November 2026
Short answer
Time-Contrastive Networks (TCN) are the self-supervised method in arXiv 1704.06888. A TCN learns a viewpoint-invariant image embedding from unlabeled multi-view video: simultaneous frames from different cameras are pulled together while temporal neighbors are pushed apart. The authors use it for robot reward learning and pose imitation from human video. Third-party research; Firsthand is not affiliated.
01
What are Time-Contrastive Networks, and what problem do they address?
TCN is named in Time-Contrastive Networks: Self-Supervised Learning from Video (arXiv 1704.06888) by Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal and Sergey Levine of Google Brain and the University of Southern California. The authors argue that robots in the real world need to learn the relevant attributes of an object interaction purely from observation, because labelling every aspect of a task such as pouring is impractical. Their answer is to learn from unlabeled videos recorded from multiple viewpoints, so that the representation keeps what changes over time and ignores nuisances such as occlusion, motion blur, lighting and background.
02
Where does Firsthand fit next to TCN?
TCN is third-party research. Firsthand is not affiliated with the authors, and no named model, TCN included, was trained on Firsthand data. Firsthand does not claim to use the Inception network, PILQR, a KUKA arm, a Fetch robot or any tool the paper names. The paper trains on multi-view video; the site lists a multiview rig among its capture setups but does not state that Firsthand supplies time-synchronized multi-view video for TCN training. The site states custom egocentric collection, fast and on demand: first-person human video with calibrated stereo depth, 3D hand pose at 21 joints per hand and 6-DoF IMUs at 200 Hz, with one task instruction per episode. Embeddings, reward functions, robot trajectories and trained policies are not Firsthand deliverables.
03
How does a TCN learn from multi-view video?
Per the paper, the method uses multi-view metric learning with a triplet loss. A frame and a simultaneous frame from another viewpoint (the anchor and the positive) must sit closer together in the embedding than a visually similar frame from a different time in the same sequence (the negative), by a margin. The paper describes two constraints at once: along the view axis the model explains what is common between different-looking images, and along the time axis it explains what differs between similar-looking ones.
The network starts from ImageNet-pretrained Inception layers up to “Mixed 5d”, then adds two convolutional layers, a spatial softmax layer and a fully connected layer, ending in a 32-unit embedding. The authors also describe a single-view variant, with the positive picked within 0.2 s of the anchor, for cases where no multi-view video exists.
04
How is a TCN used for imitation from human video?
The paper uses the embedding two ways. For object interactions, the reward is based on the distance between the embedding of a human demonstration video and the embeddings of the robot’s own camera images, a squared Euclidean term plus a Huber-style term, with weights chosen empirically. The authors optimise it with the PILQR reinforcement learning algorithm and include the TCN features in the system state. For pose imitation, they add a joint decoder and train with self-regression, so the robot imitates humans without human pose labels or explicit joint-level correspondence.
05
What video did the TCN authors train on?
The paper reports smartphone capture by two operators, one holding a fixed view while performing the task and one moving freely, synchronized with an off-the-shelf app. The first table lists the training data the authors state. The paper says its multi-view capture is cheap compared with human labelling. The abstract says open-source code and a dataset are on the project page; the licence is Not stated in the sources read.
06
What did the authors report, and what are the limits of this page?
The second table repeats the authors’ pouring errors from their first results table; Firsthand has not verified them. The authors also report that the multi-view TCN beat single-view models with 15x less training time on the same data, and that the pouring policy converged after 10 iterations on one demonstration (20 on another). They state limits: training needs multi-viewpoint video, which is less available than standard video, and they train a separate TCN per task. They suggest stereo cameras as one way to record multi-viewpoint video.
07
How do TCN and related methods on this site differ?
The third table sets TCN beside four site entries. TCN dates from 2017, so the later methods are described only as those entries state them.
08
What would a TCN-style approach need from custom collection?
Per the paper, it needs multi-viewpoint videos of one task, including some clips of the robot’s own arm, one human demonstration video, and a simulator or robot with a reinforcement learning stack, all the buyer’s own. The paper uses smartphone capture and does not describe egocentric or head-worn capture. Firsthand covers the human video side only, as custom collection at scale through a network of thousands of capture providers, for a consistent, recurring data flow to labs, confirmed on a Spec Pilot.
09
TCN at a glance, as stated in the paper and project page
| Item | What the sources state |
|---|---|
| Paper | Time-Contrastive Networks: Self-Supervised Learning from Video (arXiv 1704.06888) |
| Method name | Time-Contrastive Networks (TCN) |
| Institutions | Google Brain and the University of Southern California |
| Input | Unlabeled videos recorded from multiple viewpoints |
| Loss | Multi-view metric learning with a triplet loss; single-view variant also studied |
| Embedding | 32-dimensional, from an Inception-based network initialised with ImageNet weights |
| Reward | Distance between embeddings of a human demonstration and the robot camera images |
| Policy learning | PILQR on a simulated dish rack task and a real pouring task; self-regression for pose imitation |
| Licences | Not stated in the sources read |
| Firsthand relationship | None |
11
Training data stated by the authors (approximate, as written in the paper)
| Setting | Data stated in the paper |
|---|---|
| Pouring attributes | Two smartphones, sequences of about 5 s; 133 training sequences (about 11 minutes), 17 validation, 30 test |
| Real robot pouring | About 20 minutes of humans pouring, 20 of other cup manipulation and 20 of robot arms, about 1 hour in total; one human video for the policy |
| Dish rack (simulation) | VR demonstrations plus randomized motions; Bullet physics engine and a 7-DoF KUKA arm |
| Pose imitation | About 2 hours of random human motion, 3 hours of random robot motion and 40 minutes of human supervision; Fetch robot |
12
Pouring errors reported by the authors, models selected by validation loss (Firsthand has not verified them)
| Method | Alignment error | Classification error |
|---|---|---|
| Random | 28.1% | 54.2% |
| Inception-ImageNet | 29.8% | 51.9% |
| Shuffle & learn | 22.8% | 27.0% |
| Single-view TCN (triplet) | 25.8% | 24.3% |
| Multi-view TCN (triplet) | 18.8% | 21.4% |
13
TCN beside related entries on the site
| Site entry | What the entry states | How TCN differs (sources read) |
|---|---|---|
| R3M | A visual representation pre-trained on Ego4D with time-contrastive learning, video-language alignment and an L1 penalty, used as a frozen module | The TCN paper trains per-task embeddings on its own multi-view captures and does not mention Ego4D or language |
| HOLD | Learns a goal-image reward from unlabeled human videos via time regression or time-contrastive embeddings | TCN uses simultaneous views as positives and compares a human demonstration video with robot camera images |
| XIRL | Learns an embedding with temporal cycle-consistency and uses distance to a goal state as the reward | TCN uses a triplet loss with cross-view positives and temporal-neighbor negatives |
| GraphIRL | Builds an object graph per frame and matches task progress with temporal cycle-consistency | TCN embeds whole images with an Inception-based network, with no object graph |
14
Frequently asked questions
- What are Time-Contrastive Networks?
- Per arXiv 1704.06888, TCN is a self-supervised method that learns a viewpoint-invariant embedding from unlabeled multi-view video and uses it for robot imitation. It is third-party research.
- Why use multiple viewpoints?
- Simultaneous frames from different cameras serve as positives, which teaches invariance to viewpoint, scale, occlusion, motion blur, lighting and background. Temporal neighbors serve as negatives, so the model must capture what changes over time.
- Can TCN learn from single-view video?
- Yes. The paper studies a single-view variant, and says it improves on ImageNet-trained features but that the multi-view TCN achieves substantially better results.
- How does TCN imitate a human from video?
- The robot optimises a reward based on the distance between embeddings of one human demonstration and its own camera images, using PILQR. For poses, a joint decoder trained with self-regression replaces human pose labels.
- What data did TCN use, and is it licensed?
- Smartphone multi-view video of pouring, robot arm clips and random motion, as the first table lists. The abstract points to a dataset on the project page; the licence is Not stated in the sources read.
- Was TCN trained on Firsthand data?
- No. Firsthand is not affiliated with the authors, and no named model was trained on Firsthand data. Embeddings and reward functions are not Firsthand deliverables.
- How can I request a fast, on-demand custom egocentric dataset for this research?
- Request a custom dataset with your tasks and signals. Collection is fast and on demand, with a first validated batch in a median of 48 hours where coverage is deep.
Related terms
Sources
- Time-Contrastive Networks: Self-Supervised Learning from Video (arXiv 1704.06888)
- TCN project page (official, linked from the paper)
- FaceNet: A Unified Embedding for Face Recognition and Clustering (arXiv 1503.03832), the triplet loss the paper cites
- Shuffle and Learn: Unsupervised Learning using Temporal Order Verification (arXiv 1603.08561), the paper’s baseline
- XIRL: Cross-embodiment Inverse Reinforcement Learning (arXiv 2106.03911)
- Learning Reward Functions for Robotic Manipulation by Observing Humans (arXiv 2211.09019)
- R3M: A Universal Visual Representation for Robot Manipulation (arXiv 2203.12601)