Get a quote
Use cases

Put it to work.

Six things labs build from first-person footage of real work.

01Pretraining

Pretrain on the real thing.

Long takes across trades, every clip licensed for training.

Built from

  • Camera IMU on the video clock
  • Language descriptions
  • Hand keypoints
  • LeRobot v3, RLDS, HDF5 or MCAP

Tasks that fit

FootageFig. 1.1

FV-C00046·109.5–117.5 s·Retailcamera file 1920×1080 · 59.94 fps· stabilised picture
Reviewer's reading: Ties the handles of a packed carry bag at the till

Cropped.

DataFig. 1.2

The selected clip as a LeRobot-style episodeFV-C00046-0109500

observation.images.head
camera file 1920×1080 · 59.94 fps · stabilised picture
observation.imu
6 axes · 1,579 samples · 197 Hz
task
Ties the handles of a packed carry bag at the tillreviewer's reading, not a label
frame_index
000 of 480
action
not produced for this clip

An IMU row appears only for a clip with its own IMU file.

  • RetailFV-C00046 · 109.5–117.5 s

  • HVAC servicesFV-B00325 · 290.0–298.0 s

  • CarpentryFV-C01442 · 1500.0–1508.0 s

  • MetalFV-C00744 · 500.0–508.0 sproxy

  • KitchenFV-C01013 · 300.0–308.0 sproxy

  • RetailFV-B00361 · 273.0–281.0 s

  • HVAC servicesFV-C01715 · 300.0–308.0 s

  • CarpentryGX010449 · 1000.0–1008.0 s

  • MetalFV-B01919 · 560.0–568.0 s

  • KitchenFV-C01132 · 300.0–308.0 sproxy

  • RetailFV-B00175 · 200.0–208.0 s

  • HVAC servicesFV-C01728 · 300.0–308.0 s

Fig. 1.1. Pick a tile to open it in the viewer; Fig. 1.2 reads it as a training episode.

Notes (12)
  • FV-C00046 · 109.5–117.5 s: camera file 1920×1080 · 59.94 fps; stabilised picture; cropped.
  • FV-B00325 · 290.0–298.0 s: camera file 1920×1080 · 59.94 fps; stabilised picture
  • FV-C01442 · 1500.0–1508.0 s: camera file 1920×1080 · 50 fps
  • FV-C00744 · 500.0–508.0 s: proxy 768×432 · 59.94 fps; stabilised picture
  • FV-C01013 · 300.0–308.0 s: proxy 640×480 · 59.94 fps
  • FV-B00361 · 273.0–281.0 s: camera file 2704×1520 · 50 fps; cropped.
  • FV-C01715 · 300.0–308.0 s: camera file 1920×1080 · 29.97 fps; cropped.
  • GX010449 · 1000.0–1008.0 s: camera file 1920×1080 · 50 fps
  • FV-B01919 · 560.0–568.0 s: camera file 1920×1080 · 29.97 fps
  • FV-C01132 · 300.0–308.0 s: proxy 640×480 · 59.94 fps; blurred: co-workers at the right as the head turns.
  • FV-B00175 · 200.0–208.0 s: camera file 1920×1080 · 29.97 fps; stabilised picture; blurred: another person at the top right as the head turns.
  • FV-C01728 · 300.0–308.0 s: camera file 1920×1080 · 29.97 fps; blurred: co-workers at the top left. Cropped.

Built from

  • Camera IMU on the video clock
  • Language descriptions
  • Hand keypoints
  • LeRobot v3, RLDS, HDF5 or MCAP

Tasks that fit

02One workplace in depth

Depth of field.

Pick tasks by code. We film them at real stations, with their tools.

Built from

  • Frozen task codes
  • Wrist cameras, multi-view rigs
  • Teleoperation demos
  • LeRobot v3, RLDS, HDF5 or MCAP

Tasks that fit

FootageFig. 2.1

Real footage300.0–308.0 s of 589 s
FV-B00023·Grinds the frame of a steel grille with an angle grinderreviewer's reading, not a labelcamera file 5312×2988 · 59.94 fps · IMU ✓
IMU trace, first 2 s of the clip; solid = accelerometer, dashed = gyroscope; red x, green y, blue z.
Fig. 2.1. Metal, with the clip's own IMU.

StillsFig. 2.2

  • Metal work, first-person still: FV-B00419 at 722.1 s
  • Metal work, first-person still: FV-B00476 at 326.7 s
  • Metal work, first-person still: FV-B00704 at 317.1 s
  • Metal work, first-person still: FV-B01319 at 377.1 s
  • Metal work, first-person still: FV-B01743 at 317.1 s
  • Metal work, first-person still: FV-B01919 at 317.1 s
  • Metal work, first-person still: FV-C00742 at 214.7 s
  • Metal work, first-person still: FV-C00744 at 479.1 s
  • Metal work, first-person still: FV-C00745 at 182.3 s
  • Metal work, first-person still: FV-C01768 at 317.1 s
Fig. 2.2. One trade, many stations, many people: stills from other metal recordings.Metal datasheetMetal on the home wall

Built from

  • Frozen task codes
  • Wrist cameras, multi-view rigs
  • Teleoperation demos
  • LeRobot v3, RLDS, HDF5 or MCAP

Tasks that fit

03Task and step understanding

The order of things.

Real order, real interruptions. Labels run from the goal down to each step.

Built from

  • Language descriptions
  • Frozen task codes
  • Narration
  • Camera IMU on the video clock

Tasks that fit

FootageFig. 3.1

Real footage138.0–146.0 s of 704 s
FV-Z00908·Both hands fill and tie a chutney bag (episode 8)reviewer's reading, not a labelcamera file 2704×1520 · 50 fps · IMU ✓
Goal
Package chutney into plastic bags, stir sauce on the stove, and chop tomatoes on the cutting board.
Sub-goal
Scoop chutney into plastic bags, secure them with rubber bands, and place them into sacks.
Episode
Open plastic bag, pour chutney from pot, fasten rubber band, and store in sack.

Steps, on the clip clockauto-labelled · first-pass

  1. open empty plastic bag136.0–139.0 s
  2. pour chutney into plastic bag with ladlehold plastic bag139.0–142.0 s
  3. twist top of plastic bag142.0–144.0 s
  4. hold plastic bagpick up rubber band from container144.0–146.0 s
  5. Nexttie rubber band around plastic bag146.0–150.0 s
Fig. 3.1. Kitchen, from the session's goal down to the steps in these eight seconds. Our own auto-labels, first-pass and unreviewed.

FootageFig. 3.2Other trades

FV-C01724·300.0–308.0 s·HVAC servicescamera file 1920×1080 · 29.97 fps
Reviewer's reading: A drill and the left hand at an outdoor AC unit

  • HVAC servicesFV-C01724 · 300.0–308.0 s

  • CarpentryFV-C01442 · 900.0–908.0 s

  • MetalFV-B01036 · 300.0–308.0 sproxy

Fig. 3.2. The same structure in HVAC services, carpentry and metal.

Notes (3)
  • FV-C01724 · 300.0–308.0 s: camera file 1920×1080 · 29.97 fps
  • FV-C01442 · 900.0–908.0 s: camera file 1920×1080 · 50 fps
  • FV-B01036 · 300.0–308.0 s: proxy 768×432 · 29.97 fps

Built from

  • Language descriptions
  • Frozen task codes
  • Narration
  • Camera IMU on the video clock

Tasks that fit

04Dexterous hands

Lend your robot a hand.

Both hands in view through fine work, estimated frame by frame, gaps listed.

Built from

  • Hand keypoints
  • Camera trajectory
  • Tactile gloves and mocap
  • Wrist cameras, multi-view rigs

Tasks that fit

A simulated hand on a rebuilt door →

FootageFig. 4.1

Real footage140.0–148.0 s
FV-B01174·Both hands sort screws and nuts on a tiled floorreviewer's reading, not a labelcamera file 1920×1080 · 59.94 fps · IMU ✓
Fig. 4.1. HVAC services: no hand estimate was run on this clip.

Built from

  • Hand keypoints
  • Camera trajectory
  • Tactile gloves and mocap
  • Wrist cameras, multi-view rigs

Tasks that fit

A simulated hand on a rebuilt door →

05World models

What happens next.

The camera measures the head's motion, apart from what the hands change.

Built from

  • Native frame rate
  • Camera IMU on the video clock
  • Camera trajectory
  • Depth

Tasks that fit

Depth, measured: one door →

Footage framesFig. 5.1

Real footage92.0–100.0 s of 442 s

Context

  1. 93.0 s
  2. 94.0 s
  3. 95.0 s
  4. 95.5 s

What actually happened

  1. 96.0 s
  2. 96.5 s
  3. 97.5 s
  4. 98.5 s
FV-C00456·Both hands lift and turn the trim boardreviewer's reading, not a labelframes at 12 fps
Gyroscope y from the clip's own IMU, on the frames' clock. Shaded: the span a prediction is scored against.
Fig. 5.1. Context, then what actually happened. No prediction is drawn.
Notes (1)
  • The split is the clip's midpoint, not a label boundary.

Data and footage framesFig. 5.2

HERO10 Black Widestabilisation off12.50 frames/s steps

The earlier frame is warped by the rotation the gyro measured, through a provisional calibration. Rotation only: the head's movement through space and the rolling shutter stay in the residual, with the hands and near objects.
Picture change per step on FV-C00456: the part the head's rotation predicts and the residual
Time (s)Total changeHead rotationResidualHead share
92.003.90.63.316%
92.087.40.07.31%
92.162.50.12.35%
92.241.80.01.90%
92.321.30.01.40%
92.401.30.21.114%
92.482.41.21.152%
92.562.51.41.254%
92.642.91.71.259%
92.723.52.11.559%
92.801.90.90.950%
92.881.00.10.912%
92.962.61.11.444%
93.041.50.41.127%
93.121.50.41.126%
93.202.01.01.052%
93.281.40.50.936%
93.361.60.61.037%
93.441.80.61.136%
93.521.90.61.234%
93.603.31.32.040%
93.687.22.15.130%
93.7622.65.716.925%
93.8422.95.317.723%
93.9220.26.813.534%
94.0012.93.99.030%
94.088.01.66.420%
94.165.81.04.818%
94.245.31.34.025%
94.325.91.93.933%
94.406.22.24.035%
94.486.93.03.943%
94.566.43.13.348%
94.644.92.32.647%
94.724.31.92.543%
94.803.41.42.041%
94.889.24.74.551%
94.9613.15.27.940%
95.0420.16.913.334%
95.1222.46.915.531%
95.2019.95.114.826%
95.2813.42.910.522%
95.368.31.37.016%
95.445.00.44.69%
95.522.70.02.90%
95.603.42.01.459%
95.6812.04.08.034%
95.7620.35.714.728%
95.8422.07.614.435%
95.9217.87.710.243%
96.0011.95.36.644%
96.088.23.84.447%
96.165.22.23.042%
96.243.71.62.144%
96.323.21.61.748%
96.402.71.41.353%
96.482.91.51.353%
96.562.61.31.350%
96.642.70.81.931%
96.722.60.52.119%
96.803.71.02.727%
96.889.01.08.111%
96.965.12.52.748%
97.045.92.73.246%
97.128.24.24.051%
97.205.52.92.652%
97.281.30.01.30%
97.362.10.02.20%
97.448.43.54.942%
97.5222.39.213.141%
97.6021.98.613.339%
97.6817.95.712.232%
97.7611.72.59.222%
97.848.11.17.013%
97.925.50.06.40%
98.007.10.08.00%
98.086.30.07.80%
98.165.70.06.40%
98.245.60.15.52%
98.325.30.44.98%
98.404.70.05.30%
98.484.90.05.70%
98.564.70.05.40%
98.647.30.07.90%
98.721.80.02.00%
98.801.50.21.315%
98.885.00.94.218%
98.9613.63.510.126%
99.0425.95.520.421%
99.1218.15.812.332%
99.2012.34.47.836%
99.289.63.56.137%
99.369.33.45.937%
99.448.42.16.325%
99.527.10.76.410%
99.607.00.56.47%
99.687.31.16.215%
99.766.71.15.616%
99.845.20.64.711%
Fig. 5.2. Head or scene: how much of each step's change the head's own turn explains; the rest is the hands and the work. Rotation only, through a nominal lens, not a calibration of this camera.
Notes (3)
  • Computed on the published loop, with a nominal equidistant lens, 133.386 deg diagonal field of view.
  • The warp is the rotation the camera's gyro measured between the two frames: no translation and no rolling shutter, so near objects stay in the residual with the hands.
  • On the six fastest-turning steps the warp removes 29% of the change.

Built from

  • Native frame rate
  • Camera IMU on the video clock
  • Camera trajectory
  • Depth

Tasks that fit

Depth, measured: one door →

06Reward and progress models

Zero to done.

Each labelled step has a start and an end. Progress runs between them.

Built from

  • Segment start and end times
  • Language descriptions
  • Mistakes and long takes
  • Camera IMU on the video clock

Tasks that fit

FootageFig. 6.1

Real footage380.0–388.0 s of 712 s
FV-Z00906·Both hands iron a white undershirt (episode 9)reviewer's reading, not a labelcamera file 2704×1520 · 59.94 fps
Progress = time through each labelled step, from our own auto-labels (first-pass). The loop opens and closes mid-step.
Fig. 6.1. Laundry: eight seconds of a 12-minute ironing shift.

DataFig. 6.2

Episode 9 of 215:45–6:49 · 22 first-pass stepsSmooth, iron, and fold the white undershirt onto the stack.The 8 s clip above (380.0–388.0 s) sits in this episode.

Fig. 6.2. The whole shift, with the slow-downs, retries and idle stretches demos leave out: 21 episodes, each a ramp from 0 to 1. Pink marks the clip above; our own auto-labels, first-pass and unreviewed.

Built from

  • Segment start and end times
  • Language descriptions
  • Mistakes and long takes
  • Camera IMU on the video clock

Tasks that fit

Start where your model slips.

Test the free sample, or send task codes for a price.

Get free sample dataGet a quote

Deliveries are camera originals.