Get a quote
Why Ego

Follow the cut.

Eight seconds of a shallot being sliced, from camera to simulated robot hand.

FV-B01774 · 486.0–494.0 sthread clip, renders

One clip, task 10.1.01. Other clips appear only to show one fact each.

Its camera frames appear inside renders, each labelled as a render.

  • Real footage (dark panel)
  • Render or edited footage (solid outline, labelled)

One frame, every layer.

Drag the playhead: footage, keypoints, robot hands and edits stay on frame k.

loop frame k = camera frame 2kk = 120 · 2k = 240490.006 s in the file

Render of our pipeline output. Left: camera frames (re-encoded). Right: simulated robot hands, not camera footage.

Edited footage: the arms are painted out by a model and a simulated robot is drawn in. Not robot footage.

  • Render: Camera frame (re-encoded), FV-B01774 loop frame 120.

    Camera frame (re-encoded)

  • Hand keypoints (estimated) left right

  • Render: Allegro Hand (sim), FV-B01774 loop frame 120.

    Allegro Hand (sim)

  • Render: LEAP Hand (sim), FV-B01774 loop frame 120.

    LEAP Hand (sim)

  • Edited footage: Arms painted out (edited), FV-B01774 loop frame 120.

    Arms painted out (edited)

  • Edited footage: Robot drawn in (edited), FV-B01774 loop frame 120.

    Robot drawn in (edited)

Render and edited footage. Each robot frame k is solved from the hand pose estimated in camera frame 2k. The LEAP hand is shown on a stand-in mount. The painted-out pixels are generated, not footage, and leave a smear at the bottom right.
Is every panel on its own frame?

The render's camera panel at loop frame k, compared with the camera frames k−3 to k+3, matches its own frame best. This checks frame indexing only, not how well the hand estimate is timed.

k−3k−2k−1kk+1k+2k+3
20.922.224.836.226.124.323.2

Highest at k; its own frame scores highest in 210 of 210 frames. Luma PSNR, dB, of the camera panel against the camera frame each offset maps to, measured on the render's compressed web file. Our check on the master file gives 49.10 dB at k 0, 48.75 dB at k 120, 45.16 dB at k 239, against its 35 dB limit.

FV-B01774, loop frames 0–209: 486.0 to 493.0 s of the camera file. The loop shows every second camera frame at 30 fps, so it runs 0.1% fast: 7.000 s shown for 7.007 s filmed.

01Capture

At eye level.

A head camera sees the hands about where a humanoid's head camera would.

FV-Z00907 · at 50.0 sanother clip, frame: Wide lens

FV-B01248 · at 140.0 sanother clip, frame: SuperView lens

First-person frame in a commercial kitchen: the wearer's hands trim green beans at a steel counter; a co-worker's arm at the right edge.

Rings every 10° off the optical axis, drawn by each lens model.

HERO10 Black, Wide: provisional calibration (Kalibr, equidistant), this body and mode.

Every number 3
  • On the FV-Z00907 frame (HERO10 Black, Wide), the focal length our pipeline assumes is 8% higher than the lens model gives at the centre; read as a pinhole, depths there come out 8% too far. Field of view: 133.7°, the calibrated model's diagonal field of view.Source: this site's lens data (computed at build from the provisional calibration of this body and mode)
  • On the FV-B01248 frame (HERO12 Black, SuperView, cropped as the still is published), our pipeline's focal length is 21% to 24% high at the centre, so pinhole depths come out 23% too far there. Field of view: 148.0°, the file's own diagonal field of view.Source: this site's lens data (computed at build from the lens model the camera writes into the file)
  • FV-B01774 was filmed in the Linear lens mode; most of our footage uses Wide or SuperView.Source: the camera file's own settings, read by our pipeline

02One clock

Every head turn, on the clock.

The camera's own IMU, fitted to the video clock, sample by sample.

FV-B01774 · 486.0–494.0 sthread clip, data

Head speed · camera gyro |ω|

46 deg/s

The thread clip: p95 45.5 deg/s, median 13.2, peak 72.5, from 1613 samples.

81 clips our scouts picked for visible hands, not a library-wide distribution. The lower band shows this site's own clips, cut for what they show, so drawn apart.Stabilised clip; the IMU still records the head's real motion.
Head speed per clip: gyro p95 and the share of samples at or above the threshold
ClipWorkplaceStabilisationp95 (deg/s)Share at or above
FV-B01774 @ 486.0 s (the thread)Restaurant and Commercial Kitchenon45.55%
FV-B00860 @ 97.1 sTrade Service — Plumbing, HVAC and Electricalon2.70%
FV-B01827 @ 544.8 sCarpentry and Furniture Workshopoff5.20%
FV-B01690 @ 188.6 sCarpentry and Furniture Workshopoff6.00%
FV-C01812 @ 247.6 sMetal Fabrication and Weldingoff7.60%
FV-C01720 @ 676.5 sTrade Service — Plumbing, HVAC and Electricaloff7.90%
FV-B00838 @ 393.4 sTrade Service — Plumbing, HVAC and Electricalon8.70%
FV-C00012 @ 616.5 sRetail Store — Back of Houseoff8.80%
FV-B01132 @ 3026.3 sCarpentry and Furniture Workshopoff8.80%
FV-B00385 @ 1353.5 sMetal Fabrication and Weldingon8.90%
FV-B01743 @ 326.1 sMetal Fabrication and Weldingoff9.30%
FV-C01534 @ 107.7 sCarpentry and Furniture Workshopoff9.40%
FV-C01768 @ 223.8 sMetal Fabrication and Weldingoff9.80%
FV-B01319 @ 559.6 sMetal Fabrication and Weldingoff10.50%
FV-C00742 @ 372.8 sMetal Fabrication and Weldingon10.50%
FV-B01248 @ 96.3 sCarpentry and Furniture Workshopoff10.60%
FV-C01442 @ 171.8 sCarpentry and Furniture Workshopoff10.80%
FV-B01150 @ 153.5 sRestaurant and Commercial Kitchenoff10.90%
GX010445 @ 1135.0 sCarpentry and Furniture Workshopoff11.40%
FV-C00905 @ 357.9 sRestaurant and Commercial Kitchenoff11.71%
FV-B01919 @ 416.7 sMetal Fabrication and Weldingoff12.40%
FV-C01818 @ 193.8 sMetal Fabrication and Weldingoff12.60%
FV-C00689 @ 823.4 sTrade Service — Plumbing, HVAC and Electricalon13.30%
FV-C00684 @ 575.6 sTrade Service — Plumbing, HVAC and Electricalon14.40%
FV-C01321 @ 285.7 sCarpentry and Furniture Workshopoff14.40%
FV-B00419 @ 539.5 sMetal Fabrication and Weldingon15.20%
FV-B00476 @ 80.1 sMetal Fabrication and Weldingon15.20%
FV-C01418 @ 1591.5 sCarpentry and Furniture Workshopoff15.30%
FV-B01174 @ 62.7 sTrade Service — Plumbing, HVAC and Electricalon18.00%
FV-B01155 @ 859.5 sRestaurant and Commercial Kitchenoff18.00%
GX010005 @ 879.1 sCarpentry and Furniture Workshopoff18.20%
GX010125 @ 173.4 sCarpentry and Furniture Workshopoff18.20%
GX010445 @ 461.8 sCarpentry and Furniture Workshopoff18.20%
FV-C01726 @ 381.4 sTrade Service — Plumbing, HVAC and Electricaloff18.70%
FV-B01150 @ 165.2 sRestaurant and Commercial Kitchenoff19.00%
FV-B01794 @ 1431.7 sCarpentry and Furniture Workshopoff19.50%
FV-B01810 @ 386.5 sCarpentry and Furniture Workshopoff20.40%
FV-C01132 @ 618.1 sRestaurant and Commercial Kitchenoff20.40%
FV-B01174 @ 89.1 sTrade Service — Plumbing, HVAC and Electricalon21.01%
FV-B01822 @ 105.0 sCarpentry and Furniture Workshopoff22.10%
FV-C00744 @ 8.3 sMetal Fabrication and Weldingon22.30%
FV-B00838 @ 263.3 sTrade Service — Plumbing, HVAC and Electricalon26.50%
FV-C01419 @ 199.7 sCarpentry and Furniture Workshopoff26.70%
GX011161 @ 741.2 sCarpentry and Furniture Workshopoff26.80%
FV-C00024 @ 127.9 sRetail Store — Back of Houseoff28.80%
FV-B01835 @ 390.0 sCarpentry and Furniture Workshopoff28.90%
FV-B01149 @ 958.1 sRestaurant and Commercial Kitchenoff29.10%
FV-B01851 @ 493.3 sCarpentry and Furniture Workshopoff29.61%
FV-B01155 @ 820.0 sRestaurant and Commercial Kitchenoff29.62%
FV-C00745 @ 235.0 sMetal Fabrication and Weldingon29.82%
GX010007 @ 699.6 sCarpentry and Furniture Workshopoff29.92%
FV-B01851 @ 492.0 sCarpentry and Furniture Workshopoff30.21%
FV-B00704 @ 335.7 sMetal Fabrication and Weldingoff30.20%
FV-B01174 @ 101.1 sTrade Service — Plumbing, HVAC and Electricalon31.63%
FV-B01857 @ 670.0 sCarpentry and Furniture Workshopoff34.21%
FV-B00704 @ 493.2 sMetal Fabrication and Weldingoff34.42%
FV-B01264 @ 2316.1 sCarpentry and Furniture Workshopoff34.90%
FV-B01810 @ 222.0 sCarpentry and Furniture Workshopoff35.21%
FV-B01155 @ 860.9 sRestaurant and Commercial Kitchenoff36.22%
FV-B00476 @ 20.0 sMetal Fabrication and Weldingon36.71%
FV-C01283 @ 1140.4 sRestaurant and Commercial Kitchenoff37.21%
FV-B01827 @ 327.3 sCarpentry and Furniture Workshopoff37.51%
FV-B01319 @ 649.0 sMetal Fabrication and Weldingoff37.60%
GX010445 @ 663.0 sCarpentry and Furniture Workshopoff38.22%
FV-B01690 @ 589.0 sCarpentry and Furniture Workshopoff39.33%
FV-C01638 @ 892.0 sCarpentry and Furniture Workshopoff40.11%
FV-B00361 @ 155.4 sRetail Store — Back of Houseoff42.33%
FV-B01857 @ 636.0 sCarpentry and Furniture Workshopoff43.75%
FV-B00175 @ 9.0 sRetail Store — Back of Houseon44.34%
FV-B01155 @ 868.9 sRestaurant and Commercial Kitchenoff45.35%
FV-B01827 @ 561.3 sCarpentry and Furniture Workshopoff46.05%
FV-C01738 @ 1034.9 sRestaurant and Commercial Kitchenoff48.36%
FV-B01851 @ 493.8 sCarpentry and Furniture Workshopoff49.05%
FV-C00023 @ 190.4 sRetail Store — Back of Houseoff49.78%
FV-Z00907 @ 20.0 sRestaurant and Commercial Kitchenoff54.27%
FV-C01077 @ 142.7 sTrade Service — Plumbing, HVAC and Electricalon55.17%
FV-B01827 @ 323.4 sCarpentry and Furniture Workshopoff58.515%
FV-B01149 @ 961.0 sRestaurant and Commercial Kitchenoff64.99%
FV-B01150 @ 216.2 sRestaurant and Commercial Kitchenoff72.811%
FV-B00175 @ 118.3 sRetail Store — Back of Houseon75.915%
FV-B00836 @ 744.7 sTrade Service — Plumbing, HVAC and Electricalon78.135%
FV-B01149 @ 1011.0 sRestaurant and Commercial Kitchenoff90.718%
This site's own clips
FV-B00325 @ 270.0 s——24.70%
FV-B00023 @ 122.0 s——37.72%
FV-Z00908 @ 408.0 s——73.113%
FV-Z00906 @ 32.0 s——92.225%
FV-Z00901 @ 246.5 s——138.235%
FV-C00456 @ 16.0 s——216.916%

FV-B01857 · 300.0–310.0 sanother clip, data: frames and samples on one axis

video frame · every 20,000 µs (50 fps)IMU sample · mean spacing 5,002 µs (200 Hz measured)1 ms tick

t = ––––––––––– µs · frame –––––– from ––––––––––– µs · nearest IMU sample ––––– of 1,999 at ––––––––––– µs (frame start ––––––– µs)

FV-B01857, 300 to 310 s on its video clock: 500 video frames and 1,999 IMU samples, each sample at its own timestamp_us. Data only: this clip's footage is not shown on the site. Drag the playhead; the arrow keys step one IMU sample, Page Up and Page Down one video frame.

FV-B01827 · 323.4–331.4 sanother clip, footage: stabilisation off

Real footage323.4–331.4 s of 1800 s
FV-B01827·9.3.03first-passHardware installation2704×2028 · 50 fps · IMU ✓
Stabilisation off: the picture turns with the head.
Every number 8
  • IMU written at a nominal 200 Hz.Source: GoPro's published GPMF format: the camera writes its IMU at a nominal 200 Hz
  • Measured at 196.2 to 201.1 Hz on every clip with IMU on this site.Source: clips.json (each clip's IMU sidecar)
  • FV-B01774, 486.0–494.0 s: head speed median 13.2 deg/s, 95th percentile 45.5, peak 72.5, from 1,613 gyro samples. Stabilisation on.Source: the head‑motion data (computed at build from the camera's GPMF gyro track, extracted by our pipeline)
  • Fastest whole second: 491–492 s, 95th percentile 68.4 deg/s, peak 72.5, while the hands keep working.Source: the same samples, computed at build from them
  • Gyro 95th percentile from 2.7 to 90.7 deg/s across 81 clips our scouts picked for visible hands; not a library‑wide distribution.Source: the head‑motion data (computed at build from each clip's camera file)
  • FV-B01827 was filmed with stabilisation off.Source: its pose sidecar (the camera's own settings box, read from the file)
  • Down for the camera body: 10.8° from straight down, against 73.9° guessed from the hands.Source: our audit of the gravity worked example on this clip (axis convention not checked)
  • In the delivered _imu.csv the sensor clock is fitted to the video timeline over the whole file, so timestamps are smooth, not quantised to the container's 1 ms timescale.Source: our IMU extraction, as it writes every delivered _imu.csv

03The wearer's hands

Two hands. No filler.

Only the wearer's hands. A frame without an estimate is listed, never filled.

FV-B01774 · 486.0–494.0 sthread clip, render

RenderFV-B01774 · 486.0–494.0 s

Render of our pipeline output: estimated hand keypoints drawn over camera frames (re-encoded). An estimate, not a measurement.

Render. The hand estimate for each frame, drawn as 21 keypoints per hand over the camera frames: both hands in all 240 frames.

Not camera footage.

Notes (1)
Hand pose is a model's estimate from video. We have not measured its error against a reference, so we quote no accuracy figure.

FV-Z00907 · 20.0–28.0 sanother clip, render: whose hands, and a hand that leaves the frame

24.68 s · frame 1479
First-person frame in a commercial kitchen: the wearer's two hands trim green beans at a steel counter; at the right edge a co-worker's two hands cut carrots on a green board.
How each hand was judged
  1. Wearer's left hand. The ray from the knuckles through the wrist meets the bottom edge at 0.33 of the width: inside 0.20 to 0.95, where the wearer's forearms enter. Kept.
  2. Wearer's right hand. The ray from the knuckles through the wrist meets the bottom edge at 0.59 of the width: inside 0.20 to 0.95, where the wearer's forearms enter. Kept.
  3. A co-worker's hand. The ray from the knuckles through the wrist leaves through the right edge at 0.45 of the height: a wearer's forearm leaves the right edge only from 0.80 of the height down. Rejected. Its box also sits in a zone marked by hand for this clip's co-worker.
  4. A co-worker's hand. The ray from the knuckles through the wrist leaves through the right edge at 0.18 of the height: a wearer's forearm leaves the right edge only from 0.80 of the height down. Rejected. Its box also sits in a zone marked by hand for this clip's co-worker.
Of the 4 hands found on this frame, the ray test keeps the wearer's 2; the other 2 fail it. A fixed geometric rule, not scored against labelled frames.

RenderFV-Z00907 · 20.0–28.0 s

Render of our pipeline output: estimated hand keypoints drawn over camera frames (re-encoded). An estimate, not a measurement.

Render. The wearer's right-hand estimate drawn as 21 keypoints over the camera frames. In 54 frames there is no estimate, and the frame says so instead of drawing one.

Not camera footage.

Notes (2)
A frame with no hand estimate is listed, never filled.Hand pose is a model's estimate from video. We have not measured its error against a reference, so we quote no accuracy figure.

FV-Z00907, right hand: an estimate on 186 of 240 frames. The others (frames 184–237) are listed with the reason, never filled.

  • Estimate · 183 a 3D estimate of the right hand on this frame.
  • Estimate, low confidence · 3 an estimate our pipeline marks low confidence.
  • No estimate: fit disagreed · 6 the 3D fit disagreed with the 2D landmarks by more than the gate allows, so it was dropped.
  • No estimate: isolated · 1 a lone detection with no right-hand detection within two frames, so it was dropped.
  • No estimate: tracker box · 1 a box carried over by the visual tracker that the fitted hand covered poorly, so it was dropped.
  • No estimate: not detected · 46 no right-hand detection on this frame to fit.
0 s
1 s
2 s
3 s
4 s
5 s
6 s
7 s

Point at a frame, or tab into the strip and use the arrow keys, to read its state.

Here the right hand comes to rest at the bottom‑right edge, mostly out of view. A low‑confidence estimate is kept and marked.
Every number 5
  • FV-B01774: left hand estimated in 240 of 240 frames, right hand in 240 of 240.Source: the layer scrubber's keypoint data (computed at build from our HaMeR estimates)
  • Our MediaPipe route found 34.5% of visible wearer hands on 120 random labelled frames.Source: our own check: 120 random labelled frames, wearer hands counted by hand
  • FV-Z00907: no right‑hand estimate in 54 of 240 frames.Source: our per‑frame hand tracks for FV-Z00907 (frames k 184–237)
  • FV-Z00907, 20.0–28.0 s, right hand, frame by frame: an estimate in 186 of 240 frames (3 of them low confidence), none in 54: 6 fit disagreed, 1 isolated, 1 tracker box, 46 not detected.Source: the frame strip's data (computed at build from our per‑frame hand tracks and their drop reasons)
  • FV-Z00907 frame k 140 (24.7 s into the file): MediaPipe found 4 hands and the rule kept 2 as the wearer's. A hand is kept if the ray from its knuckles through its wrist meets the bottom edge between 0.20 and 0.95 of the width, or the right edge at 0.80 of the height or lower. Not scored against labelled frames.Source: the wearer‑reveal data (computed at build from our MediaPipe candidates for that frame; the ray is recomputed and checked against our pipeline's record)

04Labels

Coded, then spelled out.

One code per stretch, from 354 tasks; on request, words from goal to step.

FV-Z00906 · 160.0–168.0 sanother clip, footage: labels from goal to step

1 session goal, 4 sub-goals, 21 episodes, 237 steps in 711.7 sauto-labelled · first-pass

Real footage
  1. Session goal Iron, fold, and stack shirts, jackets, shorts, and trousers on the ironing table.
    1. Sub-goal 1 Iron, button, and smooth shirts on the ironing board. 0.0–216.0 s · 5 episodesholds the 8 s loop
      1. Episode 1 Iron and button the shirt on the ironing board. 0.0–30.0 s · 9 steps
        1. reposition shirt on ironing board0.0–1.8 s
        2. pick up iron from standhold shirt1.8–3.2 s
        3. iron shirt with ironhold shirt3.2–12.0 s
        4. iron shirt with ironhold shirt12.0–16.5 s
        5. place iron on standreposition shirt16.5–18.0 s
        6. reposition shirt on ironing board18.0–20.0 s
        7. button shirt20.0–23.5 s
        8. button shirt23.5–27.0 s
        9. button shirt27.0–30.0 s
      2. Episode 2 Smooth, iron, and fold the navy shirt. 30.0–88.0 s · 16 steps
        1. unfold navy shirt30.0–33.0 s
        2. align navy shirt on ironing table33.0–37.0 s
        3. smooth navy shirt on ironing table37.0–42.0 s
        4. pick up iron42.0–44.0 s
        5. hold navy shirtiron navy shirt with iron44.0–51.0 s
        6. place iron on stand51.0–52.0 s
        7. fold navy shirt on ironing table52.0–56.0 s
        8. smooth navy shirt on ironing table56.0–60.0 s
        9. hold shirt on ironing boardiron shirt with iron60.0–66.0 s
        10. place iron on stand66.0–67.5 s
        11. fold sleeve of shirt67.5–71.0 s
        12. pick up shirtplace shirt on ironing board71.0–73.0 s
        13. flatten shirt on ironing board73.0–78.0 s
        14. fold side of shirt78.0–82.0 s
        15. fold shirt82.0–85.0 s
        16. fold shirt85.0–88.0 s
      3. Episode 3 Check phone, retrieve garment, and iron it on the board. 88.0–123.0 s · 15 steps
        1. observe workspace88.0–90.0 s
        2. use phone90.0–95.0 s
        3. walk to chair95.0–98.0 s
        4. pick up garment from chair98.0–100.5 s
        5. place garment on ironing board100.5–103.0 s
        6. unfold garment on ironing board103.0–106.5 s
        7. reposition garment on ironing board106.5–110.0 s
        8. flatten garment on ironing board110.0–112.0 s
        9. pick up iron from stand112.0–113.2 s
        10. hold garment on ironing boardiron garment with iron113.2–116.0 s
        11. place iron on stand116.0–117.0 s
        12. pick up phone from ironing board117.0–118.5 s
        13. use phone118.5–120.0 s
        14. hold phone120.0–122.0 s
        15. place phone on table122.0–123.0 s
      4. Episode 4 Smooth, flip, and iron the brown shirt. 123.0–150.0 s · 10 steps
        1. unfold brown shirt on table123.0–126.0 s
        2. smooth brown shirt on table126.0–131.0 s
        3. pick up ironstabilize brown shirt131.0–132.0 s
        4. iron brown shirt with ironhold brown shirt132.0–135.0 s
        5. place iron on stand135.0–136.0 s
        6. flip brown shirt on table136.0–139.0 s
        7. smooth brown shirt on table139.0–143.0 s
        8. iron brown shirt with ironhold brown shirt143.0–147.0 s
        9. place iron on stand147.0–148.0 s
        10. flip brown shirt on table148.0–150.0 s
      5. Episode 5 Smooth, fasten buttons, and iron the polo shirt. 150.0–216.0 s · 20 stepsholds the 8 s loop
        1. spread polo shirt on ironing board150.0–152.5 s
        2. smooth polo shirt on ironing board152.5–156.5 s
        5. pick up iron from stand168.5–170.0 s
        6. iron polo shirt with ironhold polo shirt170.0–178.5 s
        7. reposition polo shirt on ironing board178.5–180.0 s
        8. smooth shirt on table180.0–182.8 s
        9. hold shirtiron shirt with iron182.8–189.0 s
        10. place iron on stand189.0–190.3 s
        11. reposition shirt on table190.3–194.5 s
        12. turn shirt over on table194.5–197.2 s
        13. smooth shirt on table197.2–201.6 s
        14. pick up iron from standhold shirt201.6–203.0 s
        15. hold shirtiron shirt with iron203.0–207.5 s
        16. place iron on stand207.5–208.8 s
        17. smooth shirt on table208.8–210.0 s
        18. smooth shirt on ironing board210.0–211.0 s
        19. iron shirt with ironhold shirt211.0–215.0 s
        20. place iron on stand215.0–216.0 s
    2. Sub-goal 2 Iron, fold, and stack shirts, a jacket, and pants. 216.0–463.0 s · 5 episodes
      1. Episode 6 Fold the shirt and place it on the stack. 216.0–242.0 s · 7 steps
        1. fold side of shirt216.0–220.0 s
        2. fold other side of shirt220.0–224.0 s
        3. smooth shirt224.0–227.0 s
        4. align shirt on ironing board227.0–231.0 s
        5. fold bottom of shirt upward231.0–236.0 s
        6. turn shirt overadjust collar236.0–240.0 s
        7. place folded shirt on stack240.0–242.0 s
      2. Episode 7 Retrieve, smooth, and fold the t‑shirt onto the stack. 242.0–273.0 s · 10 steps
        1. turn to chairpick up t‑shirt from chair242.0–245.0 s
        2. unfold t‑shirt245.0–248.0 s
        3. align shoulders of t‑shirt248.0–251.0 s
        4. spread t‑shirt on table251.0–254.0 s
        5. smooth t‑shirt on table254.0–258.0 s
        6. fold side of t‑shirt258.0–262.0 s
        7. fold side of t‑shirt262.0–266.0 s
        8. fold t‑shirt in half266.0–268.0 s
        9. smooth folded t‑shirt268.0–270.0 s
        10. place folded shirt on stack of clothes270.0–273.0 s
      3. Episode 8 Zip, smooth, and fold the black jacket onto the stack. 273.0–345.0 s · 22 steps
        1. walk to chair273.0–275.0 s
        2. sort through clothes on chair275.0–278.0 s
        3. sort through clothes on chair278.0–283.0 s
        4. pick up black shirt from chair283.0–286.0 s
        5. hold black shirtwalk to ironing board286.0–288.0 s
        6. spread black shirt on ironing board288.0–293.0 s
        7. turn sleeve of black shirt right side out293.0–297.0 s
        8. reposition black shirt on ironing board297.0–300.0 s
        9. place black jacket on table300.0–302.0 s
        10. zip up black jacket302.0–308.0 s
        11. smooth black jacket on table308.0–311.0 s
        12. lift black jacket311.0–313.0 s
        13. place black jacket on table313.0–315.0 s
        14. smooth black jacket on table315.0–318.0 s
        15. smooth black jacket on tablehold black jacket318.0–321.0 s
        16. smooth black jacket on table321.0–326.0 s
        17. fold sleeve of black jackethold black jacket326.0–330.0 s
        18. fold black garment330.0–333.0 s
        19. fold black garment333.0–336.0 s
        20. fold black garment336.0–338.0 s
        21. fold black garment338.0–342.0 s
        22. pick up folded black garmentplace folded black garment on stack342.0–345.0 s
      4. Episode 9 Smooth, iron, and fold the white undershirt onto the stack. 345.0–409.0 s · 22 steps
        1. turn toward chairpick up white undershirt from pile345.0–348.0 s
        2. unfold white undershirt348.0–350.0 s
        3. place white undershirt on table350.0–353.0 s
        4. inspect white undershirt353.0–357.0 s
        5. spread white undershirt on table357.0–360.0 s
        6. unfold undershirt360.0–362.5 s
        7. align undershirt on ironing board362.5–364.5 s
        8. smooth undershirt on ironing board364.5–368.5 s
        9. align straps of undershirt368.5–371.0 s
        10. smooth undershirt on ironing board371.0–373.0 s
        11. pick up ironhold undershirt373.0–374.5 s
        12. iron undershirt with ironhold undershirt374.5–381.5 s
        13. place iron on standhold undershirt381.5–383.0 s
        14. fold undershirt383.0–385.0 s
        15. lift and flip undershirt385.0–387.5 s
        16. smooth undershirt on ironing board387.5–389.0 s
        17. pick up ironhold undershirt389.0–390.0 s
        18. iron undershirt with ironhold undershirt390.0–399.0 s
        19. place iron on standhold undershirt399.0–400.5 s
        20. fold undershirt400.5–404.0 s
        21. fold undershirt404.0–407.0 s
        22. place folded undershirt on clothes stack407.0–409.0 s
      5. Episode 10 Smooth, iron, and fold the pants onto the pile. 409.0–463.0 s · 19 steps
        1. walk to chairpick up pants from chair409.0–412.0 s
        2. unfold pants412.0–414.5 s
        3. spread pants on ironing table414.5–417.5 s
        4. smooth pants on ironing table417.5–420.0 s
        5. smooth pants on ironing board420.0–421.0 s
        6. hold pantsiron pants with iron421.0–426.0 s
        7. hold pantsiron pants with iron426.0–431.0 s
        8. place iron on iron resthold pants431.0–432.0 s
        9. fold pants on ironing board432.0–436.0 s
        10. smooth pants on ironing board436.0–440.0 s
        11. hold pantspick up iron440.0–441.0 s
        12. hold pantsiron pants with iron441.0–446.0 s
        13. hold pantsiron pants with iron446.0–450.0 s
        14. place iron on stand450.0–451.0 s
        15. fold pants451.0–455.0 s
        16. smooth pants on table455.0–457.0 s
        17. fold pants457.0–460.0 s
        18. place folded pants onto pile on table460.0–461.0 s
        19. walk to chair461.0–463.0 s
    3. Sub-goal 3 Iron, flip, and fold plaid shorts on the table. 463.0–551.0 s · 4 episodes
      1. Episode 11 Pick up, unfold, align, and iron plaid shorts on the table. 463.0–479.0 s · 7 steps
        1. pick up plaid shorts from chair463.0–466.0 s
        2. unfold plaid shorts466.0–468.0 s
        3. align plaid shorts on table468.0–471.0 s
        4. flip plaid shorts on table471.0–473.0 s
        5. smooth plaid shorts on tablepick up iron from stand473.0–474.0 s
        6. hold plaid shorts on tableiron plaid shorts with iron474.0–478.0 s
        7. place iron on stand478.0–479.0 s
      2. Episode 12 Spread, flip, smooth, and iron shorts on the table. 479.0–503.0 s · 9 steps
        1. fold plaid shorts479.0–480.0 s
        2. spread shorts on table480.0–481.0 s
        3. smooth shorts on table481.0–486.0 s
        4. reposition shorts on table486.0–488.0 s
        5. flip shorts on table488.0–491.0 s
        6. smooth shorts on table491.0–494.0 s
        7. pick up iron from standsmooth shorts494.0–495.0 s
        8. iron shorts with ironhold shorts495.0–502.0 s
        9. place iron on stand502.0–503.0 s
      3. Episode 13 Flip, smooth, reposition, and iron shorts on the table. 503.0–530.5 s · 10 steps
        1. flip shorts on table503.0–508.0 s
        2. smooth shorts on table508.0–510.0 s
        3. smooth shorts on table510.0–512.0 s
        4. hold shortspick up iron from table512.0–513.5 s
        5. hold shortsiron shorts with iron513.5–519.5 s
        6. place iron on table519.5–521.0 s
        7. reposition shorts on table521.0–524.0 s
        8. smooth shortspick up iron from table524.0–525.5 s
        9. hold shortsiron shorts with iron525.5–529.0 s
        10. place iron on table529.0–530.5 s
      4. Episode 14 Iron, fold, and place plaid shorts onto the stack of clothes. 530.5–551.0 s · 8 steps
        1. fold shorts on table530.5–533.0 s
        2. smooth shortspick up iron from table533.0–534.5 s
        3. hold shortsiron shorts with iron534.5–538.0 s
        4. place iron on tablehold shorts538.0–539.0 s
        5. pick up shorts from table539.0–540.0 s
        6. unfold plaid shorts540.0–544.0 s
        7. fold plaid shorts on table544.0–549.5 s
        8. place plaid shorts on stack of clothes549.5–551.0 s
    4. Sub-goal 4 Iron, smooth, and fold pants and trousers on the ironing board. 551.0–711.7 s · 7 episodes
      1. Episode 15 Pick up, unfold, flatten, and iron pants on the ironing table. 551.0–573.2 s · 10 steps
        1. pick up pants from chair551.0–553.5 s
        2. unfold pants553.5–555.0 s
        3. place pants on ironing table555.0–557.0 s
        4. unfold pants557.0–559.0 s
        5. flatten pants on ironing table559.0–560.5 s
        6. pick up ironhold pants560.5–562.0 s
        7. iron pants with ironhold pants562.0–566.0 s
        8. iron pants with ironhold pants566.0–570.0 s
        9. hold pantsiron pants with iron570.0–572.0 s
        10. place iron on standhold pants572.0–573.2 s
      2. Episode 16 Pick up, operate, and place phone on the table. 573.2–577.0 s · 3 steps
        1. pick up phone from table573.2–574.2 s
        2. operate phone574.2–576.0 s
        3. place phone on table576.0–577.0 s
      3. Episode 17 Reposition and smooth pants on the ironing board and observe workspace. 577.0–604.5 s · 8 steps
        1. reposition pants on ironing board577.0–578.2 s
        2. pick up pants from ironing boardplace pants on ironing board578.2–580.2 s
        3. smooth pants on ironing board580.2–582.0 s
        4. fold pant leghold pants582.0–584.0 s
        5. smooth pants on ironing board584.0–587.5 s
        6. observe workspace587.5–594.0 s
        7. observe workspace594.0–600.0 s
        8. observe workspace600.0–604.5 s
      4. Episode 18 Flatten, reposition, and iron trouser leg and pants on the table. 604.5–643.0 s · 11 steps
        1. flatten trouser leg on ironing table604.5–605.8 s
        2. pick up iron605.8–607.0 s
        3. hold trouser leg on ironing tableiron trouser leg with iron607.0–616.0 s
        4. hold trouser leg on ironing tableiron trouser leg with iron616.0–621.5 s
        5. place iron on standsmooth trouser leg621.5–623.0 s
        6. reposition trouser leg on ironing table623.0–627.0 s
        7. pick up iron from standhold trouser leg627.0–628.5 s
        8. iron trouser leg with ironhold trouser leg628.5–630.0 s
        9. iron pants with ironhold pants630.0–636.0 s
        10. iron pants with ironhold pants636.0–641.0 s
        11. place iron on stand641.0–643.0 s
      5. Episode 19 Fold, align, and smooth pants on the ironing board. 643.0–667.0 s · 6 steps
        1. fold pants on ironing board643.0–648.0 s
        2. align pants on ironing board648.0–651.0 s
        3. fold pants on ironing board651.0–656.0 s
        4. align pants on ironing board656.0–660.0 s
        5. fold pants on ironing board660.0–663.0 s
        6. smooth pants on ironing boardhold pants663.0–667.0 s
      6. Episode 20 Pick up, operate, and place smartphone on the ironing board. 667.0–671.0 s · 3 steps
        1. pick up smartphone667.0–668.0 s
        2. operate smartphone668.0–670.0 s
        3. place smartphone on ironing board670.0–671.0 s
      7. Episode 21 Smooth, fold, and iron trousers on the ironing board. 671.0–711.7 s · 12 steps
        1. smooth pants on ironing board671.0–673.0 s
        2. fold pants on ironing board673.0–679.0 s
        3. align pants on ironing board679.0–684.0 s
        4. smooth pants on ironing board684.0–687.0 s
        5. iron pants with ironhold pants687.0–690.0 s
        6. hold trousersiron trousers with iron690.0–694.0 s
        7. flatten trousersiron trousers with iron694.0–697.0 s
        8. iron trousers with ironhold trousers697.0–700.0 s
        9. hold trousersiron trousers with iron700.0–704.0 s
        10. iron trousers with ironhold trousers704.0–709.0 s
        11. place iron on ironing board709.0–710.2 s
        12. fold trousers710.2–711.7 s
Every number 3
  • 354 tasks in 70 task groups across 16 workplaces.Source: taxonomy.json (the frozen task list), counted at build
  • The thread carries 10.1.01, Vegetable washing and chopping.Source: the audit's clip master (M64)
  • FV-Z00906: 4 sub‑goals, 21 episodes and 237 steps in 711.7 s; steps last 3.0 s on average. First‑pass, unreviewed.Source: the label hierarchy (computed at build from our auto‑labeller's first‑pass export)

05Robot hands

Handed down, frame by frame.

Each robot frame k is solved from the hand estimated in video frame k.

FV-B01774thread clip, renders

RenderFV-B01774 · 486.0–493.0 s

Render of our pipeline output. Left: camera frames (re-encoded). Right: simulated robot hands, not camera footage.

Render. 486.0 to 493.0 s of the same clip, both hands on both robots. The LEAP hand is shown on a stand‑in mount. The arms are not speed‑limited: in places they move several times faster than a real arm can.

Not camera footage.

Notes (4)
Solved with loosened continuity limits (up to 30 degrees per frame for the arms) and choices made for this clip.The arms are not speed-limited: in places they move several times faster than a real arm can.The LEAP hand is shown on a stand-in mount.The robot hands and arms are simulated renders driven by estimated hand poses. No gripper visibly holds anything.

RenderFV-B01774 · 486.0–493.0 s

Render of our pipeline output: simulated robot hands only, not camera footage.

Render. 486.0 to 493.0 s of the same clip: Allegro above, LEAP below, both hands. The LEAP hand is shown on a stand‑in mount. The arms are not speed‑limited: in places they move several times faster than a real arm can.

Not camera footage.

Notes (4)
Phone version: the camera frames are in the wide version.The arms are not speed-limited: in places they move several times faster than a real arm can.The LEAP hand is shown on a stand-in mount.The robot hands and arms are simulated renders driven by estimated hand poses. No gripper visibly holds anything.

Render: a simulated ORCA hand model replaying the hand poses we estimated in this clip. Not robot footage.

Render: the simulated ORCA v2 hand model at loop frames 0, 70, 140 and 209 of FV-B01774, posed from the right hand's estimated poses.
Loop frames 0 to 209, the right hand only. Model frame k comes from video frame k, retargeted to the hand's 16 finger joints; the wrist is fixed. Every one of the 210 frames has its own estimate.Each frame is solved on its own from that frame's estimate; the joint angles are drawn smoothed over 9 frames, centred so frame k stays on video frame k.Hand model: the ORCA v2 right hand from its open description (MIT licence), simplified for the web. ORCA is a trademark of its owners, who are not affiliated with this site and do not endorse it.

One frame clock. The render above, the ORCA replay and the keypoints in the opening figure all show loop frame k.

k 0 / 209 · 486.00 s

Without scripts each figure shows its own still; with them, one bar moves all three.

Every final we rendered14Grouped by the review's verdict, with where each one plays.

Data

FinalClip and timeHandsFrames solved of 240Flaw the review foundWhere it plays
Primary1
B3FV-B01774486.0–494.0 sTwoclose-up with Allegro mountsAllegro L 240 · R 240; LEAP L 240 · R 240LEAP hands float 24.7 mm off the arm (no LEAP mount); the left arm swings 29.5 deg per frame at k 211–214this stage
Conditional1
B6FV-B01149958.1–966.0 sTwoclose-up with Allegro mountsAllegro L 240 · R 240; LEAP L 240 · R 240a readable delivery‑app banner from k ~125; LEAP hands detached; the Allegro hands touch at k 100 (1.2 mm)home Fig. 3, Spring onions trial
Honesty example1
S1FV-Z0090720.0–28.0 sOne(right)Allegro 186; LEAP 18654 frames (k 184–237) have no hand estimate and stay grey; small hands; an arm jump at k 238 after the gaphome Fig. 3, Green beans trial
Variants6
B1FV-B01774486.0–494.0 sTwostandard viewAllegro L 240 · R 240; LEAP L 240 · R 240as B3Not showna standard view variant of B3, which shows the same solution
B2FV-B01774486.0–494.0 sTwoclose-upAllegro L 240 · R 240; LEAP L 240 · R 240as B3Not showna close‑up variant of B3, which shows the same solution
B4FV-B01149958.1–966.0 sTwostandard viewAllegro L 240 · R 240; LEAP L 240 · R 240as B6Not showna standard view variant of B6, which shows the same solution (and the same banner)
B5FV-B01149958.1–966.0 sTwoclose-upAllegro L 240 · R 240; LEAP L 240 · R 240as B6Not showna close‑up variant of B6, which shows the same solution (and the same banner)
B7FV-B0047620.0–28.0 sTwostandard viewAllegro L 240 · R 240; LEAP L 240 · R 240as B9Not showna standard view variant of B9, which shows the same solution
B8FV-B0047620.0–28.0 sTwoclose-upAllegro L 240 · R 240; LEAP L 240 · R 240as B9Not showna close‑up variant of B9, which shows the same solution
Not recommended1
B9FV-B0047620.0–28.0 sTwoclose-up with Allegro mountsAllegro L 240 · R 240; LEAP L 240 · R 240the two robot hands pass into each other (Allegro, 54 frames, up to 11.5 mm); hidden hands; a seam at the mounthome Fig. 3, Latch trial
Not used4
S2FV-B01774486.0–494.0 sOne(right)Allegro 240; LEAP 240the knife grip barely movesNot showna one‑hand run of the thread, which B3 shows two‑handed
S3FV-B01773256.8–264.8 sOne(right)Allegro 239; LEAP 240a legible brand logo at k ~230–239; small hands; a 0.28 m jump at the loop seamNot showna legible brand logo at k ~230–239
S4FV-B01149958.1–966.0 sOne(right)Allegro 240; LEAP 240a readable delivery‑app bannerNot showna readable delivery‑app banner; B6 shows the clip
S5FV-B0047620.0–28.0 sOne(right)Allegro 240; LEAP 240the robot hands bob up to ~11 cm (a depth swing)Not showna one‑hand run of the latch clip, which B9 shows two‑handed

14 rows, none left outHand estimates checked on 12 recordings; retargeted on five.

A bar is one robot arm: the lit part is the frames with a robot solution.

Notes (3)
  • Frames solved: our retargeting summary for each robot arm; a frame without a hand estimate stays grey and is not solved.
  • Review: our audit of the robot-hand renders. Continuity limits were loosened per clip (0.03 to 0.12 m and 15 to 35 deg per frame against the published 0.02 m and 15 deg).
  • Hand poses are monocular estimates; nothing here measures their accuracy. Simulated in PyBullet, no object modelled.
Every number 8
  • Both hands solved in 240 of 240 frames on 4 arms.Source: our audit of the robot‑hand renders (their retargeting summaries and the sync check)
  • Continuity limits of 0.1 m per frame for the hand and 30° for the arm, against the method's defaults of 0.02 m and 15°; at the defaults the left hand solves in 198 of 240 frames. Solved with per‑clip choices: wearer hints, exclusion zones and a camera fit.Source: our audit of the robot‑hand renders (their retargeting summaries and the sync check)
  • Sync check on the render masters: 49.10, 48.75, 45.16 dB against a 35 dB floor. It checks frame indexing only, not how well the hand estimate is timed.Source: our audit of the robot‑hand renders (their retargeting summaries and the sync check)
  • Robot grip opening against the estimated human one (correlation, Allegro / LEAP): left 0.97 / 0.98, right 0.81 / 0.91 (the right hand holds the knife).Source: our audit of the robot‑hand renders (their retargeting summaries and the sync check)
  • In this clip the Allegro thumb misses its target by 37.4 mm (left) and 43.1 mm (right) on average.Source: our retargeting summary for each hand of this clip, mean over the solved frames
  • ORCA replay: FV-B01774's right hand at loop frames k 0–209 (486.0–493.0 s, 30.0 frames a second), with its own estimate in 210 of 210 frames, retargeted to 16 finger joints, each frame on its own. Before the display smoothing, a joint moves a median 2.6° between frames, 16.1° at the 95th percentile and at most 79.3°.Source: the replay's data (computed at build from our HaMeR estimates, retargeted with the hand prior's retargeter)
  • The loop shows frames k 0–209: 7.0 s shown for 7.007 s filmed (loop frame k = camera frame 2k; it runs 0.1% fast).Source: the layer scrubber's clock (computed at build from the render and the camera frames)
  • Across all 14 of our Allegro renders, each hand's mean thumb miss lies between 31.7 and 43.3 mm.Source: the retargeting summaries of every final in stage 5's ledger (mean thumb miss per hand, over the solved frames)

06Robot video

Arms out. Robot in.

Our re-implementations of Masquerade and a contact-aware pass, on the same frames.

FV-B01774 · 486.0–494.0 sthread clip, edited footage

Edited footage

Edited footage: the arms are painted out by a model and a simulated robot is drawn in. Not robot footage.

Edited footage: three panels of hands slicing shallots on a green board; in the middle and right panels the arms are replaced by simulated robot arms.
k 0

Edited footage. One three-panel file, so both sides of the divider are the same decoded frame.

Notes (4)
  • The arms are painted out by a model and simulated robot arms are drawn in, each posed from the hand estimated in the same frame. No gripper visibly holds anything: held objects are partly painted out with the hands.
  • Our re-implementations of published methods (Masquerade, arXiv 2508.09976; a contact-aware pass after arXiv 2610.06137, whose authors released no code). Not the authors' code or results.
  • On this clip the pads sit 2.2 to 5.8 mm from another model's contact estimates, against 13.5 to 42.6 mm; those estimates are uncertain by centimetres, so this is not measured contact.
  • The contact-aware depth test hides up to 27.2% of the right gripper at k 155.
The edit, in six stepsOne frame of the thread through each step of our re-implementation.

Edited footage: the arms are painted out by a model and a simulated robot is drawn in. Not robot footage.

  1. Edited footage: the worker's two arms covered by orange and green masks with keypoint dots.
    Masks A segmentation model outlines each arm in every frame (orange right, green left), with the hand estimate's keypoints.
  2. Edited footage: two white simulated robot arms with black grippers on a grey background.
    Robot Two simulated robot arms posed from the hand estimates, before they are drawn into the frame.
  3. Edited footage: simulated robot arms drawn over the unedited frame, the worker's arms still showing underneath.
    Overlay An alignment check, with the human arms still visible underneath.
  4. Edited footage: the kitchen board with the worker's arms painted over by a model, leaving soft smears.
    Rough paint Every masked pixel is painted over, including frames without a hand estimate. It is not an output.
  5. Edited footage: the kitchen board with the worker's arms painted out.
    Paint out The arms painted out where they will be replaced.
  6. Edited footage: the painted kitchen frame with two simulated robot arms drawn in.
    Composite The simulated robot drawn into the painted frame.

Three fixes we tried, before and afterEach pair is one frame, before and after the change.

Edited footage: the arms are painted out by a model and a simulated robot is drawn in. Not robot footage.

  • Paint at full size

    At 342×256 the painted area smears. The same painter at 960×720 is the size we kept.

    Edited footage: the thread's frame with the arms painted out at 342×256 and scaled up; the painted area smears.
    Before 342×256not kept
    Edited footage: the same frame with the arms painted out at 960×720.
    After 960×720kept

  • Try a second painter

    ProPainter in place of E2FGVI‑HQ, on the same masks: both leave a teal smear, so E2FGVI‑HQ stayed.

    Edited footage: the same frame with the arms painted out by E2FGVI-HQ; a teal smear at the bottom right.
    Before E2FGVI‑HQkept
    Edited footage: the same frame with the arms painted out by ProPainter; a teal smear at the bottom right.
    After ProPainternot kept

  • Roll the gripper the authors' way

    Our gripper roll, against the authors', which closes the gripper across the thumb-index line. We kept theirs.

    Edited footage: the same frame with two simulated robot arms drawn in, the grippers rolled by our fix.
    Before Our rollnot kept
    Edited footage: the same frame with two simulated robot arms drawn in, the grippers rolled the authors' way.
    After The authors' rollkept
Every Masquerade clip14Passing or failing the both-arms rule, with where each one plays.

Data

Clip, workplaceFrames edited of 240What the QA foundWhere it plays
Passes the rule7
FV-C01442 171.8–179.8 sCarpentryR 240 · L 240a smear; the contact‑aware right gripper is hidden at k 33–45 and 228–239home Fig. 7a, Carpentry tab
FV-C01720 676.5–684.5 sHVAC servicesR 240 · L 240the grippers pass through each other in 43 frames; shown from k 46home Fig. 7a, HVAC tab
FV-B01919 416.7–424.7 sMetalR 240 · L 240a 49.8 deg joint step at k 169–170; the contact forearm covers 22–24% of the frame at k 0–32home Fig. 7a, Metal tab
FV-B01174 62.7–70.7 sHVAC servicesR 240 · L 240the grippers fuse in 46–49 frameshome Fig. 7a, Small parts tab
FV-B01774 486.0–494.0 sKitchenR 240 · L 240the contact‑aware depth test hides up to 27.2% of the right gripper at k 155; a teal smearthis stage
FV-B01149 958.1–966.0 sKitchenR 240 · L 240the wearer's fingers left in the painted frame at k 180Not showna readable delivery‑app banner
GX010005 879.1–887.1 sCarpentryR 240 · L 240the elbow snaps between framesNot shownreligious print on the wearer's garment in every frame
Fails the rule7
FV-C01283 1140.4–1148.4 sKitchenR 239 · L 240right arm unedited in 1 frameNot shown
FV-C01321 285.7–293.7 sCarpentryR 194 · L 240right arm unedited in 46 framesNot shown
FV-B00476 20.0–28.0 sMetalR 240 · L 221left arm unedited in 19 framesNot shown
FV-Z00907 20.0–28.0 sKitchenR 174 · L 213right arm unedited in 66 frames; left arm unedited in 27 framesNot shown
FV-B01773 80.0–88.0 sKitchenR 193 · L 217right arm unedited in 47 frames; left arm unedited in 23 framesNot shown
FV-B01773 256.8–264.8 sKitchenR 234 · L 225right arm unedited in 6 frames; left arm unedited in 15 framesNot shown
FV-B01784 122.2–130.2 sKitchenR 217 · L 222right arm unedited in 23 frames; left arm unedited in 18 framesNot shown

14 rows, none left out1,183 archive rows (renders, intermediates and encoded frame folders) from 13 recordings went into these.

7 of 14 meet the rule (both arms edited in every one of the 240 frames). A bar is one arm: the lit part is the frames where it was replaced.

Notes (3)
  • Frames edited: our final edit plan for each clip, per arm, of 240; the rule is both arms edited in every one of the 240 frames.
  • Our re-implementation of Masquerade (arXiv 2508.09976); no gripper holds anything, and painted pixels are generated, not footage.
  • Passing clips that are not shown fail on content, not on the rule.
Every number 4
  • Our re‑implementations: Masquerade (arXiv 2508.09976), and a contact‑aware pass after arXiv 2610.06137.Source: the papers, cited as the authors' methods; the code and the results here are ours
  • Both arms replaced in all 240 frames of the clip, in both edited panels.Source: renders.json (the render's own length and frame rate) and our QA of this clip
  • Contact‑aware pads 2.2 to 5.8 mm (median) from HACO's estimated contacts, against 13.5 to 42.6 mm for Masquerade on the same rig. Distances to another model's estimates, not measured contact.Source: our comparison of the two methods on this clip, against HACO's estimated contacts
  • Occlusion is not depth‑correct: up to 27.2% of a gripper is hidden at k 155.Source: our audit of the edited‑footage renders

07Hands in common

Hands have habits.

Eight of 16 directions hold 91.8% of the posture variance of 55,088 estimated poses.

Render: the ORCA v2 hand model at −2 σ, the mean posture and +2 σ along PC1 (top row: open hand to fist) and PC2 (bottom row: knuckles flexed with straight fingers against curled tips).
Render of the simulated hand model at three settings of the two stable directions.

Allegro mappingTop-2 overlap with EigenDEXplore's basis (arXiv 2610.07681v1): 0.81, where random planes reach 0.27 (95th percentile); 0.86 for our earlier retargeting. Our first direction is no closer to theirs than a plain closing motion (0.49 against 0.49).

Published bases need 3 to 10 directions for 90%; the 16-joint Allegro one needs 10.

  • PC1 and PC2, stable. They keep their identity under the official ORCA retargeting objective (cosine 0.95 and 0.87).
  • PC3 and PC4, provisional. Under that objective they swap and mix (cosine 0.14 and 0.33), though the top-4 subspace largely holds (overlap 0.86).
  • PC5 to PC8, unstable. They change even between random halves of our own data.

Not shown: a training gain. The prior has not been used to train a robot; in our only pipeline check, random directions learned faster at that budget. The poses are monocular estimates, and the hand detector and wearer rule behind them have not been scored against labelled frames.

Watch the six-beat tour on the home page

Every direction, every trade

The first direction, closing the whole hand, holds 42.9% of the variance; the top 3 hold 69.1%. Each trade alone needs 7 or 8 directions, and the pooled top 8 capture 89.4% to 92.6% of each trade's variance.

Share of the posture variance per direction (55,088 poses, 16 ORCA v2 finger joints)
DirectionShareCumulativeσ (rad)
PC142.9%42.9%0.91
PC216.8%59.7%0.57
PC39.4%69.1%0.43
PC47.0%76.1%0.37
PC55.2%81.2%0.32
PC64.5%85.8%0.30
PC73.2%88.9%0.25
PC82.9%91.8%0.23
Per trade
TradeDirections for 90%Pooled top 8 capturePC1 share
Kitchen792.6%43.8%
HVAC and electrical891.9%46.5%
Carpentry792.0%47.9%
Welding and fabrication889.4%33.3%

Real hands: the poses nearest the model's posture, each from a different recording, cropped to the hand's box from a frame of the camera's low‑resolution proxy, with HaMeR's estimated 21‑keypoint skeleton. Left hands were mirrored onto the right hand for the fit, not in the crops.

Hand model: the ORCA v2 right hand from its open description (MIT licence), simplified for the web; a simulated model, not a robot recording. ORCA is a trademark of its owners, who are not affiliated with this site and do not endorse it.

More clips across workplaces

Proxy clips, cut from the camera's low-resolution file. Each task line is a reviewer's reading, not a label.

Every number 4
  • 55,088 wearer hand poses; 8 of 16 directions hold 91.8% of the posture variance; the first holds 42.9%.Source: the figure's data, generated from our hand‑prior fit report
  • About 2 hours of kitchen, HVAC, carpentry and welding work, estimated at 6 frames a second; each trade alone needs 7 or 8 directions, and the pooled top 8 capture 89.4% to 92.6% of each trade's variance.Source: our hand‑prior report
  • Published bases need 3 to 10 directions for 90%; the 16-joint Allegro one needs 10.Source: published hand‑posture bases, as compared in our hand‑prior report
  • On an Allegro mapping, top-2 overlap with EigenDEXplore's basis 0.81, where random planes reach 0.27 (95th percentile).Source: our hand‑prior report

08Training

Next frame. Next move.

World models predict the next frames; policies choose the next action.

FV-Z00907 · 20.0–28.0 sanother clip, data: head or scene

HERO10 Black Widestabilisation off14.98 frames/s steps

The earlier frame is warped by the rotation the gyro measured, through a provisional calibration. Rotation only: the head's movement through space and the rolling shutter stay in the residual, with the hands and near objects.
Picture change per step on FV-Z00907: the part the head's rotation predicts and the residual
Time (s)Total changeHead rotationResidualHead share
20.0122.97.015.931%
20.0726.111.514.644%
20.1427.514.912.654%
20.2124.414.79.760%
20.2715.14.210.928%
20.3420.15.115.125%
20.4124.25.818.424%
20.4723.93.320.714%
20.5425.43.921.615%
20.6120.40.719.74%
20.6719.03.515.419%
20.7416.93.513.521%
20.8116.85.711.134%
20.879.20.09.80%
20.9410.50.010.60%
21.0113.73.510.325%
21.0713.34.29.132%
21.1415.82.213.614%
21.2119.82.817.014%
21.2715.40.018.10%
21.3410.30.017.10%
21.4111.60.016.10%
21.4713.40.017.40%
21.5416.40.018.20%
21.6120.04.016.020%
21.6825.612.213.548%
21.7427.114.512.753%
21.8127.315.411.856%
21.8822.514.48.164%
21.9424.114.010.258%
22.0132.017.314.754%
22.0735.018.516.553%
22.1430.916.014.952%
22.2121.28.013.338%
22.2736.016.819.247%
22.3432.816.316.650%
22.4124.814.210.657%
22.4823.79.114.639%
22.5432.315.217.147%
22.6123.96.617.328%
22.6822.12.719.412%
22.7423.52.421.110%
22.8129.07.421.625%
22.8823.01.022.04%
22.9417.00.020.70%
23.0120.92.418.511%
23.0823.98.515.336%
23.1420.69.411.146%
23.2115.15.99.239%
23.2813.11.811.314%
23.3417.64.213.524%
23.4116.81.914.911%
23.4830.413.117.343%
23.5430.313.217.044%
23.6129.213.615.746%
23.6824.510.514.043%
23.7414.43.211.123%
23.8115.97.78.249%
23.8812.85.87.045%
23.9412.66.36.350%
24.019.23.55.738%
24.089.74.15.742%
24.1410.64.26.440%
24.2111.44.66.841%
24.2812.95.77.244%
24.3414.17.46.852%
24.4112.56.06.548%
24.487.72.45.331%
24.5411.45.36.147%
24.6111.75.95.850%
24.688.02.75.334%
24.746.71.55.223%
24.818.53.05.436%
24.8811.15.35.847%
24.958.73.55.240%
25.017.41.16.315%
25.0811.34.07.335%
25.149.82.37.523%
25.2111.63.97.634%
25.2810.63.47.232%
25.3415.48.46.955%
25.4116.99.17.754%
25.4823.013.09.957%
25.5526.315.211.158%
25.6115.94.811.130%
25.6832.418.314.156%
25.7532.219.013.259%
25.8125.213.012.252%
25.8832.918.014.955%
25.9529.915.214.751%
26.0124.713.111.653%
26.0817.07.010.041%
26.1525.112.512.650%
26.2120.58.112.439%
26.2822.910.512.446%
26.3525.013.911.156%
26.4124.614.010.657%
26.4826.315.211.258%
26.5525.412.812.651%
26.6122.19.912.245%
26.6822.611.011.649%
26.7522.311.111.250%
26.8121.99.712.244%
26.8818.89.09.848%
26.9513.26.37.047%
27.015.80.55.38%
27.084.10.04.90%
27.1522.414.18.463%
27.2120.210.79.553%
27.2818.69.78.952%
27.3522.211.610.652%
27.4118.49.58.852%
27.4815.95.110.732%
27.5517.22.814.416%
27.6116.81.715.010%
27.6824.06.917.129%
27.7528.57.820.827%
27.8130.56.024.520%
27.8824.62.022.68%

FV-B00476 · 36.0–44.0 sanother clip, footage: what happens next, in metal work

Reviewer's reading, not a label: Both hands work a small bolt on the gate latch, then the right hand takes a tool.

Real footage36.0–44.0 s

Context

  1. 37.0 s
  2. 38.0 s
  3. 39.0 s
  4. 39.5 s

What actually happened

  1. 40.0 s
  2. 40.5 s
  3. 41.5 s
  4. 42.5 s
FV-B00476·frames at 12 fps
Published work, the authors' results4What labs report from human video. None of it is our result.
  • DreamDojo arXiv 2602.06949, ICML 2026: pretraining a robot world model on about 44,000 h of egocentric human video improved its prediction of robot video, mostly from adding the stage at all.
  • EgoVerse arXiv 2604.07607: co-training policies on human video, with hand pose as the action.
  • Human2Sim2Robot arXiv 2504.12609, CoRL 2025: retargeting human hand motion onto a robot hand, then refining it in simulation.
  • Masquerade arXiv 2508.09976: editing the human arms in video into a robot's.
Every number 2
  • FV-Z00907: a rotation‑only model gives the head a median 40.3% of each step's picture change (38.3% pooled over 119 steps).Source: the head‑or‑scene data (computed at build from the clip's frames, its gyro and the provisional lens calibration)
  • DreamDojo: about 44,000 h of egocentric human video in pretraining.Source: DreamDojo, arXiv 2602.06949, ICML 2026 (the authors' result)

09What a lab receives

Four files. Then round two.

One clip's delivery, unboxed.

  • FV-B01774.MP4

    59.94 fps

    picture, sound, motion track

    The camera's own file.

  • FV-B01774_imu.csv

    head speed

    1,613 samples

    486.0–494.0 s

    Head motion, on the video clock.

  • FV-B01774_labels.json

    10.1.01 Vegetable washing and chopping 

    taxonomy 1.0.0

    Which task, and when.

  • FV-B01774_meta.json

    environment_id
    10
    fov
    linear
    imu
    true

    Workplace, lens and consent.

FV-B01774/

Also as LeRobot v3 · RLDS · HDF5 · MCAP

What each file holds

.MP4 The camera original

File
The camera's own file: picture, sound, and the camera's metadata and motion track.
Worn
A head- or chest-mounted action camera, worn by the worker on their own shift.
This clip
59.94 fps, Linear lens.
On this site
Camera modes of the clips here: 1920×1080 at 29.97 fps, 50 fps or 59.94 fps; 2704×2028 at 50 fps; 2704×1520 at 50 fps, 59.94 fps or 100 fps; 3840×2160 at 50 fps; 5312×2988 at 59.94 fps.

Lens and mount: stage 1 →

_imu.csv Columns and units

timestamp_us
Integer microseconds on the video's own clock. The first video frame is 0.
ax, ay, az
Accelerometer, m/s², gravity included: about 9.8 on one axis at rest.
gx, gy, gz
Gyroscope, rad/s.

Frames and samples on one clock: stage 2 →

_labels.json Codes and segments

Codes
ENV.C.SS, frozen: 10.1.01 is task 01 in task group 1 of workplace 10. The files call ENV environment_id.
Segments
In order, each with its code, name, start_us and end_us on the clip's own clock.
Taxonomy
354 tasks that never change, public at /taxonomy.json and /taxonomy.csv.
Narration
In some workplaces the worker describes each action aloud, on the clip's own audio.
Mistakes
Drops, re-grips and do-overs are labelled as such. Long takes, not highlights.

Codes, then words: stage 4 →

_meta.json One record per clip

Ids
Sites and workers appear only as pseudonymous ids.
Keys
clip_iddevicemountsite_idcollector_idcountryenvironment_idcaptured_atduration_usresolutionfpsfovimuhands_in_frame_pctpeople_in_clipincidental_individualsconsent
Consent
Workers consent to capture and to machine-learning training on their footage. Anyone regularly in shot signs a consent form before capture begins.

The consent policy →

Tell us where it slips.

Only footage an independent expert confirms meets your criteria is billed.

Request a round
Every number 3
  • 4 files per clip, 3 without IMU.Source: the delivery layout (the free sample's folder holds the same four files)
  • FV-B01774 was filmed at 59.94 frames a second: 480 frames in 486.0–494.0 s.Source: the layer scrubber's clock (computed at build from the render and the camera frames)
  • The _imu.csv card draws all 1,613 gyro samples of 486.0–494.0 s as head speed, the norm of the three gyro axes.Source: the head‑motion data (computed at build from the camera's GPMF gyro track, extracted by our pipeline)

Credit where it's due.

The models, code, meshes and libraries behind the renders and figures here and on the home page, each under its licence.

Hand estimates and the hand‑posture figure (stages 3, 5 and 7)

  • HaMeR (MIT (code); licensed by the owner): estimates the 21 hand keypoints frame by frame (frames without an estimate are listed), drawn as skeletons and driving every robot render.
  • MANO hand model (licensed by the owner): the hand model inside HaMeR's estimates; never drawn on this site.
  • WiLoR (detector weights CC BY‑NC‑ND; licensed by the owner): its hand detector finds the hands behind the hand‑posture figure (stage 7).
  • Ultralytics (AGPL‑3.0): runs that detector, offline.
  • MediaPipe (Apache‑2.0): hand detection in the robot‑render pipeline.
  • ORCA v2 hand description (orcahand_description) (MIT): the 16‑joint hand model in stages 5 and 7. Copyright (c) 2025 ORCA. "ORCA Hand by ORCA Dexterity, Inc.", CC BY 4.0, cited to IROS 2025. ORCA is a trademark of ORCA Dexterity, Inc.; its use here names the hand model only and is not an endorsement.
  • orca_core and orca_retargeter (MIT): map the estimated hand poses onto the ORCA v2 joints.
  • three.js (MIT): draws the stage 5 hand replay and the stage 7 hand model and posture cloud.

Simulated robot hands (the opening figure and stage 5)

  • Human2Sim2Robot (MIT): the authors' retargeting code, used with four documented line changes.
  • LEAP_Hand_Sim (MIT): the LEAP hand model.
  • cuRobo (licensed by the owner; meshes BSD‑2‑Clause): solves the robot arms; its Allegro and iiwa meshes.
  • PyBullet (zlib): renders the simulated hands and arms.

Edited footage (the opening figure and stage 6)

  • Masquerade code base (Phantom) (MIT): the starting point of our re‑implementation. Copyright (c) 2025 Stanford Interactive Perception and Robot Learning Lab.
  • SAM 2 (Apache‑2.0): outlines the arms and held objects in every frame.
  • E2FGVI‑HQ (licensed by the owner): paints the arms out.
  • ProPainter (licensed by the owner): the alternative painter we compared.
  • HACO (licensed by the owner; its checkpoint is CC BY‑NC‑SA 4.0 (share‑alike)): the contact estimates the contact‑aware pass aims at.
  • MoGe‑2 (MIT): estimated depth for the contact‑aware pass.
  • robosuite models of the Kinova Gen3 arm and Robotiq 2F‑85 gripper (MIT): the simulated arm and gripper drawn in.
  • MuJoCo (Apache‑2.0): the physics engine under those robosuite models.

Robot and hand models in the home page figures (Fig. 3 and Fig. 4)

  • KUKA iiwa 14 and Allegro hand meshes (from cuRobo) (meshes BSD‑2‑Clause, from cuRobo (licensed by the owner)): the simulated arm and hand in the 3D replay of home Fig. 3. Copyright (c) 2016‑2019 Salvatore Virga, Marco Esposito (Technische Universität München); Copyright (c) 2015 Robert Krug, Todor Stoyanov (AASS Research Center, Örebro University); Copyright (c) 2016 SimLab. All rights reserved. Redistributed under the BSD 2‑Clause License, which the copyright holders and contributors provide "as is", without any express or implied warranty.
  • LEAP Hand (from LEAP_Hand_Sim) (MIT): the second simulated robot hand in home Fig. 3. Copyright 2023 Ananye Agarwal.
  • ORCA v2 hand (from orcahand_description) (MIT): the ORCA hand option in home Fig. 3 and the 16‑joint hand in home Fig. 4 and stage 7. Copyright (c) 2025 ORCA. ORCA is a trademark of ORCA Dexterity, Inc., which does not endorse this site.
  • Sharpa HA4 left hand (meshes from github.com/tylerlum/simtoolreal @ 313d5ae) (MIT): the left Sharpa HA4 hand beside our ORCA hand in home Fig. 4, beat 6, converted to a web mesh. Copyright (c) 2026 Tyler Lum. The package's own package.xml says BSD; the repository licence is MIT. No official Sharpa model was used, and Sharpa does not endorse this site.
  • Sharpa and Allegro bases: EigenDEXplore (hgupt3/eigendexplore @ 5d57180) (MIT): the published posture directions that drive the Sharpa hand in home Fig. 4, beat 6, and the Allegro basis stage 7 compares against (the authors' results, shown as published). Copyright (c) 2026 Action‑Bench contributors.

Measured depth: the door model, the simulation and the figures (/depth)

  • FoundationStereo (licensed by the owner): the stereo matcher: for each pixel it finds the shift between the two cameras' pictures; depth follows from that shift.
  • COLMAP (BSD): the rig's calibration, fitted to its own footage.
  • MediaPipe (Apache‑2.0): marks the hand joints in the left picture; each joint is placed at the stereo depth of the skin under it. Its own hand model gives the wrist chart's one‑camera line.
  • SAM 2 (Apache‑2.0): outlines the hands, the forearms and the door leaf in the frames that are measured.
  • PP‑OCRv3 text detector (Apache‑2.0): finds printed paper in the camera's pictures so that it can be blurred.
  • three.js (MIT): draws the measured depth points of the lock clip.
  • MuJoCo (Apache‑2.0): the physics simulator: the door model, and the episodes run again with every contact recorded (the film and the contact shares).
  • MuJoCo Warp (with NVIDIA Warp) (Apache‑2.0): the GPU version of the simulator that trained the hand and ran the test episodes of each recipe.
  • PyTorch (BSD‑3‑Clause): trains the simulated hand.
  • ORCA v2 hand model (orca_sim and orcahand_description) (orcahand_description MIT; orca_sim licensed by the owner): the simulated hand at the door's lever, on a floating wrist. Copyright (c) 2025 ORCA. "ORCA Hand by ORCA Dexterity, Inc.", CC BY 4.0, cited to IROS 2025. ORCA is a trademark of ORCA Dexterity, Inc.; its use here names the hand model only and is not an endorsement.
  • Blender (Cycles) (GPL): a tool: it renders the door still and the simulation film from the model's geometry and the recorded simulator states, one state a frame.

Related work

  • EigenDEXplore (arXiv 2610.07681v1, preprint): related work: its published Allegro basis is what stage 7 compares against, using its code (MIT) for the Allegro mapping. Cited as the authors' result; they have no part in this site.

No author or maker, including ORCA Dexterity, Sharpa, KUKA, Wonik Robotics (Allegro), Kinova and Robotiq, made, reviewed or endorses these renders or this site.