Get a quote
Stereo depth

Get a handle on it.

A robot hand has to know how far the handle is. Flat video leaves it guessing.

Head-worn stereo measures depth, frame by frame; we rebuild the door for simulation.

Data

One stereo frame of the lock fitting, seen from outside the worker's head. Hand joints: Estimated.

  • Measured
  • Estimated
Data: the lock fitting seen from outside the camera as a shell of coloured points, both hands in front of the lock and the door, with the unmeasured pixels left behind on a flat sheet.

Data

Lock fitted600.0 to 608.0 s

Door pushed open869.0 to 874.2 s

Door pulled shut903.0 to 906.4 s

  1. Matched
  2. Lifted
  3. Hinged
off the recording
  1. Rebuilt
  2. Tested
  3. Counted
Same worker, same rig, same door: the lock clip, then the door clips minutes later.
  • Measured

The gap does the measuring.

Two lenses, set like eyes: the shift between pictures gives depth, pixel by pixel.

Footage

The stereo pair from the head rig, right picture over left. Lime: checked pixels at a shift close to the read-out's. Printed paper is blurred.

Footage: the two cameras' pictures of a brass lever lock being fitted into a door, laid over each other, with a lime band on the lock where the two line up.
shift122.5pxdepth0.29m
Drag the picture or the slider. The lime lights every checked pixel at the read-out's depth. Near things need a long slide; far things hardly move.Drag sideways, or use the slider. The lime lights every checked pixel at the read-out's depth. Near things need a long slide; far things hardly move.
Every number 10
  • The lenses sit 62.9 mm apart, and every distance on this page scales with that figure.Source: our self‑calibration of the rig, fitted to its own footage and head IMU.
  • Two fisheye cameras, each seeing about 158 by 94 degrees, at 30 frames a second. The head IMU adds about 400 readings a second.Source: our calibration of the rig: the field of view from the fitted lens model; the two rates from the recording's own timestamps.
  • Focal length of the straightened pictures: 560.2 px. Shifts are counted in pixels of that frame, 1600 px wide.Source: the same calibration.
  • Depth is distance along the camera's line of sight, f b / d. A point off to one side is farther away in a straight line.Source: the geometry of two parallel cameras.
  • Lock: a shift of 122.5 px, 0.29 m deep. Pillar outside: about 9 px, about 3.9 m.Source: this pair's checked shifts from the same matcher as the band, read at the two marked points, each the median of a small patch.
  • At the pillar outside, two matchers read 9.0 px and 9.2 px.Source: the page's matcher and a second, plain block matcher, on this pair.
  • A pixel counts as checked where the matches from the left picture and from the right picture differ by at most one pixel.Source: the left‑right check, as run on this pair.
  • The band lights every checked pixel whose shift lies within 1.5 px of the slider.Source: the figure's own rule.
  • The figure shows one stereo pair, taken 600.2 s into the recording.Source: the pair's own time stamp.
  • The shifts in the band come from the same stereo matcher as the rest of this page.Source: this frame's depth data.
  • Measured
  • Estimated

Measure twice.

A frame later, still surfaces repeat to 0.5 mm (median, under 1.0 m).

Data

A grid of points, each coloured from the video picture and placed at its stereo depth.

Data: the lock fitting seen from outside the camera as a shell of coloured points, both hands in front of the lock and the door, with the unmeasured pixels left behind on a flat sheet.
10 cm · scale known to about 4%Where the lenses wereHead pathUnchecked: left flat
  • Left hand
  • Right hand
  • Hand joints: Estimated, at the skin
frame 1 of 240

Drag sideways to turn the view.

Flat, until you move. Step outside the worker's head and each frame stands in metres: hands, lock and door in one measured space.

How it is drawn
  • The flat sheet stands 0.37 m from the lens: a drawing choice.
  • A few checked points at a sharp jump in depth are also left on the sheet. In the outside view, points fade out beyond 1.2 m.
  • The outside view is a virtual camera.

Data

straight-line distance, left lens to right wrist

  • StereoMeasured
  • One camera, hand of assumed sizeEstimated
  • unseen
Estimated where. Measured how far. Same wrist, same frames: with stereo, the reach reads steady from frame to frame.
Where the depth was checked

Data

Each dot is a patch of the camera's view, larger where more of the clip's frames had a checked depth.

Lock
Every number 10
  • A still surface, measured again in the next frame, differs by 0.5 mm (median, at 0.15 to 1.0 m).Source: our repeat‑reading test on the stretch of the recording around this clip: still surfaces, hands and forearms left out, the head's own motion removed. A calm stretch, the head nearly still.
  • By distance: about 0.3 mm at 0.15 to 0.25 m, 1.4 mm at 0.5 to 0.7 m and 2.4 mm at 0.7 to 1.0 m (medians).Source: the same test, split by measured depth.
  • Nine in ten of the same differences are 3.0 mm or less.Source: the same test.
  • About 1.3 million of 1.92 million pixels per frame carry a checked depth (median).Source: the left‑right check on every frame of this clip: a pixel is kept where the matches from the two pictures differ by at most one pixel.
  • The figure draws a thinned grid of 400 by 300 points. The receipt counts the full frame.Source: the figure's own data pack.
  • The clip is 240 frames at 30 frames a second, 600.0 to 608.0 s of the recording.Source: the data pack's index.
  • The scale bar inherits the rig's scale, known to about 4%.Source: our calibration of the rig, fitted to its own footage and motion sensor.
  • Each hand is drawn with its 21 joints where the detector found them; a missing joint stays missing.Source: a hand detector's landmarks in the left picture, each placed at the stereo depth of the skin under it.
  • The head path, and the receipt's step between frames, come from the head's pose in each frame.Source: stereo odometry on this clip's own pairs, levelled with the motion sensor's gravity.
  • The one‑camera line places the same wrist by fitting a hand of assumed size to the left picture alone.Source: the hand detector's own hand model, in the same frames.
  • Measured
  • Assumed

Hinged on evidence.

Hinge fitted from the stereo footage; the swing replays the worker's, reading by reading.

Data

One line for each stereo frame that saw the door, drawn from above, around the fitted hinge.

door 44° from assumed shut

shut: assumedwall10 mmfitted hinge

Drag the door's free edge, or step through the measured readings with the arrow keys.

unseen
about 0.78 mabout 2.11 mspindle about 1.02 m above the floor, about 4 cm either waytop edge,
within several cm

hatched: assumed

From the room10 mmlever, 94.6 mmlatch
From aboveabout 34 mm thickfar-side lever
From above, the lines nearly meet at one point: that tight scatter is the hinge's evidence, its axis and swing taken from the worker's own push.
Every number 6
  • The hinge line is fitted to the plane of the leaf (the door panel itself) as measured in 178 stereo frames. Some of them rest on a head path that was wrong in places.Source: our measurement of the door from the door clips of this recording; each plane was placed in the room by the head path.
  • Leaf about 0.78 × 2.11 m, its top uncertain by several centimetres.Source: the same measurement.
  • Flat brass lever, 94.6 mm long, its spindle (the bar the lever turns) about 1.02 m above the floor (about 4 cm either way).Source: the lever as seen from above and from the front in the same clips, and the floor plane fitted in them.
  • Leaf about 34 mm thick.Source: the median of the frames that see the leaf's edge face.
  • The opening angle is read 30 times a second, every reading as measured, breaks left open.Source: the swing data behind this figure.
  • Every length here scales with the gap between the lenses, known to about 4%.Source: our calibration of the rig, fitted to its own footage and motion sensor.
  • Measured
  • Assumed

Sim-ready, every value tagged.

Shape from the stereo footage; mass, friction, spring and latch chosen.

Render

Render of the door model: solid parts measured from the stereo footage, glass parts assumed; the glass band on top spans the doubt in the leaf's height.

Render: a door standing free on a pale ground. Its leaf, brass plate, rose and flat lever are solid; its latch, far-side lever and the top band of the leaf are frosted glass.
Room side
Far side
Each dot opens its part's row below.

Measured25

PartValueSeen inSpread
Leaf (the door panel)
Widthabout 0.78 m6 frames for the free edge, 16 for the hinge endfree edge 1.8 mm
Thicknessabout 34 mm10 frames (median)
Seen in · Spread
Width
Seen in 6 frames for the free edge, 16 for the hinge end
Spread free edge 1.8 mm
Thickness
Seen in 10 frames (median)
Top of the leaf
Heightabout 2.11 m25 frame pairsseveral centimetres: two parts of the recording disagree
Seen in · Spread
Height
Seen in 25 frame pairs
Spread several centimetres: two parts of the recording disagree
Brass plate
Width44.7 mm5 frames2.3 mm
Height44.3 mm1 frame of the lock clipnone: one frame
Thickness8.6 mm2 readings
Seen in · Spread
Width
Seen in 5 frames
Spread 2.3 mm
Height
Seen in 1 frame of the lock clip
Spread none: one frame
Thickness
Seen in 2 readings
Rose (round collar behind the lever)
Radius18.1 mm5 front viewsdiameter 1.1 mm
Front, off the face28.2 mm2 readings
Centre above the floorabout 1.02 m5 front viewsabout 4 cm either way: two parts of the recording disagree
Seen in · Spread
Radius
Seen in 5 front views
Spread diameter 1.1 mm
Front, off the face
Seen in 2 readings
Centre above the floor
Seen in 5 front views
Spread about 4 cm either way: two parts of the recording disagree
Lever
Length94.6 mm4 views from above1.1 mm
Thickness5.8 mm6 views from above0.6 mm
Height of the bar34.2 mm267 image rows in 6 frames3.0 mm; a stereo cross‑check reads 26.5 mm
Stand-off at the grip41.9 mm8 readingsouter face 1.1 mm
Rest angle15.2° below level4 framesabout 1°
Directiontoward the hingeevery frame agrees
Seen in · Spread
Length
Seen in 4 views from above
Spread 1.1 mm
Thickness
Seen in 6 views from above
Spread 0.6 mm
Height of the bar
Seen in 267 image rows in 6 frames
Spread 3.0 mm; a stereo cross‑check reads 26.5 mm
Stand-off at the grip
Seen in 8 readings
Spread outer face 1.1 mm
Rest angle
Seen in 4 frames
Spread about 1°
Direction
Spread every frame agrees
Hinge lineA fitted line, not the hinge pin. Some of its planes were placed with a head path that was wrong in places.
Fitted tothe door's plane178 framesthe planes lean off it by a median 0.37° and pass a median 6 mm from it
Seen in · Spread
Fitted to
Seen in 178 frames
Spread the planes lean off it by a median 0.37° and pass a median 6 mm from it

Spread: the standard deviation between frames, unless the cell says otherwise.

Also counted among the 25: measured values for the worker's swings, the floor, the room, a picture of the door's face, and the place of a lower fitting (a key cylinder or thumb-turn) that is not built.

Assumed13

PartValueWhy
Brass plate, place in heightFrom one frame of the lock clipIts lower edge was never seen in the door clips.
Latch bolt and keeper (the tongue and its slot)Released at 30° of lever travel; 12 mm of bolt travelThe bolt was never visible. A strike plate being fitted on the jamb shows a latch exists.
Far-side leverA mirror copy of the measured leverIts top edge was traced between the worker's fingers; the rest of its shape was never seen.
Not in the picture
Mass of the leaf33 kgNothing was weighed: an assumed 600 kg per cubic metre times the measured box.
Hinge friction and damping0.5 N m of friction; 1.0 N m s per radian of dampingThe door never moved without a hand on it.
Lever travel and spring45° of travel; a chosen springThe lever was never seen pressed.
Clearance under the leaf4 mm, a defaultNot measured: the leaf's bottom edge was never clearly seen.

Also counted among the 13: assumed values for the closed position, the floor, the room and the picture of the door's face, and the size of that lower fitting, which was never measured.

Where the values disagree10

Each row says where two readings disagree and which one the model keeps.

  1. Where the hardware sits. An estimate from fingertip positions near the door puts the hardware about a centimetre away from the spindle measured directly, along the leaf and in height. Kept: the direct measurement.
  2. Leaf width. The free edge marked in the pictures gives a leaf about 0.78 m wide. One frame's reading of the whole face is wider, and the opening between the jambs measures 0.77 m. Kept: the edge as marked.
  3. Leaf height. A typical height stood in until the top was measured. Kept: the measured top, with its doubt shown as the glass band.
  4. Opening angle in the swings. A slower way of reading the angle went far wrong while the door moved: its head path followed the door. Kept: the faster readings, inside the two swings.
  5. Closed position. The door was never seen closed. The model's file and the room around the door place the closed position differently. Kept: the room's reading, in the model as built.
  6. Hinge lean. The fitted hinge leans a little from vertical, and the assumed friction lets the door drift with that lean: one of the two needs a second look. Kept: the fitted axis.
  7. Latch. Its values were never seen. Kept: a latch, with assumed values.
  8. Lever mid-line. It runs a little below the spindle. Kept: the measured line in three dimensions.
  9. Rose against plate. The rose's base reads closer to the door's face than the plate is thick. Kept: both as measured; the model starts the rose on the plate.
  10. Leaf top against lever height. Two parts of the recording disagree on the lever's height above the floor, so the model's top sits several centimetres closer to its lever than one of them shows. Kept: both values, each with its doubt.
Every number 3
  • 25 entries measured, 13 assumed, 10 where two readings disagree.Source: the door model's own parameter file, counted when the page was made. One entry can cover several values.
  • The glass band above the leaf's measured top is as tall as the doubt in the leaf's height: several centimetres.Source: the same file.
  • Every length in the table scales with the gap between the lenses, known to about 4%. Its spread is between frames.Source: our calibration of the rig, fitted to its own footage and motion sensor.
  • Simulated
  • Assumed

Same lever. Two ways in.

Rewarded for the door alone: the back of the hand. Following the worker: fingers.

Simulation1×

Simulation. Each hand was trained by trial and error on a floating wrist, reading the simulator's exact state in fixed physics; one training run per recipe.

  • Door alone
  • Door and the worker's motion
A back-handed answer. A recipe is what a hand is rewarded for and how it is trained. The right-hand recipe is also rewarded for placing its fingertips where the worker's were, early in the swing, and its door version and training schedule differ too. Lime marks simulated contact: the part pressing the lever, in each frame with a recorded force. One start for both; the camera rides on the door, so the door holds still as it opens.

Data

opening angle of the simulated door, frame by frame with the film above

  • Door alone
  • Door and the worker's motion
  • Simulated

Data

Share of the force, by hand part

over 40 episodes re-run to count the forces

Door alone

Press on the lever

  • 99.8%Back of hand, at the wrist
  • 0.2%Rest of the palm

Pull on the door

  • 99.8%Back of hand, at the wrist
  • 0.2%Rest of the palm

Door and the worker's motion

Press on the lever

  • <0.1%Thumb
  • 83%Finger links
  • 17%Fingertips

Pull on the door

  • 18%Finger links
  • 82%Fingertips
Every number 7
  • Door alone: the back of the hand at the wrist carries 99.8% of the press on the lever, over 40 re‑run episodes.Source: the contact forces on the lever, summed by hand part over the whole episode, in 40 episodes re‑run on the simulator's CPU version from starts set by the task, separate from the 200 tests.
  • Door alone: the back of the hand at the wrist carries 99.8% of the pull on the door, over 40 re‑run episodes.Source: the opening torque on the door while it moves, summed by hand part, in the same re‑run episodes. Pushes back on the leaf are left out.
  • Door and the worker's motion: over 40 re‑run episodes, the fingertips carry 82% of the pull.Source: the opening torque on the door while it moves, summed by hand part, in 40 episodes re‑run on the simulator's CPU version from starts set by the task. Pushes back on the leaf are left out.
  • Door and the worker's motion: the finger links carry 83% of the press on the lever and the fingertips 17%, over 40 re‑run episodes.Source: the contact forces on the lever in the same re‑run episodes, summed by hand part over the whole episode.
  • The tint follows every contact above 0.05 N in any physics step inside the frame's own time.Source: the film's own contact record; the threshold was chosen for the analysis.
  • The film runs at true speed, 30 frames a second. Each frame shows the simulator state nearest its time; states are stored every 5 ms. Both panels are episodes run for the film from one shared start, on the simulator's CPU version, separate from the 200 test episodes.Source: the recorded episodes behind the film.
  • The hand is the ORCA v2 hand model on a floating wrist.Source: the simulation's own set‑up.
  • Simulated
  • Measured

Every test, counted.

Recipe by recipe, failures in the picture: a success rate you can check.

Data

The simulated door's angle in every test episode (one try from a fresh start) of the recipe with the worker's motion, drawn over the worker's measured door angle. Breaks in the worker's curve are left open.

  • Simulated door, one line an episode
  • The worker's door, measured
  • Failed episode
Each episode starts where the worker began. This recipe is rewarded for following the worker's door curve, and scored on it.

Data

  • Door alone, simpler doortrained on an upright plain box

    200 of 200

    Rewarded for the open door.

    Starts with the door latched, the hand off the lever.

  • Door and the worker's motion

    198 of 200

    Rewarded for the door, the worker's door curve, and the worker's estimated hand where it was seen, early in the swing.

    Starts with the hand on the lever, as the worker's was.

  • Door and the worker's door curve

    200 of 200

    Rewarded for the door and the worker's door curve.Its hand meets the lever with the finger links near the knuckles and the back of the hand.

    Starts with the hand on the lever, as the worker's was.

  • Door alone, this doortrained on the door shown

    0 of 200

    Rewarded for the open door, the same reward as the other Door alone. One training run of about 15 minutes.

    Starts with the door latched, the hand off the lever.

Baseline: a hand-written controller that reads the lever's exact position opens and holds the door in 200 of 200, from starts set by the task. Door alone, simpler door, began half its training episodes from that controller's states.

Every number 12
  • Door alone: the door opened and was held for a second in 200 of 200 test episodes on the model shown here, from starts set by the task.Source: our recorded test episodes of that hand, every one counted.
  • Door and the worker's motion: 198 of 200 test episodes from where the worker began; from starts set by the task, 200 of 200.Source: our recorded test episodes of that hand, every one counted.
  • Door and the worker's motion, from where the worker began: at the end of the 8 s, the door was still held open in 196 of 200. Of the others, 2 were never held and 2 were held and then let go.Source: our recorded test episodes of that hand, every one counted: the same held test, read at the end of each episode.
  • Door and the worker's motion, from where the worker began: the door follows the worker's curve with a typical error of 2.6° after the closest‑fitting time shift, of up to 1.5 s; with no shift, 4.4°.Source: the root‑mean‑square difference between the simulated and the measured door angle over the opening swing, as a median across test episodes.
  • Door and the worker's door curve, without the terms for the worker's hand: 200 of 200, from where the worker began.Source: our recorded test episodes of that hand.
  • Without the terms for the worker's hand, the hand meets the lever with the finger links near the knuckles and the back of the hand.Source: our contact analysis of that hand; its shares sit outside this page's bars.
  • Door alone, trained on the door shown in one run of about 15 minutes: 0 of 200, from starts set by the task.Source: our recorded test episodes of that hand, and its training record.
  • One training run for each recipe.Source: our training records.
  • A test episode counts as a success when the door reaches the worker's opening and is held there for a second, with the hand on the door's hardware.Source: the task's success rule.
  • Starts set by the task: the door shut and latched, the hand placed at random a short way in front of the lever. Starts where the worker began: the worker's estimated hand pose at the start of the swing, on the lever, with small random offsets.Source: the simulation's own set‑up.
  • The door shown sets the leaf along the measured hinge line. The simpler door stood the leaf upright as a plain box.Source: the door model's records.
  • A hand‑written controller with no training: 200 of 200 on the model shown here, from starts set by the task.Source: our recorded test episodes of that controller. It reads the door's and the lever's state from the simulator.
  • Measured
  • Estimated

Two sensors, one rhythm.

Same rig, another job: stereo sees each swing, the head IMU feels each blow.

Data

Stereo depth in every frame: pale near, deep green far, flat dark unchecked. Calibration refitted on this recording, its two pictures aligned about four times less closely than the lock clip's; the repeat figure above is the lock clip's.

Footage

The left camera's own picture at one blow, film frame 202.

The worker's view at a blow: a bare left forearm along a white wall, and the right hand holding a small hammer against a thin moulding strip.
  • Head IMUMeasuredacceleration above its 50 ms average, m/s²
  • Blow: the IMU's impulse
  • impulse rule, 0.4 m/s²
  • Right wrist, distance from the left lensMeasuredEstimatedwhere the wrist is
  • No reading
Each spike is a blow felt at the head; just before it, stereo sees the wrist draw back and swing in. One clock pairs every blow with its motion.
Every number 8
  • 57% of the picture has a checked depth in a typical frame (median over 240 frames); the lock clip, 68%.Source: The disparity of the 240 stereo pairs: a pixel counts when the match lands in the right picture and both matching directions land on the same pixel, give or take 1 px.
  • 15 impulses of the motion sensor in 8 s, 0.37 s apart on average.Source: The head‑worn motion sensor at 400 samples a second: acceleration more than 0.4 m/s² above its 50 ms average, peaks at least 0.12 s apart.
  • 15 of 15 impulses show the hammer on the strip.Source: Each impulse frame compared by eye with the frames five before and after it.
  • The right wrist has a checked depth in 181 of the 240 frames.Source: A hand detector on the left picture; a wrist counts where its pixel has a checked depth.
  • Matched patches of the two pictures differ in height by 0.95 px (median), against 0.25 px on the lock clip.Source: Cross‑correlation of small patches of both pictures, on frames of each clip.
  • The lenses are 62.9 mm apart, with a scale uncertain by about 4%, both from the recording above.Source: The rig's own calibration, carried over from the recording above.
  • Where the left forearm has a checked depth, it lies 11 to 13 cm in front of the lens (depth along the line of sight).Source: The checked pixels the forearm covers, 10th to 90th percentile.
  • In 5 frames the detector named the right hand the left; the area the arm covers shows it is the right, and it is drawn as the right.Source: the hand detector's labels, checked against the area the arm covers.

Build with us.

Your job next, in stereo depth, collected to your brief.

Get a quoteGet a free sample →

We can't wait to see what you build.

Credits for these figures →