feat(viewer): the hand in front hides the model, and the model stays while a hand covers the marker #55

Merged
dawid merged 1 commit from hand-occlusion into main 2026-11-18 13:57:41 +00:00
Owner

Occlusion. While hand tracking is on, your hand hides the model where it's in front of it.

  • Hand shape: each hand becomes a rough shape of spheres and tubes, drawn into the depth
    buffer only. It's invisible, but the model's pixels behind it aren't drawn, so the real hand
    shows through.
  • Hand position: MediaPipe's world landmarks (meters about the hand's center) are fitted to
    its image points by linear least squares. That gives the hand's offset from the camera,
    taking its size and turn into account.
  • Margin: the hand is put 10 mm farther than measured, so a slightly near guess doesn't cut
    into a model the hand is only next to.
  • Debug and comparison: the debug view draws the hand shape in color; ?occlusion=0 turns
    occlusion off.

Keeping the model. A hand reaching for the model usually covers the marker, and the model
used to go after 0.3 s.

  • Under a hand: while hand tracking sees a hand over the marker's last place, the model
    stays where it was, up to 4 s.
  • Otherwise: the model fades out over 1 s instead of disappearing at once.
  • Found again: the pose glides to the new place instead of jumping.

WebXR depth sensing was considered and left out: it's Chrome-only on ARCore phones, absent from
the Android app's WebView and iPhones, takes over the camera, and is tuned for half a meter and
more. The reasons are recorded in docs/dev/interaction.md.

Tests: new test/occlusion.test.mjs (hand placement within 0.5 mm, hand-over-marker check,
hold and glide). Viewer tests pass (77 of 77). Not tried with a real hand in front of a camera
yet; the device test should check the margin and the hold.

**Occlusion.** While hand tracking is on, your hand hides the model where it's in front of it. - **Hand shape:** each hand becomes a rough shape of spheres and tubes, drawn into the depth buffer only. It's invisible, but the model's pixels behind it aren't drawn, so the real hand shows through. - **Hand position:** MediaPipe's world landmarks (meters about the hand's center) are fitted to its image points by linear least squares. That gives the hand's offset from the camera, taking its size and turn into account. - **Margin:** the hand is put 10 mm farther than measured, so a slightly near guess doesn't cut into a model the hand is only next to. - **Debug and comparison:** the debug view draws the hand shape in color; `?occlusion=0` turns occlusion off. **Keeping the model.** A hand reaching for the model usually covers the marker, and the model used to go after 0.3 s. - **Under a hand:** while hand tracking sees a hand over the marker's last place, the model stays where it was, up to 4 s. - **Otherwise:** the model fades out over 1 s instead of disappearing at once. - **Found again:** the pose glides to the new place instead of jumping. WebXR depth sensing was considered and left out: it's Chrome-only on ARCore phones, absent from the Android app's WebView and iPhones, takes over the camera, and is tuned for half a meter and more. The reasons are recorded in `docs/dev/interaction.md`. Tests: new `test/occlusion.test.mjs` (hand placement within 0.5 mm, hand-over-marker check, hold and glide). Viewer tests pass (77 of 77). Not tried with a real hand in front of a camera yet; the device test should check the margin and the hold.
feat(viewer): the hand in front hides the model, and the model stays while a hand covers the marker
All checks were successful
CI / checks (pull_request) Successful in 19s
CI / relay (pull_request) Successful in 51s
CI / image (pull_request) Successful in 58s
CI / e2e (pull_request) Successful in 3m54s
CI / watch (pull_request) Successful in 3m13s
CI / android (pull_request) Successful in 2m6s
b8823ab7e3
Occlusion: while hand tracking is on, each hand is placed in millimeters in the camera's frame
(its world landmarks fitted to its image points by least squares) as spheres and tubes drawn
into the depth buffer only, so the model's pixels behind it aren't drawn; 10 mm farther than
measured, so a near guess doesn't cut into the model.

Hold: with a hand over the marker's last place the pose stays up to 4 s; otherwise the model
fades out over 1 s instead of going after 0.3 s. Found again, the pose glides there.
dawid merged commit 4aee1a215b into main 2026-11-18 13:57:41 +00:00
dawid deleted branch hand-occlusion 2026-11-18 13:57:41 +00:00
Sign in to join this conversation.
No description provided.