fix(viewer): hand tracking loads on the web, models free their GPU memory, and a stalled or switched camera is handled #61

Merged
dawid merged 1 commit from viewer-robustness into main 2026-11-19 10:11:15 +00:00
Owner

Fixes from the review of the viewer, the Android app and the release workflow. Independent of #60.

Finding Effect Fix
vision_bundle.mjs served as application/octet-stream with nosniff (production too) Hand tracking never starts on the web .mjs as text/javascript; the smoke test checks it
Models never disposed GPU memory grows with every marker change, worst under auto cycle geometries, materials and textures disposed
Camera frames stop (permission revoked, another app, iOS background) The model frozen as if tracking fades out after 1 s; an ended track says so; the video plays again on return
Switching cameras opens the new one first NotReadableError on phones with one camera open at a time; the error flashed away old one stopped first, the one before reopened on failure, one switch at a time
FOV applied to the image's width Distances 44 % short with the phone upright (gesture thresholds and the guide too) FOV across the longer side (docs/dev/decisions/003-pose.md)
dblclick anywhere on the stage A double press of a button resets size and turn ignored on the stage's own controls; the wheel only over a model
One exception in the loop The viewer stops, silently try/finally per frame; hands off when they fail; URL parameters checked
No Cache-Control An old cached index.html with a new app.js breaks no-cache (revalidated by ETag)
allowBackup="true" Auto Backup clones the paired session onto another phone off
Full screen in the Android app Undone at once by Capacitor's WebView the stage fills the window there

Smaller: the pairing code isn't drawn as models once read, the hand tracker loads once when toggled during loading, the wake lock of the guided test is taken again, the report download's URL kept a minute, a hidden page's long frame no longer spins the model, the camera advice names Android's settings in the app, and the release tag must be vX.Y.Z with Y, Z below 100.

Tests: viewer 79 of 79 (a new test for the focal length upright), lint, the image's smoke test. The camera and full-screen paths need a phone: switch cameras, revoke the camera permission while running, and try full screen in the app.

Fixes from the review of the viewer, the Android app and the release workflow. Independent of #60. | Finding | Effect | Fix | |---|---|---| | `vision_bundle.mjs` served as `application/octet-stream` with `nosniff` (production too) | Hand tracking never starts on the web | `.mjs` as `text/javascript`; the smoke test checks it | | Models never disposed | GPU memory grows with every marker change, worst under auto cycle | geometries, materials and textures disposed | | Camera frames stop (permission revoked, another app, iOS background) | The model frozen as if tracking | fades out after 1 s; an ended track says so; the video plays again on return | | Switching cameras opens the new one first | `NotReadableError` on phones with one camera open at a time; the error flashed away | old one stopped first, the one before reopened on failure, one switch at a time | | FOV applied to the image's width | Distances 44 % short with the phone upright (gesture thresholds and the guide too) | FOV across the longer side (`docs/dev/decisions/003-pose.md`) | | `dblclick` anywhere on the stage | A double press of a button resets size and turn | ignored on the stage's own controls; the wheel only over a model | | One exception in the loop | The viewer stops, silently | `try`/`finally` per frame; hands off when they fail; URL parameters checked | | No `Cache-Control` | An old cached `index.html` with a new `app.js` breaks | `no-cache` (revalidated by ETag) | | `allowBackup="true"` | Auto Backup clones the paired session onto another phone | off | | Full screen in the Android app | Undone at once by Capacitor's WebView | the stage fills the window there | Smaller: the pairing code isn't drawn as models once read, the hand tracker loads once when toggled during loading, the wake lock of the guided test is taken again, the report download's URL kept a minute, a hidden page's long frame no longer spins the model, the camera advice names Android's settings in the app, and the release tag must be `vX.Y.Z` with Y, Z below 100. **Tests:** viewer 79 of 79 (a new test for the focal length upright), lint, the image's smoke test. The camera and full-screen paths need a phone: switch cameras, revoke the camera permission while running, and try full screen in the app.
fix(viewer): hand tracking loads on the web, models free their GPU memory, and a stalled or switched camera is handled
All checks were successful
CI / checks (pull_request) Successful in 35s
CI / relay (pull_request) Successful in 48s
CI / image (pull_request) Successful in 56s
CI / e2e (pull_request) Successful in 3m59s
CI / watch (pull_request) Successful in 2m51s
CI / android (pull_request) Successful in 2m8s
a4e88cdd25
From the review of the viewer:
- nginx serves .mjs as JavaScript: MediaPipe's bundle came as
  application/octet-stream with nosniff, so hand tracking never started on
  the web (the app was fine); the smoke test checks the type. Files are
  revalidated (Cache-Control: no-cache), so a cached index.html never
  meets the next release's app.js.
- A model's geometries, materials and textures are disposed when another
  replaces it: memory grew with every marker change.
- A camera whose frames stop fades the model out instead of freezing it;
  a track that ends says so, with a way to start again; the video plays
  again after the page was in the background.
- Switching cameras stops the open one first (many phones open one at a
  time), opens the one before again on failure, says the error on the
  stage, and doesn't run twice at once.
- The field of view spans the sensor's longer side: held upright, every
  distance read about 44 % short.
- A double press of a stage button no longer resets size and turn; the
  wheel zooms only over a model and not over the stage's panels.
- The render loop survives an exception in one frame; a failing hand
  tracker turns itself off; URL parameters are checked.
- The pairing code's markers aren't shown as models once read; hand
  tracking toggled while loading loads once; a hidden page's long frame
  no longer spins the model.
- The guided test takes its wake lock again after the page was hidden,
  and keeps a downloaded report's URL for a minute.

Android: no Auto Backup (it would clone the paired session onto another
phone), the stage alone for full screen, and the camera's advice names
the app's permission. Release: the tag must be vX.Y.Z with Y and Z below
100, which the version code needs.
dawid merged commit e598029974 into main 2026-11-19 10:11:15 +00:00
dawid deleted branch viewer-robustness 2026-11-19 10:11:15 +00:00
Sign in to join this conversation.
No description provided.