feat: viewer detection with an axis gizmo, a desktop marker page, and the watch app #1

Merged
dawid merged 11 commits from viewer-detection into main 2026-11-16 04:56:33 +00:00
Owner

Milestone 2, apart from compiling the watch app, which waits for the Connect IQ SDK on the runner (HomeLab #122).

Viewer (viewer/public)

  • index.html:
    • opens the rear camera and finds the marker with js-aruco2, restricted to DICT_4X4_50 with one corrected bit;
    • estimates the pose with coplanar POSIT, assuming a 65° field of view (?fov=, ?size=);
    • keeps the solution closest to the current pose against the planar flip, and eases toward each frame;
    • draws an axis gizmo and the marker's outline with three.js.
    • The decision record is docs/decisions/003-pose.md.
  • markers.html: the watch's markers on screen, for testing without a watch (arrow keys, ?id=).
  • nginx gzips JavaScript: three.js's core goes from 1.5 MB to 378 KB.

Watch app (watch/, not compiled yet; see docs/watch.md)

  • Marker: drawn with rectangles, as large as the round display allows with a cell of quiet zone at its corners: 39 px cells, about 18.6 mm on the D2 Mach 1's 1.3″ screen.
  • Screen: black everywhere else, so the AMOLED lights as little as possible.
  • Controls: up and down switch the ID, enter or a tap toggles a 5-second auto-cycle.
  • Display: kept on with Attention.backlight until the burn-in protection throws BacklightOnTooLongException (API reference). Any input starts a new session.
  • No permissions, no glance (docs/watch.md explains why).

Tests (new CI job e2e, in mcr.microsoft.com/playwright:v1.64.1-noble, under the production CSP read from nginx.conf)

  1. All 50 markers of the marker page are detected with their ID, straight on and at three angles. Breaking the marker drawing on purpose makes this test fail.
  2. A frontal marker's distance is within 3 % of the camera model's prediction, and it faces the camera.
  3. The viewer, fed a marker through Chromium's fake camera, reports its ID with no console error or CSP violation.
  4. The watch's generated table equals the detector's codes, and its round 416 px layout (as MarkerView draws it, mirrored in the test) is detected for every ID.

Renovate updates the npm playwright package and the image together. The smoke test also checks three.js and the marker page.

Not verified

  • The watch code has never been compiled.
  • No real phone has run the viewer yet: speed on mid-range Android and the flip handling at angles are untested.
  • The 33 mm display diameter comes from spec sites, not Garmin.

Update: the watch app compiles and runs in the simulator (milestone 2 done)

  • CI job watch (runs-on: ubuntu-latest, the runner container ci with the SDK read-only at /opt/connectiq, from HomeLab #122–#124): tools/build-watch.sh compiles for every device in the manifest. It uses the strict type check (--typecheck 3), shows warnings, and signs with a throwaway key made and deleted in each run. It's green.
    • monkeyc creates a default.jungle next to the real path of monkeybrains.jar. I traced that with strace on a read-only copy of the SDK on studio. So on the read-only mount, the script builds writable directories of links and copies the jar (23 MB).
    • Java takes its home from the account, not from $HOME, so the script redirects it with JAVA_TOOL_OPTIONS=-Duser.home=…, on Linux only.
  • Launcher icon: the compiler requires one. It's marker 0 at 60×60 (the D2 Mach 1's size, from its compiler.json), generated from the marker table by tools/generate-icon.mjs and checked in the checks job.
  • Simulator: tools/simulate.sh runs the app in an Ubuntu 22.04 container, because the simulator needs WebKitGTK 4.0. It uses Xvfb, mounts the SDK read-only and saves a screenshot. Marker 0 shows on the D2 Mach 1, and js-aruco2 detects ID 0 in the screenshot, with a side of 233 px. The app used 7.6 of 763.6 kB. The screenshot in docs/watch.md masks out Garmin's picture of the watch, which is part of the SDK.
    • The simulator needs the devices' fonts (HomeLab #125, --include-fonts). Without them, text fails with "Invalid Font Specified".
  • docs/watch.md now plans for as many devices as possible: icon sizes, display types, API levels, one .iq build for all.

Still not verified: anything on the real watch or a phone. That's milestone 4, in #3.

Milestone 2, apart from compiling the watch app, which waits for the Connect IQ SDK on the runner (HomeLab #122). **Viewer** (`viewer/public`) - `index.html`: - opens the rear camera and finds the marker with js-aruco2, restricted to DICT_4X4_50 with one corrected bit; - estimates the pose with coplanar POSIT, assuming a 65° field of view (`?fov=`, `?size=`); - keeps the solution closest to the current pose against the planar flip, and eases toward each frame; - draws an axis gizmo and the marker's outline with three.js. - The decision record is `docs/decisions/003-pose.md`. - `markers.html`: the watch's markers on screen, for testing without a watch (arrow keys, `?id=`). - nginx gzips JavaScript: three.js's core goes from 1.5 MB to 378 KB. **Watch app** (`watch/`, not compiled yet; see `docs/watch.md`) - **Marker:** drawn with rectangles, as large as the round display allows with a cell of quiet zone at its corners: 39 px cells, about 18.6 mm on the D2 Mach 1's 1.3″ screen. - **Screen:** black everywhere else, so the AMOLED lights as little as possible. - **Controls:** up and down switch the ID, enter or a tap toggles a 5-second auto-cycle. - **Display:** kept on with `Attention.backlight` until the burn-in protection throws `BacklightOnTooLongException` ([API reference](https://developer.garmin.com/connect-iq/api-docs/Toybox/Attention.html)). Any input starts a new session. - No permissions, no glance (`docs/watch.md` explains why). **Tests** (new CI job `e2e`, in `mcr.microsoft.com/playwright:v1.64.1-noble`, under the production CSP read from `nginx.conf`) 1. All 50 markers of the marker page are detected with their ID, straight on and at three angles. Breaking the marker drawing on purpose makes this test fail. 2. A frontal marker's distance is within 3 % of the camera model's prediction, and it faces the camera. 3. The viewer, fed a marker through Chromium's fake camera, reports its ID with no console error or CSP violation. 4. The watch's generated table equals the detector's codes, and its round 416 px layout (as `MarkerView` draws it, mirrored in the test) is detected for every ID. Renovate updates the npm `playwright` package and the image together. The smoke test also checks three.js and the marker page. **Not verified** - The watch code has never been compiled. - No real phone has run the viewer yet: speed on mid-range Android and the flip handling at angles are untested. - The 33 mm display diameter comes from spec sites, not Garmin. ## Update: the watch app compiles and runs in the simulator (milestone 2 done) - **CI job `watch`** (`runs-on: ubuntu-latest`, the runner container `ci` with the SDK read-only at `/opt/connectiq`, from HomeLab #122–#124): `tools/build-watch.sh` compiles for every device in the manifest. It uses the **strict type check** (`--typecheck 3`), shows warnings, and signs with a throwaway key made and deleted in each run. It's green. - monkeyc creates a `default.jungle` next to the *real* path of `monkeybrains.jar`. I traced that with `strace` on a read-only copy of the SDK on studio. So on the read-only mount, the script builds writable directories of links and copies the jar (23 MB). - Java takes its home from the account, not from `$HOME`, so the script redirects it with `JAVA_TOOL_OPTIONS=-Duser.home=…`, on Linux only. - **Launcher icon:** the compiler requires one. It's marker 0 at 60×60 (the D2 Mach 1's size, from its `compiler.json`), generated from the marker table by `tools/generate-icon.mjs` and checked in the `checks` job. - **Simulator:** `tools/simulate.sh` runs the app in an Ubuntu 22.04 container, because the simulator needs WebKitGTK 4.0. It uses Xvfb, mounts the SDK read-only and saves a screenshot. Marker 0 shows on the D2 Mach 1, and js-aruco2 detects ID 0 in the screenshot, with a side of 233 px. The app used 7.6 of 763.6 kB. The screenshot in `docs/watch.md` masks out Garmin's picture of the watch, which is part of the SDK. - The simulator needs the devices' fonts (HomeLab #125, `--include-fonts`). Without them, text fails with "Invalid Font Specified". - `docs/watch.md` now plans for **as many devices as possible**: icon sizes, display types, API levels, one `.iq` build for all. Still not verified: anything on the real watch or a phone. That's milestone 4, in #3.
The viewer opens the rear camera, finds the marker with js-aruco2 restricted to
DICT_4X4_50 with one corrected bit, estimates its pose with coplanar POSIT under
an assumed 65 degree field of view, keeps the solution closest to the current
one against the planar flip, eases toward each frame, and draws an axis gizmo
with three.js (docs/decisions/003-pose.md). markers.html shows the watch's
markers for testing without a watch. End-to-end tests in Chromium under the
production CSP: all 50 markers detected also at an angle, the pose's distance
within 3 %, and the viewer fed a marker through a fake camera.
feat(watch): the marker on the D2 Mach 1, its controls and the display held on
Some checks failed
CI / checks (pull_request) Has been cancelled
CI / e2e (pull_request) Has been cancelled
CI / image (pull_request) Has been cancelled
3218384270
A Connect IQ device app, not compiled yet (the SDK reaches CI with HomeLab's
connectiq_sdk). MarkerView draws the marker with rectangles, as large as the
round display allows with a cell of quiet zone at its corners (39-pixel cells,
about 18.6 mm on the 1.3-inch D2 Mach 1), black elsewhere for the AMOLED
display. Up and down switch the ID, enter starts an automatic cycle. The
display is held on with Attention.backlight until the system's burn-in
protection throws BacklightOnTooLongException
(https://developer.garmin.com/connect-iq/api-docs/Toybox/Attention.html);
input holds it again. No permissions. A browser test draws the same layout from
the generated table and detects every marker. docs/watch.md.
The tables' pipes line up, as in the homelab's documentation; markdownlint's
MD060 (aligned) is on again and CI runs markdownlint. It only reports
misaligned tables, so tools/align-tables.mjs aligns them.
docs: the planned interaction with the models
All checks were successful
CI / checks (pull_request) Successful in 23s
CI / e2e (pull_request) Successful in 56s
CI / image (pull_request) Successful in 11s
8264a40b20
Moving the watch and touch or clicks in the viewer are chosen; hand gestures in
front of the camera only if they give the hand wearing the watch meaningful
actions. After the device tests (milestone 4).
ci(watch): compile the watch app with the Connect IQ SDK on the runner's host
Some checks failed
CI / checks (pull_request) Successful in 12s
CI / e2e (pull_request) Successful in 56s
CI / image (pull_request) Successful in 12s
CI / watch (pull_request) Failing after 17s
a34a0bf1bf
tools/build-watch.sh builds every device of the manifest with the SDK from
/opt/connectiq (read-only in the runner container ci), signed with a key made
for each run and deleted afterwards.
fix(ci): give monkeyc writable directories over the read-only SDK
Some checks failed
CI / checks (pull_request) Successful in 16s
CI / e2e (pull_request) Successful in 55s
CI / image (pull_request) Successful in 11s
CI / watch (pull_request) Failing after 10s
d3dfd7cce9
monkeyc 9.3.0 writes a generated default.jungle into the Connect IQ tree and
failed on CI's read-only mount: "Unable to generate default.jungle:
Read-only file system".
fix(ci): a writable copy of the SDK's default.jungle, which monkeyc rewrites
Some checks failed
CI / checks (pull_request) Successful in 13s
CI / e2e (pull_request) Successful in 54s
CI / image (pull_request) Successful in 11s
CI / watch (pull_request) Failing after 10s
905ae10d70
fix(ci): copies of the SDK bin's small files, with a diagnosis if monkeyc still fails
Some checks failed
CI / checks (pull_request) Successful in 12s
CI / e2e (pull_request) Successful in 54s
CI / image (pull_request) Successful in 13s
CI / watch (pull_request) Failing after 13s
ee25745eef
fix(ci): writable copies of monkeyc's jar and the manifest's devices, monkeyc's debug log on failure
Some checks failed
CI / checks (pull_request) Has been cancelled
CI / e2e (pull_request) Has been cancelled
CI / image (pull_request) Has been cancelled
CI / watch (pull_request) Has been cancelled
fa2962f90a
fix(watch): compiles with SDK 9.3.0: a launcher icon, monkeyc's jar a writable copy
All checks were successful
CI / checks (pull_request) Successful in 12s
CI / e2e (pull_request) Successful in 59s
CI / image (pull_request) Successful in 12s
CI / watch (pull_request) Successful in 14s
4b618c195d
Traced on a read-only copy of the SDK: Java creates default.jungle next to
the real path of monkeybrains.jar, so CI's shadow of the SDK copies the jar.
The compiler requires a launcher icon: marker 0, 60x60 for the D2 Mach 1,
generated from the marker table and checked in CI. The strict type check
passes and is now required. docs/watch.md plans for many devices.
feat(watch): the app in the Connect IQ simulator, with a screenshot
All checks were successful
CI / checks (pull_request) Successful in 13s
CI / e2e (pull_request) Successful in 59s
CI / image (pull_request) Successful in 12s
CI / watch (pull_request) Successful in 16s
b24cb7c7fb
tools/simulate.sh runs a built app in the simulator in an Ubuntu 22.04
container (WebKitGTK 4.0) with Xvfb, the SDK mounted read-only, and fails on
an app error. Marker 0 on the D2 Mach 1, SDK 9.3.0: js-aruco2 detects ID 0 in
the screenshot. The picture of the watch around the display, from the SDK, is
masked out. Milestone 2 done; the viewer works with a laptop's webcam too.
dawid merged commit 78e1aa0381 into main 2026-11-16 04:56:33 +00:00
dawid deleted branch viewer-detection 2026-11-16 04:56:33 +00:00
Sign in to join this conversation.
No description provided.