say hello

MotionKey Scribe

Gesture learning & robot simulation · 2026 · Local research prototype

A gesture-learning prototype that connects motion classification and abstention to bounded drawing commands in a two-cable robot simulator.

What it is

The idea is to teach a private motion vocabulary and turn an accepted gesture into a drawing. The interface and simulator let the recording, recognition, and command stages be examined separately.

Uncertainty has a visible outcome: no command. A recognized label still needs a fresh prediction, a valid arm lease, and a bounded glyph before the simulator accepts it.

The loop it has to make believable

  1. 01

    Record a gesture vocabulary, compare recognition approaches, and bind an accepted label to a fixed drawing glyph.

  2. 02

    Pass a stored prediction through an armed gateway and inspect the resulting path and simulated cable motion.

How it’s built

The ML service records immutable data and compares engineered features, DTW, and a compact CNN. Evaluation separates recording sessions; the external wrist-IMU probe also holds people out and evaluates continuous background motion.

The gateway consumes persisted predictions rather than trusting client-supplied confidence. Pure geometry derives cable paths, while tests exercise replay protection, lease and freshness failures, canvas margins, step-rate bounds, and concurrent commands.

Architecture

Data
Immutable Parquet recordings, session-separated datasets, and prediction-to-model lineage
Inference
Engineered features, DTW, and compact CNN comparisons with a threshold that permits no action
Gateway
Stored prediction IDs, short arm leases, freshness checks, replay defense, cooldown, and fixed glyph bindings
Geometry
Two-cable path planning with canvas margins, path-length limits, and derived motor-step-rate bounds
Hardware starter
ESP32 firmware, calibration tools, and watchdog logic; automatic serial integration remains unfinished

What has to hold up

  • Reject stale, replayed, unarmed, out-of-margin, or over-rate commands instead of allowing model output to specify arbitrary motor steps.
  • Evaluate continuous held-out motion and report both missed gestures and false triggers, including negative results hidden by easy synthetic data.

What the evidence changed

The documented external evaluation found an unsatisfactory tradeoff: the feature model detected more gestures but triggered too often, while the CNN largely abstained and missed most events. Neither result establishes useful recognition on the intended phone domain.

The documented smoke run connects a model prediction to deterministic simulated geometry. Firmware compilation establishes toolchain compatibility. Neither proves mechanical accuracy, physical stop behavior, serial transport, or a working phone-to-robot loop; no physical drawing result is claimed.

Proof

Local source and recorded evaluation support the simulation and research case study. A public repository, real phone validation, and physical hardware evidence are not supplied here.

VerifyRepository, checks, stack, status
  1. 01

    Read the local README and docs/EVALUATION.md, starting with the current conclusion and continuous replay results.

  2. 02

    Follow the synthetic demo and smoke_api.py path to inspect prediction-to-glyph integration.

  3. 03

    Inspect test_scribe.py for freshness, replay, geometry, stop, and concurrency behavior; use make verify for the documented software gate.

  4. 04

    Keep external wrist-IMU results, synthetic integration checks, firmware compilation, and physical validation as separate evidence.

Role
Recording contracts, model comparison, evaluation, command gateway, and simulation
Scope
Motion-learning app, ML service, cable-plotter simulation, and firmware starter
Proof
docs/EVALUATION.md; services/ml/tests/test_scribe.py; services/ml/scripts/smoke_api.py; hardware firmware and calibration tooling
Stack
Python, FastAPI, React Native, Expo, Parquet, SQLite, ESP32, PlatformIO
Models
Features, DTW, compact CNN
Decision
Thresholded abstention
Actuation
Bounded simulated paths
Hardware
Physical loop unverified