MotionKey Scribe
A gesture-learning prototype that connects motion classification and abstention to bounded drawing commands in a two-cable robot simulator.
What it is
The idea is to teach a private motion vocabulary and turn an accepted gesture into a drawing. The interface and simulator let the recording, recognition, and command stages be examined separately.
Uncertainty has a visible outcome: no command. A recognized label still needs a fresh prediction, a valid arm lease, and a bounded glyph before the simulator accepts it.
The loop it has to make believable
- 01
Record a gesture vocabulary, compare recognition approaches, and bind an accepted label to a fixed drawing glyph.
- 02
Pass a stored prediction through an armed gateway and inspect the resulting path and simulated cable motion.
How it’s built
The ML service records immutable data and compares engineered features, DTW, and a compact CNN. Evaluation separates recording sessions; the external wrist-IMU probe also holds people out and evaluates continuous background motion.
The gateway consumes persisted predictions rather than trusting client-supplied confidence. Pure geometry derives cable paths, while tests exercise replay protection, lease and freshness failures, canvas margins, step-rate bounds, and concurrent commands.
Architecture
- Data
- Immutable Parquet recordings, session-separated datasets, and prediction-to-model lineage
- Inference
- Engineered features, DTW, and compact CNN comparisons with a threshold that permits no action
- Gateway
- Stored prediction IDs, short arm leases, freshness checks, replay defense, cooldown, and fixed glyph bindings
- Geometry
- Two-cable path planning with canvas margins, path-length limits, and derived motor-step-rate bounds
- Hardware starter
- ESP32 firmware, calibration tools, and watchdog logic; automatic serial integration remains unfinished
What has to hold up
- Reject stale, replayed, unarmed, out-of-margin, or over-rate commands instead of allowing model output to specify arbitrary motor steps.
- Evaluate continuous held-out motion and report both missed gestures and false triggers, including negative results hidden by easy synthetic data.
What the evidence changed
The documented external evaluation found an unsatisfactory tradeoff: the feature model detected more gestures but triggered too often, while the CNN largely abstained and missed most events. Neither result establishes useful recognition on the intended phone domain.
The documented smoke run connects a model prediction to deterministic simulated geometry. Firmware compilation establishes toolchain compatibility. Neither proves mechanical accuracy, physical stop behavior, serial transport, or a working phone-to-robot loop; no physical drawing result is claimed.
Proof
VerifyRepository, checks, stack, status
- 01
Read the local README and docs/EVALUATION.md, starting with the current conclusion and continuous replay results.
- 02
Follow the synthetic demo and smoke_api.py path to inspect prediction-to-glyph integration.
- 03
Inspect test_scribe.py for freshness, replay, geometry, stop, and concurrency behavior; use make verify for the documented software gate.
- 04
Keep external wrist-IMU results, synthetic integration checks, firmware compilation, and physical validation as separate evidence.
- Role
- Recording contracts, model comparison, evaluation, command gateway, and simulation
- Scope
- Motion-learning app, ML service, cable-plotter simulation, and firmware starter
- Proof
- docs/EVALUATION.md; services/ml/tests/test_scribe.py; services/ml/scripts/smoke_api.py; hardware firmware and calibration tooling
- Stack
- Python, FastAPI, React Native, Expo, Parquet, SQLite, ESP32, PlatformIO
- Models
- Features, DTW, compact CNN
- Decision
- Thresholded abstention
- Actuation
- Bounded simulated paths
- Hardware
- Physical loop unverified