// Project

Jarvis

A voice assistant on my Android phone. It handles everyday commands itself, in milliseconds and without a network, and asks Claude only when it has to. Whatever proposed an action, the same policy engine decides whether it runs.

Read it as

The repositories are private, so this page describes them instead of linking. Nothing here is a real address, key or device of mine.

Say something to it

Type a command or pick one. The trace shows which part of Jarvis understood it, and the verdict shows whether it would run, ask first, or refuse. Flip the switches to change the circumstances.

14 msto answer "what time is it", on the phone
109 msto read the SmartHome mode over Wi-Fi
307 msto read today's tasks from Horizons
2.7 sfor a cloud answer from Claude Haiku
0API keys: Claude runs on my subscription

Single observations from a review on the phone on 22 October 2026, timed inside the app; not percentiles.

§ 01

How a sentence travels

Most of what anyone says to an assistant is the same few dozen commands: pause, a timer, the time, a reminder. Jarvis answers those on the phone. It first checks a list of exact phrasings, then a small classifier for looser wording. Only what neither recognises goes to Claude.

Claude starts on its fastest model and can hand itself up to a stronger one mid-answer when the request turns out to be hard. If Claude is out of reach, Jarvis says so and the local commands keep working.

Under the hood
  • Grammar first. Deterministic patterns, specific before general, so "open https://…" never reaches the catch-all "open {app}". A match counts only if every slot parses. A test guarantees that a request to change the planner app ("add a monthly view to my planner") never matches the rule that creates a task.
  • A classifier that beat the embedding model. TF-IDF over word unigrams, bigrams and character trigrams, scored as the rank-weighted mean of each intent's three nearest examples. On 100 held-out paraphrases BGE-small acted correctly on 38 with none wrong; the n-gram model on 43 with one wrong, in 40 µs instead of loading a 34 MB model.
  • Every hypothesis counts. The recognizer returns several guesses. Each one tries the exact stage, and if two guesses mean different actions (pause or play, 5 or 15 minutes), Jarvis asks instead of picking.
  • Sound-alikes. When nothing matched, words outside the command vocabulary are re-heard through consonant-class sound keys, spelling similarity and a versioned confusion table, so "paws the music" becomes "pause the music". A sound-alike of a risky command is only offered as "Did you mean …?", and one that runs counts as uncertain.
  • Acting before you finish. A short allowlist of commands runs on a partial transcript 200 ms after it arrives. "Play" is not on it, because it starts "play some Metallica".
  • Learning from Claude. When Claude resolves a sentence to one simple local action twice, Jarvis offers to remember the phrasing as a template ("my day starts {time}"), a new classifier example rather than an alias. Never for risky actions, never from an unclear yes, never while locked.

§ 02

Who decides what runs

The language model never decides how risky something is. Every action, whether the grammar matched it or Claude asked for it, passes one policy engine that gives it a level. Reading the time runs silently. Adding a task runs and offers ten seconds of undo. Deleting a pile of tasks asks first. Anything near money or passwords needs a fingerprint.

The level moves with the circumstances. Lock the phone in the console above, or let a stranger's voice speak, and the same sentence gets stricter.

  1. L0Runs silently: the time, media, opening an app, reading tasks.
  2. L1Runs, logs it and offers undo: one task, a timer, a SmartHome mode.
  3. L2Asks first, by voice or a tap on a card showing exactly what will happen.
  4. L3Needs an unlocked phone or a fingerprint; a voice yes is not enough.
  5. L4Refuses and says why.
Under the hood
  • Effects set floors. Every tool declares what it does: read, local change, external, destructive or sensitive. External and destructive are at least L1, sensitive at least L3. A user preference can raise a tool; nothing can lower one.
  • Untrusted content raises the stakes. Once screen text, a notification or a web page has entered a conversation, every later change in it is one level stricter: a guard against prompt injection. A tool whose arguments leave the phone goes to at least L2, because an undo cannot recall data already sent.
  • Lock state is identity; voice is a hint. Locked, personal reads are refused and changes are one level stricter. With owner-only voice on, an unrecognised speaker gets only L0 actions that reveal nothing personal. Speaker checks are not security-grade, so risky actions still need the fingerprint.
  • Misheard means shown. An uncertain transcript turns an L1 action into a confirmation that displays what was heard. More than five planner changes in one request also confirm.
  • Written down before it happens. Every call above L0 is recorded on disk before it executes, so a crash mid-call still leaves proof of an uncertain outcome. Free text is redacted and the log rotates at 1 MiB.
  • Other apps can join. An app describes its tools through a signature-protected provider; approving it with a fingerprint pins its certificate. Its results are treated as untrusted, and its tools get the same floors.

§ 03

SmartHome, the flat as a tool

SmartHome is a small Windows app I wrote for my flat. It switches the TV, the monitors, an LED backlight and a lamp between a few modes, and Jarvis reaches it over home Wi-Fi, so "movie mode" works from the sofa. Pick a mode to see what changes.

It also reads temperature sensors and tells me whether opening a window will actually help. Behind that is a simple physical model of the flat: how quickly it follows the outdoor temperature, and how much the sun warms it. Move the sliders to see a day play out.

Under the hood
  • Modes are target states. A switch applies the old mode's "off" overlaid by the new mode's "on", so going from movie to night never wakes the second monitor in between. Steps on the PC itself run first; the devices then switch in parallel.
  • Nothing slow holds up the page. Each device refreshes on its own background thread with its own lifetime, and a page asks for whatever is ready. A cold dashboard took 2.6 to 3.2 seconds; it now draws in about 1.1.
  • Pure standard library. No numpy, so the whole app freezes into one .exe. The least-squares fitting is hand-written.
  • The model. dT/dt = (Tout − Tin)/τ + k·sun + g, fitted by ridge regression on a 30-minute grid, with separate τ for windows open and closed. A coefficient the data has no contrast for is reported as "not yet identifiable", not as a confident number.
  • An honesty gate. Forecasts are shown as forecasts only after a model beats the better of "same as now" and the hour-of-day average by 10% over 30 scored points. Measured: it wins at six hours (0.815 against 0.963 °C error) and still loses at one hour (0.228 against 0.227). A two-node model with hidden wall heat had the better fit and the worse forecasts, so it is not used.
  • The Jarvis side. One tool with a fixed list of intents. Jarvis speaks its own sentences and never the server's text. Its network client refuses, before connecting, any address outside the private home ranges, and pairing saves nothing until the PC answers with the right key and I confirm on the phone.

§ 04

Horizons, the planner it writes to

Horizons is my task manager: short, medium and long-term tasks, reminders that land on the minute, repeats that behave like a calendar, subtasks and a home-screen widget. It has no account and no network code. Jarvis adds and reads tasks in it, and when Jarvis proposes a whole week's plan I see the exact changes before anything is written.

Repeating dates are harder than they look. Pick a series and compare the obvious approach with what Horizons stores.

Under the hood
  • Store the rule, not the drift. The day of the month is taken from the due date when the task is saved and clamped only when a date is shown. Reading it back from each occurrence let one short February move the rent to the 28th for good.
  • Anchored on the due date. Paying September's rent late still produces October 5, not "a month from today". "Three days after I last did it" is a separate, deliberate mode.
  • One alarm at a time. A single system alarm points at the earliest pending reminder; when it fires, everything due is handled and the alarm is re-aimed. After a reboot the schedule is rebuilt by reading the table, not by remembering. Quiet hours move a reminder later, never earlier, and one more than two minutes late counts as missed.
  • Never lose years of history. The database has no destructive fallback: every version ships a real migration. A backup import validates the whole file before deleting anything, then replaces every table in one transaction.
  • The contract with Jarvis. Jarvis calls Horizons through a content provider guarded by a signature permission. Horizons owns a JSON wire contract, and Jarvis's test fails, rather than skips, if it cannot read it. A plan is validated in full before any change is written, and its dry run is the diff on Jarvis's confirmation card.
  • Warm before you ask. A cold Horizons took 448 ms to answer and a warm one about 30, so Jarvis wakes it when the assistant opens.

§ 05

Updates it can install, and undo from a phone

Jarvis can install its own updates, and I can roll one back with nothing but the phone. That raises an obvious problem: whatever can publish an update can take over the assistant. So signing lives in a separate repository that never runs the app's code next to the key, and a release must pass six checks in order. Break one to see what happens.

Under the hood
  • Why a separate repository. On GitHub's free plan a private repository has no branch protection or environment secrets, so every workflow on every branch can read every secret. With the key beside the app, any pushed branch could sign code that Jarvis would install.
  • App code never shares a runner with a secret. The build job runs Gradle and npm from a read-only checkout with no secrets and outputs an unsigned APK. The publish job signs it and runs the app only inside an emulator.
  • A pull request carries its own CI. So the CI files and the scripts they call must match pinned hashes byte for byte; a PR that weakened its own tests would otherwise pass them.
  • Rollback keeps your data. It rebuilds the previous code under a higher version number, so Android treats it as an upgrade and keeps data and permissions. About fifteen minutes, started from the GitHub app.

§ 06

What is not proven yet

Jarvis runs on my phone, and Horizons has signed releases through v2.2. These are the parts I have not seen work yet, so I do not claim them:

  • Owner-only voice is tuned on synthetic voices only, not yet on a human one.
  • The battery cost of the always-on wake word since its last change. Before it, the phone lost 1.2% an hour with the screen off.
  • A silent self-update past Google Play Protect; today an update needs one tap.
  • A spoken cloud question in the last phone review: the typed ones were answered, and no spoken one was recognised.