0060: One local model service, run by leashd

Status: accepted. Revised after the local AI spike (stage 0): leashd starts the helper, because Metal works from leashd on Apple Silicon, and the GPU works inside the sandbox.

Context

Summaries, chat and later features share one language model of 2.7 to 18 GB. Running it inside leashd would let a model crash or a memory spike take the daemon down. A launchd daemon has no GUI session, which was expected to hide Metal's GPU; the spike found it doesn't on Apple Silicon with macOS 26.3 (the same speed as from a terminal). Model files are parsed by llama.cpp, whose GGUF parser has had memory-safety bugs. The llama-cpp-2 crate removed its chat-template and tool-call functions in June 2026.

Decision

• Who starts it. leashd starts leash-llm on demand and talks to it only over the stdin and stdout pipes it gives it. The helper opens no socket and no port, so a compromised helper can't reach leashd's API, and it exits when its stdin closes. leashd publishes its state (state.ai.llm: loading, the backend and the model) and restarts it at most three times a minute; a crash, including llama.cpp aborting on misuse, fails the request in hand and nothing else.
• First load. The first load after an update compiles the GPU's kernels (about 15 s on an M3 Max), so the state says "Preparing the model" until the helper reports ready.
• Jobs. Summary jobs carry a state and an attempt count, and each finished chunk is saved, so a crash, a restart or an update resumes where it stopped.
• Prompt reuse. Chat's tools take 4,000 prompt tokens, several seconds before the first word on a small Mac, so the helper reuses the evaluated prefix (system prompt and tools) between requests.
• Memory. One model is loaded at a time, and the helper exits after five minutes idle. On machines with 16 GB or less, leashd unloads it while the dictation model is loaded, and background work pauses during dictation.
• Chat code. Short Leash vendors llama-cpp-sys-2 with a patch that links the prebuilt libraries (0058), and a small C++ shim over llama.cpp's own chat-template and tool-call parsing.
• Build. It is built without OpenMP and links CoreGraphics explicitly; the backend it picked is logged, and the UI says "Running on the processor" when it isn't the GPU.
• Sandbox. The environment is cleared at spawn.
  • On macOS: a deny-default sandbox-exec profile with no network, no sockets and read-only access to the model.
  • On Windows: a restricted token in a Job Object, with no AppContainer, so uninstalling has nothing extra to remove.
  • If the GPU can't be used inside the macOS sandbox, models from Short Leash's catalog run on the GPU without it, and imported models stay sandboxed on the processor.
• Packaging on Windows.
  • Each engine lives in its own folder (engines\voice\, engines\llm\), because both ship DLLs with the same names from different ggml versions.
  • The DLLs use the dynamic C runtime, so the Visual C++ redistributable (0032) becomes required and is checked at start.
  • The installer renames the helpers and their DLLs aside during updates, as 0056 does for the other programs.

Consequences

• CLI chat, summaries and the opt-in endpoint work whenever leashd runs, with or without the app.
• Short Leash owns a few hundred lines of C++ that track llama.cpp's chat code, and every llama.cpp bump is re-tested. The weekly ShortLeash/runtimes job watches llama.cpp's security advisories.
• If a Mac shows no GPU from leashd (base M1, Intel and macOS 27 are still to be checked), the backend log and "Running on the processor" show it, and this decision gets revisited.