0063: An opt-in OpenAI-compatible endpoint

On this page

Status: accepted

Context

Editors and scripts on the same computer often speak the OpenAI API. Letting them use Short Leash's model saves a second download and a second runtime. Local model servers have a history of serious bugs: unauthenticated access, path traversal through model fields, DNS rebinding from web pages, and memory errors from unchecked parameters.

Decision

  • Off by default. Settings › Models › More › Let other apps use the model serves /v1/chat/completions, /v1/completions and /v1/models on 127.0.0.1 at defaults::AI_API_PORT, next to the dump and OpenTelemetry ports. If the port is busy, Short Leash names the process holding it rather than picking another.
  • The key. It is at least 256 bits from the operating system's random source, starts with dk-local-, and is stored in secrets/ai.json at 0600. It is shown only through its own route and can be replaced. It is compared in constant time, checked before the body is read, required on every route and never accepted in a query string.
  • Refused: a Host other than 127.0.0.1 or localhost with the port; any request with an Origin or Sec-Fetch-* header; unknown fields; numbers out of range. No CORS headers are sent.
  • Limited:
    • No route or field names a file, a digest or a model path, and there are no routes that manage models.
    • Body and header sizes are capped. Slow or idle clients time out, and generation stops when the client disconnects.
    • One request runs at a time, below chat in priority, and the rest get 429.
    • Only the model already loaded is served, and Short Leash's tools are never exposed.
  • While the model loads the endpoint answers 503 "The model is loading. Try again in a moment." (0060).

Consequences

  • Invariant 2 holds: the listener is loopback only.
  • Another app can use the model only with the key the user copied into it, and a web page in a browser can't use it at all.
  • Apps written for other local servers need Short Leash's port and key; the docs list both.