0052: The Monitor keeps requests on disk

Status: accepted. It replaces the row for requests and dumps in 0029: they're no longer memory only.

Context

The Monitor shows, for every project, what each request did: its stages, queries, cache, mail, outgoing calls, exceptions, the jobs it queued and, for a page, the browser's timings and errors. A request you want to look at is often one from yesterday, or from before an update, and slow routes and N+1s only show over many requests. So the history has to survive restarts. Until now requests stayed in memory, and the docs promised that nothing from a request was written to disk, because headers and bodies carry cookies, tokens and passwords.

Decision

• Its own database. The Monitor writes to db/monitor.db, a second SQLite database next to leash.db, opened, recovered and kept private (0600 in a 0700 folder) the same way. It gets no daily copy: it's history, not settings, and a damaged file is moved aside and a new one started. Bodies over 64 KB go to db/monitor-blobs/.
• What it keeps:
  • each request's trace (the proxy's record, the app's execution and its events), and jobs, commands and page views on their own;
  • routes, query shapes, exceptions, jobs, outgoing hosts and pages, counted per hour with a percentile sketch, so P95 outlives the rows it came from;
  • the canonical log line of each request, job and command.
• Masked on disk, as it happened in memory. Before anything is written, header and cookie values, keys that look secret (password, token, secret, key, code, session, auth…) in query strings, forms, JSON bodies, event attributes and URLs, query values bound to such columns, and values that look like tokens (JWTs, bearer tokens, API keys, the Monitor's own token) become [masked]. The live view, the last 200 traces per site, keeps the request as it was while Short Leash runs.
• Bodies only where they help. HTML pages, API responses and error pages keep their bodies, up to 1 MB. Assets keep none.
• Shared traffic is kept, marked shared. It's capped at 64 KB per body and at 20% of the size limit, so a busy share can't push out your own history. AI tools get shared bodies only when they ask for them, since a visitor wrote them.
• Retention.
  • Age: 7 days, with a size limit of 1 GB, both adjustable in Settings › Monitor.
  • Size: past the limit, the oldest go first, 500 at a time, until it's under 90%.
  • Trends: hourly counts are kept 30 days.
  • Clear history: clears it, with the live requests.
• Never in the way. The proxy never waits on the Monitor. A writer saves in batches (500 rows or every 250 ms); if it falls behind, rows are dropped and counted. Without its database Short Leash starts as before, and the Monitor keeps only what it sees while Short Leash runs.

Alternatives

• Requests in leash.db would put a gigabyte of churn on the database that holds tasks and deliveries, and into its daily copy.
• Saving only aggregates keeps no trace to open when something went wrong yesterday.
• Masking in memory too makes the live view useless for debugging the request in front of you.

Consequences

• Request headers and bodies are on disk now, masked. Masking by name and shape can miss a secret in free text, such as a dump of an object holding one; the docs say so.
• A busy app at 1 GB holds hours rather than days. The usage line in Settings shows how far back it goes.
• Invariant 11 changes: captured data on disk now includes the Monitor's database and its body files, under the same permissions and retention rules.