0054: OpenTelemetry endpoint for the Monitor

On this page

Status: accepted

Context

Some apps Short Leash serves have no Short Leash agent (Go, Deno, Bun, Elixir, anything unusual), and some are already instrumented with OpenTelemetry. They should show in the Monitor as fully as the frameworks Short Leash knows, without a collector to install and without overriding where a project already sends its telemetry.

Decision

  • An endpoint in leashd. OTLP over HTTP on 127.0.0.1 (port 9914, or a free one): /v1/traces and /v1/logs, in protobuf or JSON, gzipped or not. /v1/metrics is taken and dropped. Exports need the Monitor's token in x-leash-token. The messages are read with Short Leash's own copy of the few OTLP types it uses (prost, Apache-2.0), not the full gRPC stack.
  • Requests name themselves. The proxy replaces any traceparent on a request with one whose trace id is leash\0\0\0 followed by the request's id. Its span id stands for the proxy, so the app's server span hangs from it, and a span that comes back names its request without a lookup.
  • Spans become the Monitor's events.
    • A request's server span becomes the app's execution (source otel), with its route and status.
    • Database spans become queries, client HTTP spans become outgoing calls, producer spans become queued jobs, and exception events become exceptions with frames read from the stack trace.
    • Spans wait for their server span (it ends last) or up to 10 seconds. If they land after their trace closed, they still join it while it's live.
    • Logs written inside a request join its trace, and canonical lines among them are merged like any other.
  • Only when unset. An app process gets OTEL_EXPORTER_OTLP_ENDPOINT, its protocol, the token header, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES=leash.site=… and short export delays. It gets them only when neither its Short Leash environment nor its .env, .env.local, .env.development or .env.development.local sets up an exporter. Deno also gets OTEL_DENO=true. The site's "Send OpenTelemetry to Short Leash" switch sends it anyway: Short Leash's values then beat what those files set, per-signal endpoints included, but never what the site's Short Leash environment sets. Short Leash's token header only goes beside Short Leash's own endpoint, so it never reaches another collector.

Alternatives

  • Bundling an OpenTelemetry Collector is a large separate process to download, verify and run for each user.
  • The standard port 4318 would collide with a collector the developer runs themselves.
  • Mapping trace ids through a table would need spans to wait for the proxy's record before they could be placed. The id carries the request directly.

Consequences

  • An instrumented app's spans reach Short Leash with no setup, and an app with its own exporter keeps sending where it did.
  • Every request's traceparent is Short Leash's. A service an app calls through Short Leash's proxy gets a request and trace of its own, so calls between local services show as separate requests, not one trace across them.
  • Without its values, an OpenTelemetry query can't be told from a repeat with the same values, so duplicates aren't found for it. N+1s still are.