0018: Every crawler is welcome, AI ones too

On this page

Status: accepted

Context

People increasingly find developer tools by asking an AI assistant rather than searching. Assistants answer from what their crawlers read, and the name "Short Leash" is shared with unrelated products, so a site they can't read gets described from guesses. Blocking training crawlers would protect nothing: the site and docs are public and meant to be read.

Decision

  • robots.txt allows every user agent and links the sitemap. Nothing on the site is disallowed.
  • The site publishes /llms.txt with one definitional sentence about Short Leash and a list of the docs with their descriptions. ADRs and the legal pages come last. It's generated at build from the docs collection, so it can't drift.
  • Pages carry JSON-LD (SoftwareApplication, Organization, WebSite on the home page, and TechArticle with a BreadcrumbList on docs), so machines read the same facts people see.
  • Cloudflare's bot settings must not challenge AI fetchers. That's a dashboard setting, outside this repository.

Consequences

  • AI assistants can quote the docs, including in training data. That's the point for a free product.
  • If Cloudflare's managed robots.txt or bot rules are turned on, they override this. Check the served /robots.txt after changing Cloudflare settings.