0018: Every crawler is welcome, AI ones too
On this page
Status: accepted
Context
People increasingly find developer tools by asking an AI assistant rather than searching. Assistants answer from what their crawlers read, and the name "Short Leash" is shared with unrelated products, so a site they can't read gets described from guesses. Blocking training crawlers would protect nothing: the site and docs are public and meant to be read.
Decision
robots.txtallows every user agent and links the sitemap. Nothing on the site is disallowed.- The site publishes
/llms.txtwith one definitional sentence about Short Leash and a list of the docs with their descriptions. ADRs and the legal pages come last. It's generated at build from the docs collection, so it can't drift. - Pages carry JSON-LD (
SoftwareApplication,Organization,WebSiteon the home page, andTechArticlewith aBreadcrumbListon docs), so machines read the same facts people see. - Cloudflare's bot settings must not challenge AI fetchers. That's a dashboard setting, outside this repository.
Consequences
- AI assistants can quote the docs, including in training data. That's the point for a free product.
- If Cloudflare's managed
robots.txtor bot rules are turned on, they override this. Check the served/robots.txtafter changing Cloudflare settings.