AI in Practice
AI & the Web
How sites are read by machines now - llms.txt, agents.txt, and the audits that have started measuring it.
Everything in AI & the Web
09/27/2026
Fake AI Crawler User Agents in Nine Days of Server Logs
Nine days of server logs, one odd day, and AI crawler names that were rare visitors or not who they said they were.
09/27/2026
llms.txt: "File does not appear to contain any links"
Lighthouse fails an llms.txt with no Markdown links. Bare URLs and autolinks don't count. Here is the exact check it runs, and the fix.
09/27/2026
What AI Crawlers Do With robots.txt: Nine Days of Logs
How often each AI crawler rereads robots.txt, which ones mostly read nothing else, and which never ask for it under their own name.
09/24/2026
Who's Crawling Us? An AI Bot Census of Five Sites' Logs
One week of Apache logs from five small sites: which AI crawlers came, how often, whether they read robots.txt, and who fetched llms.txt (none of them).
09/02/2026
Lighthouse now grades your site for AI agents
Lighthouse has a fifth category, Agentic Browsing, and one of its audits reads your llms.txt. Three sites failed it for the same undocumented reason, and the API will not give you the category unless you ask for it by a name that is not in the docs.