09/27/2026
What AI Crawlers Do With robots.txt: Nine Days of Logs
My AI bot census counted who crawls my five sites. Since then my Site Ops dashboard has kept a daily summary of every site's access log, so I now have nine complete days to look at a narrower question: what does each AI crawler do with robots.txt?
The window is September 18 to 26, 2026, server time, across belchamber.us, brandager.com, nextdayvideos.com, prolificfutility.com and this site. All five robots.txt files allow everything except a couple of leftover paths, so this is about reading the rules, not obeying them.
The short answer: three quite different habits, and one crawler that visited a site over a hundred times without ever looking at a page.
Three robots.txt habits
Rereads it all day
ClaudeBot fetched robots.txt on every site, roughly every two hours, while also crawling pages.
Reads the rules and little else
OAI-SearchBot spent most of its requests on robots.txt. On two of the five sites that was all it did.
Never asks under its own name
GPTBot and meta-externalagent made hundreds of requests and asked for robots.txt zero times as themselves.
The numbers
Requests by user agent, all five sites combined, September 18 to 26, 2026:
| Agent | Requests | robots.txt fetches | Share that was robots.txt |
|---|---|---|---|
| meta-externalagent (Meta) | 2,730 | 0 | 0% |
| ClaudeBot (Anthropic) | 1,555 | 473 | 30% |
| Bytespider (ByteDance) | 1,412 | 68 | 5% |
| Amazonbot | 505 | 2 | 0.4% |
| OAI-SearchBot (OpenAI) | 487 | 348 | 71% |
| GPTBot (OpenAI) | 314 | 0 | 0% |
| For comparison: Googlebot | 788 | 82 | 10% |
| For comparison: bingbot | 655 | 86 | 13% |
A user agent is a claim, not an identity, and I haven't checked these against the vendors' published IP ranges. The census post covers that caveat and the counting method.
OAI-SearchBot mostly checks the rules
OpenAI's crawler overview says OAI-SearchBot is used to surface websites in ChatGPT's search features. On my sites it behaves more like a doorman than a reader. From September 18 to 26, 2026, all 104 of OAI-SearchBot's requests to brandager.com were for robots.txt; it never fetched a page there. prolificfutility.com got the same treatment, 53 requests and all of them robots.txt. Only on this site did it read pages in any volume.
ClaudeBot rereads robots.txt constantly
From September 18 to 26, 2026, ClaudeBot fetched robots.txt 473 times across belchamber.us, brandager.com, nextdayvideos.com, prolificfutility.com and tools.belchamber.us, about ten and a half times per site per day. Anthropic says its bots honor robots.txt, and RFC 9309 says a crawler shouldn't trust a cached copy for more than 24 hours. ClaudeBot is comfortably inside that. A rule you add today should reach it within hours, not days.
Two quick answers
How often did OAI-SearchBot fetch pages on brandager.com?
From September 18 to 26, 2026, all 104 of OAI-SearchBot's requests to brandager.com were for robots.txt; it never fetched a page there.
How often did ClaudeBot fetch robots.txt on these five sites?
From September 18 to 26, 2026, ClaudeBot fetched robots.txt 473 times across belchamber.us, brandager.com, nextdayvideos.com, prolificfutility.com and tools.belchamber.us, about ten and a half times per site per day.
What I'd take from it
- A rule reaches ClaudeBot and OAI-SearchBot fast. Both reread robots.txt many times a day.
- Don't read GPTBot's silence as defiance. Zero robots.txt fetches under its own name doesn't prove it ignores the file. A vendor can fetch it under another agent name or cache it. It does mean you can't confirm it from your own logs.
- Check your logs before you rely on a rule. Nine days of summaries answered this in minutes. Without them, I'd be guessing.
This post is also half of a small test I'm running on whether structured data changes what AI answer engines repeat about a page. I'll write up what happened once there's enough to say.
If you'd like the same kind of look at who's reading your business website, and what to do about it, that's what my one-on-one AI guidance sessions are for. Let my Claude help your Claude. Book a free 30-minute consult.