Who's Crawling Us? An AI Bot Census of Five Sites' Logs

Everybody has an opinion about AI crawlers. Very few of those opinions come with a log file attached. So I pulled one week of raw Apache access logs from the five sites I run and counted who actually showed up: ClaudeBot, GPTBot, Meta's crawler, Bytespider and the rest. I also checked whether they read robots.txt, and whether any of them fetched llms.txt.

The five are belchamber.us, brandager.com, nextdayvideos.com, prolificfutility.com and this site, tools.belchamber.us. They're all small static sites on the same DreamHost plan. The window runs from 00:00 on September 17, 2026 to about 03:00 on September 24, Pacific time, the server's own clock. That's roughly seven days and three hours, which is all the rotation had kept.

The counting was done by Claude Code over a read-only SSH session while I directed it. Nothing on the server was changed. The method is below, and so are the two ways it went wrong before it went right.

Gotcha 1: the log folder that looks right and isn't

On DreamHost, each domain has a folder under ~/logs/<domain>/ holding two symlinks, http and https. The http link points at a real folder full of real, rotated, gzipped access logs. For these five sites, though, its newest file was anywhere from early September back to 2024. Every site now answers over HTTPS, so the plain-HTTP folder stopped getting traffic, and nothing tells you it's stale.

The live logs are under https. If your numbers look oddly quiet, check the dates on the files before you trust the counts.

Method, and what it can't tell you

The logs are in Apache's combined log format, where the user agent is the third quoted field. Splitting each line on " puts the user agent in $6 and the request line in $2. A request counts for a bot when the bot's name appears anywhere in the user-agent field, compared case-insensitively.

Caveats, stated plainly:

• A user agent is a claim, not an identity. Anyone can send ClaudeBot in a header. I didn't verify source addresses against the vendors' published ranges. OpenAI's crawler documentation and Perplexity's crawler documentation both publish IP lists, and Google explains how to verify that a request really came from Google. If a decision depends on these numbers, do that step.
• Requests, not pages. A crawler fetching a page, its redirect target and its images shows up as several requests.
• One week is a small sample. Within a minute of my extraction closing, GPTBot made 44 requests to this site. All week it had made 24.
• An escaped quote inside a request line can throw off the field split. That's rare, but it happens.

As a cross-check, the first pass matched names anywhere on the line, which included URLs and referrers. After I removed the duplicated day, the two methods agreed to within a few hits per bot. So the substring shortcut wasn't the problem. The symlink was. A second, independent tally from my Site Ops dashboard, which stores a summary per site per day and so stops at the end of September 23, comes within about 1% of these totals for every bot it tracks.

robots.txt: who read it, and who obeyed it

First, what the files say. All five robots.txt files allow everything. Three also disallow leftover paths: /_site/ on nextdayvideos.com, /_site/ and /cms/ on prolificfutility.com, and /cms/ here. None of them names an AI crawler. According to each site's git history, none of the files changed during the window. The robots.txt standard itself is RFC 9309.

Zero violations, and I wouldn't lean on that. With only three narrow Disallow rules on paths nothing links to, a crawler had almost no reason to go there. Across the week, 208 requests hit a disallowed path. None came from a named AI crawler. The biggest single source was WordPress.com's Photon image service, still asking for images from an install that no longer exists. A real compliance test needs a rule that a crawler actually wants to break. I'll run that as a follow-up, with a Disallow aimed at one named bot on a path it's already crawling.

Some bots mostly just check the rules. On three of the five sites, every OAI-SearchBot request was for robots.txt: 85 of 85, 42 of 42 and 42 of 42. ClaudeBot re-read each site's robots.txt about ten times a day.

Three busy crawlers never asked for robots.txt under their own names: GPTBot, meta-externalagent and Amazonbot. That doesn't prove they ignore it. A company can fetch robots.txt under another agent name or cache it for longer than a week. (One thing it isn't: OpenAI says it may add a robots.txt marker to a crawler's user agent when fetching the file, but a GPTBot request marked that way still contains GPTBot, so it would have been counted here.) But if you're relying on a rule for one of these, check your logs and don't assume.

Remember that user-triggered fetchers play by different rules. OpenAI says that for ChatGPT-User, "robots.txt rules may not apply," and Perplexity says Perplexity-User "generally ignores robots.txt rules." Amazon's Amazonbot page says something similar for Amzn-User. Blocking the training crawler doesn't block the assistant a reader is using.

A reusable census script

This is a tidied version of the script behind the per-bot counts. Its list covers the main agents; add YouBot, DuckAssistBot, facebookexternalhit or any other name you want counted. It's read-only: it takes a log directory, reads regular files only (skipping the access.log.0 symlink) and uses zcat -f so gzipped and plain files work the same way.

#!/usr/bin/env bash
# bot-census.sh -- count AI and search crawler requests in Apache combined logs.
# Usage: bot-census.sh ~/logs/example.com/https
# The trailing slash on "$dir/" lets find descend a symlinked log directory.
dir="${1:?usage: bot-census.sh LOG_DIR}"
find "$dir/" -maxdepth 1 -type f -name 'access.log*' -print0 |
  xargs -0 zcat -f |
  awk -F'"' '
    BEGIN {
      n = split("ClaudeBot Claude-User Claude-SearchBot GPTBot OAI-SearchBot ChatGPT-User " \
                "PerplexityBot Perplexity-User meta-externalagent Bytespider Amazonbot " \
                "PetalBot Googlebot bingbot", bots, " ")
    }
    {
      ua = tolower($6)            # user agent: third quoted field
      split($2, req, " ")         # "GET /path HTTP/1.1"
      path = req[2]
      for (i = 1; i <= n; i++) {
        if (index(ua, tolower(bots[i]))) {
          hits[i]++
          if (path == "/robots.txt") robots[i]++
          if (path ~ /^\/llms(-full)?\.txt/) llms[i]++
          break
        }
      }
    }
    END {
      printf "%-20s %8s %8s %8s\n", "agent", "hits", "robots", "llms"
      for (i = 1; i <= n; i++)
        printf "%-20s %8d %8d %8d\n", bots[i], hits[i], robots[i], llms[i]
    }'

To check compliance, add one line per Disallow prefix inside the loop, for example if (path ~ /^\/cms\//) blocked[i]++, and print it with the rest. Run it against a single day's file first and compare the result to a quick grep -ci claudebot on the same file. My first pass double-counted a day, so review AI-written scripts like this one before you trust their numbers.

Want this for your own sites?

This whole census took one conversation: I asked, Claude Code did the digging over a read-only connection, and I checked the numbers. If you run a business website and want to know who's really reading it, human or bot, and what to do about it, that's the kind of thing my one-on-one AI guidance sessions cover. Let my Claude help your Claude. Book a free 30-minute consult.