09/15/2026

A 403 Isn't a Broken Link: Bot Blocking in Link Checkers

Usually not. A 403 Forbidden means the server received the request and chose not to answer it. RFC 9110, the HTTP specification, defines it as a server that "understood the request but refuses to fulfill it." To a link checker, that refusal looks just like a failure. To a person clicking the same link in a browser, the page loads fine.

Plenty of sites refuse requests that look automated. So when your checker reports a batch of 403s from one big site, the likeliest explanation is that the site is blocking your checker, not that the links are dead.

A 404 isn't proof either

The same specification lets a server hide a page it doesn't want to serve: an origin server "MAY instead respond with a status code of 404 (Not Found)." So a 404 can mean the page is gone, or it can mean the server doesn't want to talk to you.

I hit exactly this. An outbound link check on one of my sites flagged three product pages on a large manufacturer's website as dead. They weren't. When I tested the host, it returned the same error for its own home page, and for a made-up URL that had never existed. A server that answers identically for a real page, its root and a nonsense path isn't telling you anything about any of them. It's refusing to talk to a script.

The user agent made it worse. Identifying the checker honestly got a 404; pretending to be a browser got a 403 for the same page. Same page, two different wrong answers.

Three outcomes, not two

A useful link checker needs three outcomes, not "fine" and "broken":

  • Reachable. The page answered normally.
  • Gone. A 404 or 410 from a host that answers normally for other pages.
  • Could not verify. Blocked, timed out, or refused. Report this on its own, never as broken.

The distinction matters because a report full of false alarms gets argued with, then skimmed, then ignored, and a check nobody believes protects nothing.

A small Python check that tells them apart

The cheap trick: when a URL fails, ask the same host for its home page and for a path that can't exist. If the host answers those normally, a 404 on your URL is real. If it fails on everything, you can't verify anything.

import secrets
from urllib.parse import urlsplit

import requests


def classify(url: str, timeout: float = 15) -> str:
    """Return 'reachable', 'gone', or 'could not verify'."""
    try:
        status = requests.get(url, timeout=timeout, allow_redirects=True).status_code
    except requests.RequestException:
        return "could not verify"
    if 200 <= status < 400:
        return "reachable"

    parts = urlsplit(url)
    root = f"{parts.scheme}://{parts.netloc}/"
    invented = root + secrets.token_hex(8) + "-does-not-exist"
    try:
        root_status = requests.get(root, timeout=timeout).status_code
        invented_status = requests.get(invented, timeout=timeout).status_code
    except requests.RequestException:
        return "could not verify"

    host_answers_normally = 200 <= root_status < 400 and invented_status in (404, 410)
    if status in (404, 410) and host_answers_normally:
        return "gone"
    return "could not verify"

A 403 always comes back as "could not verify". Even on a host that otherwise behaves, a forbidden page might be behind a login rather than missing, and the checker can't tell which.

Images are the exception

Everything above is about links you send readers to. Images embedded on your own page are different. If another host refuses to serve an image to your visitors (hotlink protection often returns a 403), the image is broken for everyone who views your page, whatever the reason.

That's why this site's Broken Image Checker counts a 403 as broken: for an image on your page, the visitor sees an empty box either way. It checks src and srcset, lazy-loading attributes and CSS backgrounds, right in your browser.

FAQ

Why do sites block link checkers? To keep out scrapers and reduce automated load. A link checker looks like any other bot unless the site has allowed it.

Should I pretend to be a browser to avoid blocks? It sometimes changes the response, but not reliably, and it can breach a site's terms. Classifying the result honestly is more dependable.

When should I fix a "could not verify" link? Open it yourself in a browser. If it loads, leave it. If it doesn't, you've found a real problem the checker couldn't prove.

Filed under