What AI crawlers actually do to your traffic

A growing share of the requests hitting any public site aren't people at all — they're automated fetchers pulling in content to train or answer questions from language models. Some announce themselves honestly in their user agent. Others don't bother.

The well-behaved ones respect robots.txt and identify themselves clearly, which makes them easy to separate from the traffic you actually care about. The rest look almost identical to a normal browser unless you're looking for the small tells: missing headers, unusual request timing, or a user agent that doesn't match how the client actually behaves.

None of this is inherently bad — a lot of that traffic is harmless. But knowing the difference between "a crawler indexed this page" and "a script is scraping every page on the site once a minute" is the whole point of watching for it.