AI Crawler
Also known as: AI bot, LLM crawler
An automated client operated by an AI vendor that requests web content, identified by user agent claim and verified where possible against published address ranges.
An AI crawler is a client operated by an AI vendor to fetch web content, for search indexing, for retrieval in response to a user prompt, or for bulk collection. Vendors publish fingerprints and address ranges for many of them, which is what makes classification possible.
Purpose matters more than identity for measurement. A crawler fetching a page to answer somebody's question right now and one collecting content in bulk are different events that should not be counted together.
A user agent string is a claim, not an identity. Address-range matching is supporting evidence, not proof, and ranges change. One address presenting several vendors' crawler identities in a day is a useful spoofing signal rather than a guarantee, since shared infrastructure genuinely exists.