Anthropic crawler controls are now separated into three different bots, giving website owners more precise choices than a single on-off switch. The system is described as using multiple automated agents to collect public web data for model development, to support search and to fetch pages when users request specific information.
The documentation lists three user agents and explains what changes when each one is blocked. ClaudeBot is described as the crawler that collects web content that could potentially be used for training and blocking it signals that a site’s future materials should be excluded from training datasets. Claude-User is positioned as a user-initiated fetcher and disabling it prevents the system from retrieving a site’s content in response to user prompts. Claude-SearchBot is described as crawling for search quality and blocking it can reduce indexing for search optimization and lower visibility in Claude-powered search results.
Anthropic says its bots follow robots.txt rules, including Disallow directives and it also supports Crawl-delay to reduce crawl intensity. It also notes opt-outs must be set for each subdomain. Anthropic adds that IP blocking is not a dependable opt-out method and that it does not currently publish IP ranges, while also stating it will not attempt to bypass CAPTCHAs.
Bot summary
- ClaudeBot: Potential training collection – blocking signals exclusion from training datasets
- Claude-User: User-triggered retrieval – blocking stops user-directed fetching
- Claude-SearchBot: Search crawling/indexing – blocking reduces indexing for search optimization
