ClaudeBot

ClaudeBot is the user agent Anthropic documents for gathering web content that could potentially contribute to training its generative models. Anthropic says that restricting it signals a site's future material should be excluded from its training datasets. Two other agents decide whether a store can show up when someone asks Claude a question. Claude-SearchBot navigates the web to improve the quality of Claude's search results; Anthropic says disabling it prevents your content being indexed for search and may reduce your visibility in those results. Claude-User fetches a page when a person's question needs it, on that person's behalf. The three are set separately in robots.txt, so a store can decline training while staying reachable for search. That is the same split ChatGPT's crawlers make between GPTBot and OAI-SearchBot. Blocking every Anthropic-shaped name removes you from both jobs, which is rarely what a store owner meant. Check Anthropic's crawler documentation, read 23 September 2026, before editing, because agent names have changed before. Allowing a search agent makes a page eligible to be read; it does not guarantee a mention.