Tools / ai crawler robots.txt generator
AI crawler robots.txt generator
Choose which AI crawlers may read your store. Get a robots.txt block you can paste into your theme.
Your inputs stay in your browser when JavaScript is enabled. We count anonymous results and successful file copies, without sending your inputs or using analytics cookies.
# AI crawler rules from citelift.app # Paste above your existing rules. # On Shopify: Online Store, Themes, # Edit code, robots.txt.liquid. # This is a change you make to your own # theme. Citelift never edits your theme. # Allowed. Permission to crawl does not guarantee inclusion or citations. User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: GPTBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: Amazonbot Allow: / User-agent: meta-externalagent Allow: / User-agent: CCBot Allow: / # Blocked. User-agent: Bytespider Disallow: /
How it works.
Read what each crawler is for
Every agent in the list says who operates it, what it fetches for, and why you might want it reading your store or not.
Tick the ones to block
Nothing is blocked by default except the agents that neither answer questions nor send traffic. The file rewrites itself as you tick.
Add your sitemap
Optional, and up to 2,048 characters, which is the longest URL anything will follow. Shopify already serves yours at /sitemap.xml, and naming it in robots.txt costs nothing.
Paste it into your theme
Shopify serves robots.txt from a template you can edit in the theme code editor. Copy the block and add it there yourself. Citelift never edits your theme; its storefront block is added through the theme editor, not through code.
Questions about this tool.
- Will blocking GPTBot keep my products out of ChatGPT?
- It stops that crawler fetching your pages. It does not remove anything a model already learned, and it does not stop an assistant citing a page about you on somebody else's site.
- Should I block anything at all?
- If you want to be named in answers, blocking the agents that produce those answers works against you. The list marks which is which so the choice is an informed one.
- Is robots.txt enforceable?
- No. It is a request that well-behaved crawlers honour. Treat it as a statement of intent rather than access control.
A good page still has to get named.
Get a search-backed snapshot of brand mentions and cited store URLs. No sign-in. Takes about a minute.