Free online tools
robots.txt generator
Decide what crawlers may fetch, point to your sitemap and optionally block crawlers that train AI models. Copy or download the finished file.
For example /admin or /cart. A path starts with a slash.
GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and others.
Your data never leaves your browser. Nothing is sent to a server.
How to use it
- Leave crawling of the whole site allowed by default.
- Enter paths to exclude, one per line.
- Add the full sitemap URL.
- Copy or download the file and place it in the domain root as /robots.txt.
What robots.txt is not
It is a request to crawlers, not a security control. Well-behaved bots follow it, but the file is public and protects no data. Protect private content with a login, not with a robots.txt entry.
Blocking AI bots
The option adds rules for popular crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended. Blocking does not affect ordinary Google or Bing search. Consider whether you want to be cited in AI assistant answers, since blocking limits that as well.
Disallow does not remove pages from results
A blocked URL can still appear in results if other pages link to it. To remove a page from results use a noindex tag and let the crawler read it.
Frequently asked questions
Related tools
- Google SERP previewSee how your title and description look in Google and whether they get truncated.
- Meta tag and Open Graph generatorReady code for title, description, canonical, Open Graph and Twitter Card.
- JSON-LD schema generatorStructured data for Organization, LocalBusiness, FAQPage and Article.
Need more than a tool?
If there is a bigger project behind it, a website, a shop or an app, describe it in a short brief. We reply with specifics and no obligation.
Describe your project