What is a robots.txt file?

A robots.txt file is placed at the root of your website and tells web crawlers which pages they are allowed or not allowed to crawl and index. It follows the Robots Exclusion Standard.

How do I block AI training bots with robots.txt?

Use the "Block AI Bots" preset to add Disallow: / rules for GPTBot, ChatGPT-User, Claude-Web, anthropic-ai, CCBot, Google-Extended, PerplexityBot, and others.

Where do I upload my robots.txt file?

The robots.txt must be placed at the root of your domain — at https://yourdomain.com/robots.txt. Placing it in a subdirectory will not work.

Does robots.txt affect SEO?

Yes. Blocking important pages prevents them from being indexed. A well-configured robots.txt focuses crawlers on valuable content and improves crawl budget efficiency.

Robots.txt Generator

Create a valid robots.txt file for your website. Block AI crawlers, set crawl rules for specific bots, and add your sitemap — all with a live preview.

Quick presets:

Rule 1

User-agent

Common bots

Allow / Disallow paths

Crawl-delay (optional)

Sitemap URLs

Add the full URL(s) to your sitemap(s)

Live Preview

robots.txt — updates in real time

User-agent: *
Disallow: /

Real-time Preview

See your robots.txt update instantly as you fill in the form — no guessing.

Block AI Crawlers

One-click preset blocks GPTBot, Claude-Web, CCBot, Google-Extended, and more.

Valid Syntax

Generates correctly formatted output that search engine crawlers can parse.

How to Create Your robots.txt

Four simple steps

Choose a preset

Start with a one-click template or begin from scratch.

Add rules

Set User-agent, Allow/Disallow paths, and crawl delay for each bot.

Add sitemap URLs

Paste your sitemap URL(s) so search engines can find them.

Download

Download robots.txt and upload it to the root of your website.

Frequently Asked Questions

Robots.txt Generator

Real-time Preview

Block AI Crawlers

Valid Syntax

Choose a preset

Add rules

Add sitemap URLs

Download

What is a robots.txt file?

How do I block AI training bots?

Where do I place the robots.txt file?

Does robots.txt affect my SEO?

What is Crawl-delay?