A free robots.txt generator for every website
Every website that wants to rank on Google should tell search engines where they can and cannot go. That job belongs to a small text file called robots.txt. The BlogDada robots.txt generator lets you create that file in a few clicks, without memorising any syntax or worrying about typos that could accidentally hide your whole site from search engines.
Pick a ready-made preset for WordPress, WooCommerce or Blogger, add your XML sitemap, choose which folders to keep private, and decide whether AI training bots should be allowed. The file updates live as you type, warns you about common mistakes, and includes a built-in tester so you can check any URL before you publish. When you are happy, download it or copy it straight to your clipboard.
Whether you run a personal blog, an online store, a business website or a large content site, this robots.txt generator gives you a correct, search-friendly file in under a minute.
What is a robots.txt file?
A robots.txt file is a plain text file that sits in the root folder of your website, for example at https://example.com/robots.txt. It follows a standard called the Robots Exclusion Protocol. When a search engine crawler such as Googlebot or Bingbot visits your site, the first thing it does is read this file to learn which areas it is allowed to crawl.
The file is made of groups. Each group starts with a User-agent line that names a crawler, followed by Allow and Disallow lines that list the paths that crawler may or may not visit. You can also add Sitemap lines that point crawlers to your XML sitemap.
It is important to understand what robots.txt does not do. It is a set of instructions, not a lock. Reputable crawlers such as Google, Bing and most AI companies follow it, but malicious bots can ignore it completely. It also controls crawling rather than indexing, which we explain in more detail below.
Why robots.txt matters for SEO
Search engines don't have unlimited time for every website. Google calls the number of pages it is willing and able to crawl on your site the crawl budget. On a small blog this rarely becomes a problem, but on stores, forums and large content sites, thousands of low-value URLs can compete for attention: internal search results, filtered category pages, session parameters, printer-friendly versions and cart pages.
A well-planned robots.txt file keeps crawlers away from those areas so they reach your important pages, such as new blog posts, product pages and updated guides, sooner. It also reduces unnecessary load on your server, which matters on shared hosting where aggressive bots can slow the site down for real visitors.
Just as important, the Sitemap line in robots.txt gives every crawler a direct path to your list of URLs. Even crawlers that never see your Search Console settings will find your sitemap there. That is why a robots.txt generator that makes adding sitemaps easy is one of the simplest technical SEO wins available to any site owner.
How to use this robots.txt generator
Creating your file takes five short steps:
- Pick a starting pointChoose Allow all for a simple site, or a preset for WordPress, WooCommerce or Blogger. Presets fill in sensible rules that most sites on that platform use.
- Add your sitemapPaste the full URL of your XML sitemap, such as
https://example.com/sitemap.xml. If you have several sitemaps, put each one on its own line. Don't have one yet? Create it with our free XML sitemap generator. - Adjust the rulesAdd any folders or pages you want to block, such as internal search results or thank-you pages. If one search engine needs a different rule, change it in the crawler list. Tick the AI option if you want to opt out of AI training.
- Test a URLType a path or full URL into the tester and pick a crawler. The tool tells you instantly whether that page is allowed or blocked, and which rule decided it.
- Download and uploadClick Download robots.txt and upload the file to the root folder of your website. Then open
yourdomain.com/robots.txtin a browser to confirm it is live.
Robots.txt syntax explained
You don't need to write any rules by hand when you use the generator, but understanding the syntax helps you make better decisions. Here are the directives the tool supports:
| Directive | What it does | Example |
|---|---|---|
User-agent | Names the crawler the following rules apply to. * means all crawlers. | User-agent: Googlebot |
Disallow | Tells the crawler not to visit a path. An empty value means nothing is blocked. | Disallow: /wp-admin/ |
Allow | Permits a path inside a blocked folder. The longest matching rule wins. | Allow: /wp-admin/admin-ajax.php |
Sitemap | Gives the full URL of an XML sitemap. It applies to all crawlers. | Sitemap: https://example.com/sitemap.xml |
Crawl-delay | Asks a crawler to wait between requests. Bing supports it; Google ignores it. | Crawl-delay: 10 |
Wildcards and pattern matching
Major search engines support two special characters. The asterisk * matches any sequence of characters, so Disallow: /*?replytocom= blocks every URL containing that parameter. The dollar sign $ marks the end of a URL, so Disallow: /*.pdf$ blocks every PDF file. When several rules match the same URL, Google follows the most specific one, meaning the longest path. If an Allow and a Disallow rule are equally long, Allow wins. The built-in tester follows these same rules.
Ready-made robots.txt examples
Allow everything
The safest file for most small websites simply allows all crawling and lists the sitemap:
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
WordPress
WordPress sites usually block the admin area while keeping the AJAX file open, because themes and plugins rely on it to display content correctly:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap_index.xml
Blogger
Blogger sites commonly block the internal search and label pages, which create thin, duplicate pages, while allowing everything else. You can paste the result into Blogger under Settings, Crawlers and indexing, Custom robots.txt.
WooCommerce and online stores
Stores often block cart, checkout and account pages, along with sorting and filter parameters, so crawlers spend their time on product and category pages instead of endless URL variations.
Should you block AI crawlers?
A growing number of AI companies crawl the web to collect training data. Many of them publish user-agent names and promise to respect robots.txt, including OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot and Google-Extended, which controls whether Google may use your content for its Gemini models.
Blocking these crawlers is a business decision rather than a technical one. Publishers who sell original content often block them to protect their work. Others allow them, hoping their brand will be mentioned more often in AI answers. Our robots.txt generator makes it one checkbox either way. Blocking Google-Extended does not affect your normal Google Search rankings, because regular search crawling is handled by Googlebot.
Robots.txt vs. noindex: know the difference
This is the most common robots.txt mistake. Blocking a page in robots.txt stops search engines from crawling it, but it does not guarantee the page stays out of search results. If other websites link to a blocked URL, Google can still index it and show it with no description.
If you want to keep a page out of Google, do the opposite: allow crawling and add a noindex robots meta tag or X-Robots-Tag header to the page. Google must be able to crawl the page to see that tag. Google also stopped supporting noindex rules inside robots.txt in 2019, so don't add them there.
Rule of thumb: use robots.txt to manage crawling, and use noindex to manage what appears in search results.
Common robots.txt mistakes to avoid
- Blocking the whole site by accident. A single
Disallow: /underUser-agent: *tells every crawler to stay away. This often happens when a staging site's file is copied to the live site. The generator warns you whenever this rule is active. - Blocking CSS and JavaScript. Google renders pages like a browser. If it cannot load your stylesheets and scripts, it may misjudge your layout and mobile-friendliness.
- Putting the file in the wrong place. Crawlers only read robots.txt from the root of the host. A file at
example.com/blog/robots.txtis ignored, and each subdomain needs its own file. - Using relative sitemap URLs. The Sitemap directive needs a complete URL including
https://. - Forgetting paths are case-sensitive.
/Private/and/private/are different paths. - Treating robots.txt as security. The file is public, and anyone can read it. Never list secret folders there; protect them with a password instead.
How to upload your robots.txt file
Hosting with cPanel or FTP
Open your hosting file manager or FTP client, go to the root folder of your domain (usually public_html), and upload the downloaded robots.txt. Replace any existing file. The name must be exactly robots.txt in lowercase.
WordPress
WordPress creates a virtual robots.txt automatically. To use your own, upload the file to the root folder as above, or paste the content into the robots.txt editor provided by SEO plugins such as Yoast SEO or Rank Math.
Blogger
Go to Settings, scroll to Crawlers and indexing, turn on Enable custom robots.txt, and paste the generated content.
Check it in Google Search Console
After uploading, open the robots.txt report in Google Search Console. It shows the version of the file Google last fetched, when it fetched it, and any lines it could not understand. You can also request a fresh crawl there after making changes.
Frequently asked questions
Is this robots.txt generator free?
Yes. The BlogDada robots.txt generator is completely free, needs no sign-up, and has no limit on how many files you create.
Where do I upload the robots.txt file?
Upload it to the root folder of your domain, usually public_html, so it opens at https://yourdomain.com/robots.txt. Crawlers do not look for it in subfolders.
Does robots.txt stop a page from appearing in Google?
Not reliably. Robots.txt controls crawling, not indexing. A blocked URL can still be indexed if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag instead.
Does Google support Crawl-delay?
No. Google ignores the Crawl-delay directive. Bing and some other crawlers respect it.
Should I block AI crawlers like GPTBot?
It depends on your goals. Blocking AI training crawlers keeps well-behaved bots from using your content for model training, but it may also reduce how often your content is referenced in AI tools. The choice is yours; the generator makes it one click either way.
Do I need a robots.txt file at all?
It is not required, but it is recommended. Without one, crawlers assume they can visit every page. A simple file that lists your sitemap and blocks admin or duplicate pages helps search engines crawl your site more efficiently.
Is robots.txt case-sensitive?
The file name must be lowercase robots.txt, and the paths in your rules are case-sensitive. Directive names such as Disallow are not case-sensitive.
How do I check if my robots.txt is working?
Open yourdomain.com/robots.txt in a browser to confirm it is live, then check the robots.txt report in Google Search Console, which shows the version Google fetched and any errors.
Create your robots.txt now
A correct robots.txt file helps search engines spend their time on the pages that matter, keeps private and duplicate areas out of the crawl, and points bots straight to your sitemap. With the BlogDada robots.txt generator you can build, test and download one for free in less than a minute. Go back to the tool and create yours.