What is a robots.txt file?
Robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they may or may not request. It is a set of instructions, not a lock: well-behaved crawlers like Googlebot and Bingbot read it before crawling and respect it, but it cannot force a bot to comply and it does not encrypt or hide anything from a determined visitor.
Every B2B SaaS site needs one, if only to point crawlers at the sitemap and keep them out of admin, checkout and internal search paths that add no SEO value. It is also the first place to decide which AI crawlers, like GPTBot or ClaudeBot, are allowed to read your content at all.
How this robots.txt generator works
The syntax: User-agent: [which bot this block applies to, or * for all] Disallow: [a path this bot should not crawl] Allow: [a path allowed even inside a disallowed folder] Sitemap: [absolute URL to your sitemap.xml] Crawl-delay: [seconds between requests, non-standard]
Each User-agent block only applies to the bot(s) named in it. A bot that matches a specific block, like GPTBot, follows only that block; it does not also inherit the rules written under User-agent: *. Crawl-delay is honored by some crawlers like Bingbot but is explicitly ignored by Googlebot, which paces its own crawl rate through Search Console instead.
Robots.txt directives, explained
| Directive | What it does | Example |
|---|---|---|
| User-agent | Names the bot the following rules apply to | User-agent: Googlebot |
| Disallow | Blocks that bot from crawling a path | Disallow: /admin/ |
| Allow | Re-opens a path inside a disallowed folder | Allow: /wp-admin/admin-ajax.php |
| Sitemap | Points crawlers to your sitemap.xml | Sitemap: https://site.com/sitemap.xml |
| Crawl-delay | Requests a pause between crawl hits | Crawl-delay: 10 |
Should you block AI crawlers like GPTBot and ClaudeBot?
For most B2B SaaS sites, no. Blocking AI crawlers keeps your product docs, comparison pages and blog posts out of ChatGPT, Claude and Perplexity answers at the exact moment a buyer is researching, which is a growing share of top-of-funnel research. The tradeoff is real: allowing GPTBot also means your content can be used in model training, and there is no way to opt into citation without also opting into that.
There are legitimate reasons to block one anyway: a site under heavy AI-scraper load, content you specifically don't want reused for training, or a company policy that treats this as an IP decision rather than a marketing one. Read how to get crawled by AI bots before you block anything, since the wrong Disallow line here has a direct, measurable cost to AI visibility.
Common robots.txt mistakes to avoid
Disallowing the Entire Site With a Single Slash
Disallow: / under User-agent: * blocks every crawler from every page on your domain. This is the single most common and most damaging robots.txt error, and it usually happens by accident: a staging-site rule that never got removed before launch, or a developer testing a lockdown that shipped to production. The effect is not subtle. Search engines stop crawling new content, existing pages can eventually drop out of the index as their cached data ages out, and organic traffic can fall to close to zero within weeks. Always load the live file at yourdomain.com/robots.txt after any deploy and confirm the root block reads Disallow: with nothing after it, or lists specific paths, never a bare slash.
Using Robots.txt to Hide Pages From Search Results
Disallow stops crawling, not indexing. If other sites link to a blocked URL, Google can still index and show it in results, typically as a bare link with no title or description pulled from the page, because Google never crawled it to read that content. Site owners who disallow a login page or a thin internal-search results page to keep it out of Google are often surprised to find it indexed anyway months later. The fix for genuinely wanting a page out of search is a noindex meta tag or X-Robots-Tag header, and that page has to stay crawlable so the bot can actually see the noindex instruction. Robots.txt and noindex solve different problems and combining them incorrectly, disallowing a page you also noindex, can backfire since the crawler may never reach the tag.
Forgetting the Sitemap Directive
A Sitemap line costs one line and gives every crawler a direct map to your most important URLs instead of relying on internal links alone to be discovered. Skipping it isn't fatal, since sitemaps can also be submitted directly in Search Console, but it means anonymous crawlers checking robots.txt as their first move, which includes most AI crawlers, never find it. For a site adding or updating pages regularly, that is a small, free discovery signal left on the table for no reason.
Blocking CSS and JavaScript Files
- The old advice was wrong for modern search. Blocking
/assets/,/wp-content/themes/or similar folders used to be common practice to "save crawl budget," but Google explicitly asks for CSS and JS to stay crawlable so Googlebot can render the page the way a visitor sees it. - A blocked stylesheet can hide mobile-usability problems. If Google can't fetch your CSS, it may render a broken or unstyled version of the page and misjudge things like mobile responsiveness or layout shift.
- Check with a live render test, not just the file list. Google Search Console's URL Inspection tool shows exactly what Googlebot rendered; if it looks stripped of styling, a robots.txt rule is the first thing to check.
Placing the File Anywhere Other Than the Root
Robots.txt is only honored at the exact root of a domain or subdomain, such as https://yourdomain.com/robots.txt. A copy sitting at /blog/robots.txt or on a different subdomain is simply invisible to crawlers checking the real one; each subdomain, including a www and non-www version if both resolve, needs its own file at its own root. This trips up multi-subdomain B2B SaaS setups often, where a marketing site, an app, and a docs portal each need their own robots.txt reviewed separately rather than assuming one file covers the whole domain.
Frequently asked questions
How do I create a robots.txt file?
Write a plain text file that starts with a User-agent line, followed by Disallow or Allow rules for the paths you want to restrict or open up, then upload it to the root of your domain as yourdomain.com/robots.txt. This generator builds that file for you: pick your rules, add a sitemap URL, and copy or download the result.
Where do I upload my robots.txt file?
Robots.txt must sit at the root of your domain, for example https://yourdomain.com/robots.txt, not in a subfolder. Search engines and other crawlers check that exact location automatically; you don't submit it anywhere. Most CMS platforms have a settings page or file manager where you can upload or edit it directly.
Can robots.txt block AI crawlers like GPTBot or ClaudeBot?
Yes, by adding a dedicated User-agent block naming that bot with Disallow: /. This tool includes presets for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and OAI-SearchBot. Blocking them keeps your content out of AI training and out of AI answer citations, so most B2B SaaS sites are better off leaving them allowed.
Does robots.txt guarantee a page won't appear in Google?
No. Disallow only stops crawling, and a blocked URL can still be indexed and shown in search results, usually without a description, if other sites link to it. To fully keep a page out of search results, use a noindex meta tag or header instead, which requires the page to be crawlable so Google can see the noindex instruction.