When to use this check
Are you limiting crawling, excluding a page from the index or protecting private data? These are three different tasks. The Googlebot rules below do not necessarily describe how other crawlers behave.
Know what you are checking.
Robots.txt controls access for cooperating crawlers. For Googlebot, the User-agent group and most specific matching path determine access. A Disallow rule alone can leave the URL visible in search results.
A practical example
User-agent: *
Disallow: /offers/
Allow: /offers/public/
Sitemap: https://example.com/sitemap.xmlA common mistake
Disallow does not reliably remove a URL from search or protect private data. A robots.txt server failure is also not an empty allow-all file; investigate availability before changing path rules.
Test your robots.txt rules
Paste the file contents, enter a URL and choose a crawler. Fukamo shows the deciding rule for that path.
The most specific crawler group and longest path match win; Allow wins a tie. The # fragment is ignored. This does not check the HTTP response, WAF or actual crawler visits. Provider behavior can differ.
An allow result does not prove indexing. Disallow is not noindex. This tool evaluates Allow and Disallow; it is not a full-file validator. Google robots.txt ↗ · OpenAI bots ↗
Understand the result. Make the right change.
Which crawler and host a robots.txt file applies to
Check the robots.txt belonging to the exact protocol, host and port. A file on the main domain does not automatically describe a separate subdomain.
- Select the applicable User-agent group before reading its paths.
- For Googlebot, specific groups are separate from the wildcard group.
- Keep one allowed and one blocked test URL beside each changed rule.
How to test Allow and Disallow rules
In the example above, /offers/public/bike is allowed but /offers/draft/bike is blocked. A similar-looking path outside /offers/ should remain unaffected.
- Paths are case-sensitive; test /Offers/ separately.
- A longer matching path is more specific; Allow wins an equally specific conflict for Google.
- Do not confuse an empty Disallow value with Disallow: /, which targets the whole site.
When to use robots.txt, noindex or authentication
Disallow manages crawling. It does not protect a private document or reliably remove an already known URL from search.
- For indexing exclusion, let the crawler retrieve the noindex instruction.
- For private content, require authentication.
- Keep resources needed for public page rendering accessible when checking JavaScript SEO.
How to check robots.txt availability and file size
A correct CMS setting is not enough if the public endpoint returns an error or stale content. Check the response before interpreting a crawler’s behavior.
- Confirm a text file rather than an HTML login or error page.
- Google ignores content beyond 500 KiB; consolidate oversized rule lists.
- Investigate 5xx and 429 responses. A robots.txt 404 is not a crawl prohibition for Google.
Primary reference: developers.google.com ↗
Frequently asked questions
Will Disallow remove a URL from Google?
Disallow restricts crawling, not necessarily the appearance of a URL in results. A blocked crawler may never see a page-level noindex. For a public page you want excluded, allow retrieval and use the appropriate indexing directive. Protect private content with authentication; robots.txt is not access control.
Does robots.txt belong in an individual page folder?
Crawlers look for /robots.txt at the root of the relevant host. Subdomains do not automatically inherit each other’s rules. Check the file on the host that serves the URL being inspected.
Why is a robots.txt tester not sufficient on its own?
A tester evaluates the supplied rules for a chosen user agent and path. A CDN, firewall, login requirement or server response can independently affect access. Compare the result with an actual request and response.
When the task is complete
Check the effective rule for the exact path and user agent, then inspect the unauthenticated page response. A valid file is insufficient if its behavior contradicts the intended access policy.