Robots.txt Checker
Fetch the site’s robots.txt and evaluate crawler rules for the submitted page.
Test your robots.txt rules
Paste the file contents, enter a URL and choose a crawler. Fukamo shows the deciding rule for that path.
The most specific crawler group and longest path match win; Allow wins a tie. The # fragment is ignored. This does not check the HTTP response, WAF or actual crawler visits. Provider behavior can differ.
An allow result does not prove indexing. Disallow is not noindex. This tool evaluates Allow and Disallow; it is not a full-file validator. Google robots.txt ↗ · OpenAI bots ↗
Will Disallow remove a URL from Google?
Are you limiting crawling, excluding a page from the index or protecting private data? These are three different tasks. The Googlebot rules below do not necessarily describe how other crawlers behave.
If robots.txt allows a product but Search Console reports exclusion, inspect the product’s canonical URL and indexing directives. The XML sitemap should list the preferred product address. Compare all three signals for the same URL.
What to do step by step
- Fetch the exact host’s /robots.txt and check its response status.
- Select the applicable User-agent group and test an allowed and a blocked URL.
- Check path case and specificity; equally specific Allow wins for Googlebot.
- Publish the repair and retest the delivered file, including rendering resources.
How to allow a page blocked by robots.txt
Teaching example; example.com addresses are illustrative.
Before
User-agent: *
Disallow: /offers/After
User-agent: *
Disallow: /offers/
Allow: /offers/public/Enter https://example.com/offers/public/bike in the rules tester. It is blocked before the change; the longer Allow match wins afterward. https://example.com/offers/draft/bike stays blocked. Test both paths.
A common mistake
Disallow does not reliably remove a URL from search or protect private data. A robots.txt server failure is also not an empty allow-all file; investigate availability before changing path rules.
Try the report before checking your site.
Explore the sample report, then enter your own URL to see findings, HTML evidence and a repair plan for your page.
Run your audit
Which crawler and host a robots.txt file applies to
Check the robots.txt belonging to the exact protocol, host and port. A file on the main domain does not automatically describe a separate subdomain.
- Select the applicable User-agent group before reading its paths.
- For Googlebot, specific groups are separate from the wildcard group.
- Keep one allowed and one blocked test URL beside each changed rule.
How to test Allow and Disallow rules
In the example above, /offers/public/bike is allowed but /offers/draft/bike is blocked. A similar-looking path outside /offers/ should remain unaffected.
- Paths are case-sensitive; test /Offers/ separately.
- A longer matching path is more specific; Allow wins an equally specific conflict for Google.
- Do not confuse an empty Disallow value with Disallow: /, which targets the whole site.
When to use robots.txt, noindex or authentication
Disallow manages crawling. It does not protect a private document or reliably remove an already known URL from search.
- For indexing exclusion, let the crawler retrieve the noindex instruction.
- For private content, require authentication.
- Keep resources needed for public page rendering accessible when checking JavaScript SEO.
How to check robots.txt availability and file size
A correct CMS setting is not enough if the public endpoint returns an error or stale content. Check the response before interpreting a crawler’s behavior.
- Confirm a text file rather than an HTML login or error page.
- Google ignores content beyond 500 KiB; consolidate oversized rule lists.
- Investigate 5xx and 429 responses. A robots.txt 404 is not a crawl prohibition for Google.
How to verify the result
Test https://example.com/offers/public/bike and https://example.com/offers/draft/bike against the example. The public path should be allowed and the draft blocked. Account for the crawler’s cached version after publication.
Primary documentation developers.google.com ↗
Frequently asked questions
Will Disallow remove a URL from Google?
Disallow restricts crawling, not necessarily the appearance of a URL in results. A blocked crawler may never see a page-level noindex. For a public page you want excluded, allow retrieval and use the appropriate indexing directive. Protect private content with authentication; robots.txt is not access control.
Does robots.txt belong in an individual page folder?
Crawlers look for /robots.txt at the root of the relevant host. Subdomains do not automatically inherit each other’s rules. Check the file on the host that serves the URL being inspected.
Why is a robots.txt tester not sufficient on its own?
A tester evaluates the supplied rules for a chosen user agent and path. A CDN, firewall, login requirement or server response can independently affect access. Compare the result with an actual request and response.
When the task is complete
Check the effective rule for the exact path and user agent, then inspect the unauthenticated page response. A valid file is insufficient if its behavior contradicts the intended access policy.
Related SEO guides
Check a specific page.
Fukamo shows findings, evidence and the next step.
679 rules + SEO, AI readability, performance and accessibility measurements for free.