Can I allow search retrieval while restricting model training?
Define your policy before editing robots.txt. Verify current provider documentation instead of copying an old list of crawler names. CDN and firewall rules can also affect access.
What to do step by step
- Record the intended access policy.
- Verify current identifiers in provider documentation.
- Check effective rules for the specific path.
- Review network and firewall responses.
- Record changes and inspect relevant server logs.
A practical example
A publisher wants search discovery but wishes to restrict another use. A blanket User-agent: * block would affect more than the intended service, so identify the relevant controls first.
A common mistake
User-agent strings can be imitated. Robots.txt is a cooperative protocol, not authentication or proof that every visitor follows the rules.
Try the report before checking your site.
Explore the sample report, then enter your own URL to see findings, HTML evidence and a repair plan for your page.
Run your audit
fukamo.com/skSeptember 16, 2026
59/ 100
HTML response size · Fukamo 200 kB budget
- Evidence
- 314476 bytes of decoded HTML; this is not the transferred compressed size.
- Repair
- Reduce unnecessary HTML and inline data. The 200 kB threshold is a Fukamo budget, not a Google limit or compressed transfer size.
Inline JS size · Fukamo 64 KiB budget
- Evidence
- 4 blocks; 179621 bytes before transfer compression.
- Repair
- Reduce inline data exceeding the Fukamo 64 KiB budget. Compare an external cacheable file and measure the impact.
ImageObject · basic data
- Evidence
- 1 entity. Missing data ImageObject #1: contentUrl.
- Repair
- Check appropriate properties for this type. Checking a selected set does not replace complete validation or establish rich-result eligibility.
Illustrations have the context they need
- Evidence
- 1 of 1 figure element has no text figcaption.
- Repair
- Add a caption where the image needs explanation. Figcaption is optional and does not replace alt.
OAI-SearchBot and GPTBot serve different purposes
OpenAI documents separate controls for OAI-SearchBot search discovery and GPTBot training. A rule for one name does not express a complete policy for both purposes.
- Decide which public material may support search.
- Record training restrictions separately.
- Keep private accounts behind authentication.
Source OpenAI · crawler purposes ↗
Which Anthropic crawler should a rule apply to?
Anthropic distinguishes ClaudeBot, Claude-SearchBot and Claude-User. Keep the provider’s documentation with your policy so a future editor understands each rule.
- Map the identifier to its documented purpose.
- Define the host and path affected by the rule.
- Assign an owner and a review trigger.
Source Anthropic · crawlers ↗
Why HTTP 200 can hide a blocked crawler
A WAF challenge can return a successful status while replacing the article. Missing headings may therefore describe the challenge, not your page.
- Check final URL, body and redirect chain.
- Compare a public article and a restricted path.
- Verify crawler identity through the provider’s documented method.
How to verify changes to crawler access
Keep configuration verification separate from real crawler visits. An absent visit immediately after deployment does not show that a rule failed.
- Retrieve the published robots.txt.
- Check representative allowed and restricted URLs.
- Save configuration history and meaningful log responses.
How to verify the result
Test one allowed and one restricted path for the intended crawler. Save the effective robots.txt rule and server response; use provider verification guidance when interpreting visits in logs.
Primary documentation developers.openai.com ↗
Frequently asked questions
Can I allow search retrieval while restricting model training?
Providers may expose different identifiers and controls for different purposes. Base decisions on the specific service’s current documentation. Blocking every crawler may also block search access you intended to allow. A policy for one provider does not automatically apply to another.
When the task is complete
Record the purpose, identifier source, and verification date. Compare robots.txt with CDN controls and actual responses. A claimed user-agent name alone does not authenticate a visitor.
Related SEO guides
Check a specific page.
Fukamo shows findings, evidence and the next step.
836 rules + SEO, AI readability, performance and accessibility measurements for free.