Google has announced plans to support a shared AI training control in robots.txt. The proposal is designed to let website owners express a broad training preference across participating AI providers instead of maintaining a separate rule for every company.
The announcement matters to publishers, ecommerce teams, membership sites, and organisations with valuable original content. It is also easy to misread. The shared signal is planned for 2027; it is not a new Google Search ranking factor, a privacy control, or a reason to change robots.txt immediately.
What Google announced
Google’s official announcement says the company plans to support work from the AIPREF group at the Internet Engineering Task Force. The proposal uses a shared policy token in robots.txt so participating AI providers can interpret a site-wide training preference.
Google says that when a site uses the shared token to opt out and does not explicitly address Google-Extended, it plans to interpret the choice as if Google-Extended were disallowed. Existing provider-specific controls are expected to continue alongside the shared signal. This gives publishers a possible broad default with more specific exceptions where needed.
The important timing: this is planned, not live
Google aims to support the new mechanism with other AI providers in 2027. Its current crawler documentation identifies the planned token as user-agent: __usage_ai-training, but website owners should not treat the announcement as a completed standard or deploy an unverified rule today.
The practical action now is preparation. Record the decision makers, review existing crawler rules, identify which content types need discussion, and watch the official documentation for final syntax and support dates. A rushed robots.txt edit can affect more than the intended service when groups, wildcards, or directives are combined incorrectly.
| Control | Current role | What to do now |
|---|---|---|
| Googlebot rules | Manage access for Google Search crawling | Do not change them for AI-training policy |
| Google-Extended | Controls specified Gemini training and grounding uses | Review the existing choice and document why |
| Shared AIPREF token | Planned cross-provider training preference | Wait for confirmed support and final guidance |
| noindex or authentication | Controls search indexing or private access | Use the control that matches the real goal |
How the shared signal differs from Google-Extended
Google-Extended is an existing control. Google’s crawler documentation says it manages whether content Google has crawled may be used for training future Gemini models and for specified grounding experiences. It is a control token rather than a separate crawler making its own HTTP requests.
Google also states that Google-Extended does not affect inclusion in Google Search and is not used as a Search ranking signal. That distinction is essential. Changing a Google-Extended preference is not the same as blocking Googlebot. A considered SEO programme should keep crawling, indexing, search previews, AI training, and product-specific grounding decisions separate.
The shared policy is intended to work across participating providers. Provider-specific rules can still express a different choice for an individual service. A publisher may eventually choose a broad training preference and then explicitly allow or disallow Google-Extended, depending on its content, distribution strategy, and appetite for Gemini grounding traffic.
Decide by content type, not by fear
A single organisation can have several valid priorities. Public educational content may benefit from wider discovery. Premium reports, licensed archives, user submissions, product data, or original research may need different consideration. List the content groups, who owns them, contractual restrictions, commercial value, and acceptable uses before deciding on a site-wide policy.
Include editorial, legal, privacy, product, engineering, and search stakeholders. The decision should explain the intended outcome and the trade-off. A blanket rule without an inventory can create internal conflict later, especially when teams assume that an AI-training preference also removes content from Search.
Do not use robots.txt as a security boundary
Robots.txt communicates preferences to compliant crawlers; it does not protect confidential information. Private dashboards, customer data, paid downloads, staging environments, and unpublished files require access controls such as authentication and appropriate server permissions.
Likewise, a robots.txt disallow rule is not the correct way to guarantee removal from search results. Match the tool to the objective. If the goal is crawl management, indexing control, snippet control, or AI-training preference, document it separately and test the relevant implementation.
Audit the current file before planning a change
Retrieve the live robots.txt file and compare it with the version managed by the CMS, server, security layer, or deployment pipeline. Look for duplicated user-agent groups, broad wildcards, conflicting allow and disallow rules, environment-specific directives, and changes that are generated automatically.
A technical SEO audit can establish whether important pages or resources are already blocked. Keep a dated copy of the current file, name the owner, and document how a rollback would work. Never edit the live file from memory.
Build a safe 2027 implementation checklist
When official support is confirmed, review the final documentation before writing the rule. Confirm which providers participate, what the token covers, how specific provider rules override the shared preference, and whether the syntax has changed from the draft. Test on a non-production environment where possible.
After release, fetch the live file from several network locations, verify the HTTP status and content type, and confirm that critical Search directives remain intact. Add the change to the website log and monitor crawl, indexation, and traffic for unrelated problems. The recent guide to combined Search Console filters can help teams build consistent monitoring segments without confusing normal market changes with a technical release.
Keep the decision connected to content strategy
An AI-training preference does not replace a content strategy. Useful, original pages still need clear ownership, source checks, maintenance, internal links, and a reason to exist. Teams should avoid publishing near-duplicate pages merely because AI discovery is changing.
The governance questions in SEO Shout’s AI SEO agency checklist are also relevant here: who approves the tools, who owns the data, who validates changes, and what evidence will be reviewed? If you need help auditing crawler controls and building a practical plan, contact SEO Shout.
FAQs
Is the shared AI training token active today?
No. Google announced planned support for 2027. Wait for confirmed provider support and final official implementation guidance before changing a live robots.txt file.
Does blocking Google-Extended remove a site from Search?
No. Google says Google-Extended does not affect inclusion in Google Search and is not used as a Search ranking signal.
Should existing AI crawler rules be removed?
No. Google expects provider-specific controls to continue alongside the shared signal. Review each rule and wait for final compatibility guidance.
Can robots.txt protect private content?
No. Robots.txt is a preference file for compliant crawlers, not access control. Protect private material with authentication and suitable server permissions.
Who should approve the robots.txt change?
SEO, engineering, content ownership, legal or policy stakeholders, and an accountable business owner should approve the documented preference and rollout plan.