Skip to content
Neoval
Sign in

Blog

Your website does not need llms.txt to appear in Google AI. What does it need?

· 7 min read · by

An optional document sits beside the main path connecting crawler access, an indexed web page and an AI answer citation highlighted in lime.
An optional document sits beside the main path connecting crawler access, an indexed web page and an AI answer citation highlighted in lime.

The promise is appealing: add one file named llms.txt, summarize your website for language models, and become easier for AI search engines to understand and cite. It sounds like the AI equivalent of a sitemap. For Google Search, however, the answer is now unambiguous. Google says it does not usellms.txt, and that the file neither helps nor harms visibility or rankings in Search, including its generative AI features.

That does not make the file forbidden, and another service may choose to use it. It means a website owner should not confuse an optional convention with the technical foundations that determine whether a page can be found, processed and selected as a source. Before adding another AI-specific file, check the path that search and answer engines actually have to your content.

Four files and controls, four different jobs

The easiest way to avoid the confusion is to separate the jobs:

  • robots.txt manages crawler access. It tells supported crawlers which URLs they may request. Google stresses that this is not a reliable mechanism for removing a URL from search results; a blocked URL can still be discovered through links without its page content being crawled.
  • noindex controls index eligibility. If you do not want a public page indexed, use a supported meta tag or response header and allow the crawler to read it. A crawler cannot obey a page-level instruction it is prevented from fetching.
  • sitemap.xml supports discovery. It lists important URLs and information such as update dates or alternate-language versions. Google says a sitemap can improve discovery and crawling, but it does not guarantee that every listed page will be crawled or indexed.
  • llms.txt is an optional AI-oriented summary.Google Search does not treat it as a visibility or ranking signal. Maintaining one may make sense when a specific system you care about documents that it consumes the file. Creating it "for AI in general" is not a substitute for evidence that any target engine uses it.

ChatGPT has a different, documented control

OpenAI's current publisher guidance does not ask for an llms.txt file. It says public content should not block OAI-SearchBot if a publisher wants pages discovered, surfaced and cited in ChatGPT search. OpenAI also distinguishes that search crawler from GPTBot, which publishers can block when they want to exclude content from potential model training.

That distinction matters. Search visibility and model training are different uses, with different crawler controls. A blanket rule copied from a generic "block all AI bots" template can close the search path a business intended to keep open. The reverse is also true: allowing a crawler makes discovery possible; it does not guarantee a citation, recommendation or favorable description.

Five checks that matter more

  1. Can the intended crawler reach the page? Review the live robots.txt, CDN, firewall and bot-management rules. A file can look correct while an edge security product still returns a challenge or denial to the crawler.
  2. Does the page return a successful response?Google's minimum requirements include an HTTP 200 response and indexable content. Redirect loops, soft errors and intermittent server failures are more important than an extra summary file.
  3. Is the important explanation present as readable page text? Put the service, comparison, evidence and limitations in the page itself. Do not leave the substance only inside an image, an inaccessible interaction or a document that is harder to discover than the page promoting it.
  4. Can the page be discovered from the site? Link important pages through clear navigation and relevant internal links. Keep the sitemap accurate, especially on a new, large or frequently updated site. A page that exists only through a direct URL is a weak publication workflow.
  5. Does the page contribute something worth citing?Google's guidance favors unique, expert-led, non-commodity content over summaries anyone could generate. Publish first-hand comparisons, named methodology, original data, current prices or rules where appropriate, and sources a reader can verify.

Run a fifteen-minute reality check

Before commissioning an AI-specific file, choose one commercially important page and verify:

  1. It is public, returns 200 and is not unintentionally marked noindex.
  2. The crawler you want is not blocked by the site or its security layer.
  3. The page appears in the sitemap or is reachable through normal internal navigation.
  4. The answer to a buyer's real question is explicit, current and supported by evidence.
  5. The same buyer question is tested across the answer engines that matter to the business.

If any of the first four fails, fix it before polishing llms.txt. If the technical path works but the fifth check still shows no citation, inspect the answer: which source was used instead, what evidence did it provide, and does your page genuinely answer the same decision? That produces a content action, not a speculative file change.

Where llms.txt can still fit

There is nothing wrong with maintaining llms.txt for a documented consumer or an experiment with a clear owner and success measure. Keep it accurate, avoid presenting it as a universal standard, and do not let it drift into a second, contradictory version of the website. Its value should be demonstrated by the system that reads it, not assumed from the filename.

Neoval's free auditchecks a domain's technical controls and ChatGPT visibility against real buyer prompts. The paid audits and monitoring plans widen that observation across answer engines and turn findings into prioritized actions. They do not guarantee citations or expose private engine signals. The purpose is simpler: verify the path, inspect the answer and fix the highest-value gap first.

Sources