Home / SEO Standards / AI_Crawlers

AI Crawler & LLM Standards

Last Edited: 2026-06-29 01:32

Guidelines regarding text-based summaries and experimental files targeted at large language models (LLMs) and artificial intelligence search crawlers.

llms.txt File

A proposed plaintext file served at /llms.txt to provide concise, LLM-friendly summaries of the website's pages.

• Input: The /llms.txt text file in the site root directory.
• Standards (Online Access Stance):
  1. De-prioritization: Not implemented. We do not consider this standard important for operational SEO.
  2. Out of Sync Risk: This file represents duplicate content summaries that can easily fall out of sync with the actual page edits, creating discrepancies between what AI models read and what users see.
  3. Abuse & Manipulation: The file can be easily abused or manipulated to display details that do not accurately reflect the real page content.
  4. Industry Hype: Acknowledge that a significant portion of LLM-specific search optimizations are currently driven by market hype and financial incentives rather than verified SEO value.

cats.txt File

An experimental check file used as a test case to demonstrate how easily AI crawlers and models can be manipulated into believing a custom standard is important.

• Input: The /cats.txt text file in the site root directory.
• Standards (Online Access Stance):
  1. Not Implemented: Ignored. This file was purely an experiment demonstrating prompt manipulation and AI vulnerability to artificial standards. It is not supported.