User-agent: * Crawl-delay: 3 Disallow: /admin/ # For new training only User-agent: GPTBot Disallow: / # Not for training, only for user requests User-agent: ChatGPT-User Disallow: / # Marker for disabling Bard and Vertex AI User-agent: Google-Extended Disallow: / # Speech synthesis only? User-agent: FacebookBot Disallow: / # Multi-purpose, commercial uses; including LLMs User-agent: Omgilibot Disallow: /