Robots.txt Guidance

Robots.txt guidance

Allow search engines and AI documentation crawlers to access public documentation, help center pages, pricing pages, blog articles, llms.txt, and llms-full.txt. Review crawler policies with legal and security teams before launch.