robots.txt reachable.
The following AI crawlers are currently blocked by robots.txt: Bytespider. This reduces visibility in AI search and agent discovery.
- Edit the root robots.txt file and add Allow rules for the AI crawlers below.
- If you need to protect specific paths (e.g. admin panels), use precise Disallow rules inside the wildcard group.
- Add a Sitemap reference: Sitemap: https://www.fool.com/sitemap.xml.
- Re-run the analysis after deployment to confirm the status changes to Allowed.
txt
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Bytespider
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: KimiBot
Allow: /
User-agent: *
Allow: /