AI crawl diagnostic
This test emulates HTTP access with different User-Agents. It does not perfectly reproduce a real visit: source IP, CDN, WAF and anti-bot rules can change real behaviour. A user-agent name proves nothing; only the origin IP does.
Result
Build your robots.txt
robots.txt is a request, not a barrier: only IP filtering stops bots that ignore it. AI crawlers · FAQ