66%
robots.txt
56,487 / 86,052
68%
Sitemap in robots.txt
38,293 / 56,487
3.7%
robots.txt blocks everything
2,077 / 56,487
68%
sitemap.xml
46,708 / 68,803
By market
| Metric | Global | United Kingdom | Netherlands | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|
| robots.txt | 66% | 82% | 85% | 71% | 71% | 43% | 61% |
| Sitemap in robots.txt | 68% | 88% | 87% | 69% | 71% | 64% | 72% |
| robots.txt blocks everything | 3.7% | 58% | 52% | 3% | 3.9% | 3% | 2.7% |
| sitemap.xml | 68% | 78% | 78% | 72% | 72% | 58% | 67% |
| Blocks an AI crawler | 11% | 5.8% | 6.5% | 11% | 11% | 3.9% | 6.6% |
Medians and spread
| Metric | P10 | P25 | Median | P75 | P90 | Sites |
|---|---|---|---|---|---|---|
| robots.txt size | 61 B | 126 B | 424 B | 1.5 KB | 3.5 KB | 56,487 |
| User-agent groups | 1 | 1 | 1 | 3 | 15 | 56,487 |
SEO bots blocked
- 1MJ12bot9%5,063
- 2AhrefsBot8.6%4,831
- 3SemrushBot8.1%4,594
- 4DotBot8%4,546
- 5BLEXBot7.2%4,053
- 6DataForSeoBot6.7%3,764
- 7Barkrowler5.3%2,974
- 8rogerbot4.7%2,679
- 9serpstatbot4.5%2,523
- 10Screaming Frog SEO Spider4.1%2,304
Share of 56,487 sites in the sample.
Search bots blocked
- 1PetalBot11%6,428
- 2Baiduspider9.5%5,341
- 3YandexBot6.1%3,462
- 4Applebot4.7%2,675
- 5SeznamBot4.6%2,581
- 6Sogou4.2%2,358
- 7Naverbot4.2%2,356
- 8Yeti4.1%2,337
- 9DuckDuckBot3.3%1,870
- 10Bingbot3.2%1,786
Share of 56,487 sites in the sample.
Our crawler: 68,803 sites · Global · updated Nov 6, 2026 · Methodology
Found mistakes on your site?
We can fix them in one sprint. Reply within one business day.