65%
robots.txt
56,191 / 85,917
68%
Sitemap in robots.txt
38,099 / 56,191
3.7%
robots.txt blocks everything
2,077 / 56,191
68%
sitemap.xml
46,535 / 68,769
By market
| Metric | Global | United Kingdom | Netherlands | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|
| robots.txt | 65% | 80% | 84% | 70% | 72% | 43% | 61% |
| Sitemap in robots.txt | 68% | 86% | 87% | 69% | 70% | 64% | 73% |
| robots.txt blocks everything | 3.7% | 53% | 49% | 1.9% | 2.3% | 3.2% | 2.7% |
| sitemap.xml | 68% | 77% | 77% | 72% | 72% | 58% | 67% |
| Blocks an AI crawler | 10% | 6% | 6.4% | 11% | 11% | 3.9% | 7% |
Medians and spread
| Metric | P10 | P25 | Median | P75 | P90 | Sites |
|---|---|---|---|---|---|---|
| robots.txt size | 61 B | 125 B | 420 B | 1.5 KB | 3.5 KB | 56,191 |
| User-agent groups | 1 | 1 | 1 | 3 | 15 | 56,191 |
SEO bots blocked
- 1MJ12bot9%5,075
- 2AhrefsBot8.6%4,808
- 3SemrushBot8.2%4,611
- 4DotBot8.1%4,528
- 5BLEXBot7.2%4,039
- 6DataForSeoBot6.7%3,752
- 7Barkrowler5.3%2,953
- 8rogerbot4.7%2,664
- 9serpstatbot4.5%2,531
- 10Screaming Frog SEO Spider4.1%2,303
Share of 56,191 sites in the sample.
Search bots blocked
- 1PetalBot11%6,271
- 2Baiduspider9.2%5,194
- 3YandexBot6.1%3,450
- 4Applebot4.7%2,655
- 5SeznamBot4.6%2,589
- 6Sogou4.2%2,359
- 7Naverbot4.2%2,353
- 8Yeti4.2%2,334
- 9DuckDuckBot3.3%1,877
- 10Bingbot3.2%1,797
Share of 56,191 sites in the sample.
Our crawler: 68,769 sites · Global · updated Oct 23, 2026 · Methodology
Found mistakes on your site?
We can fix them in one sprint. Reply within one business day.