Skip to content
mistakes.show
791Kissues on 68,803 sites
  1. Home
  2. Data
  3. robots.txt
68,803 sitesGlobal

robots.txt rules & sitemaps worldwide: stats

How websites write robots.txt: sizes, sitemaps, blocked bots.

66%
robots.txt
56,487 / 86,052
68%
Sitemap in robots.txt
38,293 / 56,487
3.7%
robots.txt blocks everything
2,077 / 56,487
68%
sitemap.xml
46,708 / 68,803

By market

MetricGlobalUnited KingdomNetherlandsItalySpainJapanLatin America
robots.txt66%82%85%71%71%43%61%
Sitemap in robots.txt68%88%87%69%71%64%72%
robots.txt blocks everything3.7%58%52%3%3.9%3%2.7%
sitemap.xml68%78%78%72%72%58%67%
Blocks an AI crawler11%5.8%6.5%11%11%3.9%6.6%

Medians and spread

MetricP10P25MedianP75P90Sites
robots.txt size61 B126 B424 B1.5 KB3.5 KB56,487
User-agent groups11131556,487

SEO bots blocked

  1. 1MJ12bot9%5,063
  2. 2AhrefsBot8.6%4,831
  3. 3SemrushBot8.1%4,594
  4. 4DotBot8%4,546
  5. 5BLEXBot7.2%4,053
  6. 6DataForSeoBot6.7%3,764
  7. 7Barkrowler5.3%2,974
  8. 8rogerbot4.7%2,679
  9. 9serpstatbot4.5%2,523
  10. 10Screaming Frog SEO Spider4.1%2,304

Share of 56,487 sites in the sample.

Search bots blocked

  1. 1PetalBot11%6,428
  2. 2Baiduspider9.5%5,341
  3. 3YandexBot6.1%3,462
  4. 4Applebot4.7%2,675
  5. 5SeznamBot4.6%2,581
  6. 6Sogou4.2%2,358
  7. 7Naverbot4.2%2,356
  8. 8Yeti4.1%2,337
  9. 9DuckDuckBot3.3%1,870
  10. 10Bingbot3.2%1,786

Share of 56,487 sites in the sample.

Our crawler: 68,803 sites · Global · updated Nov 6, 2026 · Methodology

Found mistakes on your site?

We can fix them in one sprint. Reply within one business day.