Skip to content
mistakes.show
793Kissues on 68,769 sites
mistakes.show
Get in touch
  1. Home
  2. Mistakes
  3. No robots.txt
SEOMinor

No robots.txt

Crawlers get a 404 and you can't point them to the sitemap.

28%of Global websites

By market

  1. Global28%
  2. United Kingdom25%
  3. Netherlands23%
  4. Italy24%
  5. Spain23%
  6. Japan52%
  7. Latin America33%

Examples

Popular sites where we saw this on the homepage (at crawl time).

google.comfacebook.comezviz7.comfbcdn.nethicloudcam.comlinkedin.comdzen.rugooglevideo.comazure.comoffice.comlive.comshiabank.com

How to fix

  1. 1Create /robots.txt with your sitemap URL.
  2. 2Use our robots.txt generator.
robots.txt generator

Related mistakes

Our crawler: 68,769 sites · Global · updated Oct 23, 2026 · Methodology

Found mistakes on your site?

We can fix them in one sprint. Reply within one business day.

Get in touchServices