Robots.txt Explained

Robots.txt Explained

Every website has a doorman for search engines, and it’s a three-line file called robots.txt. Get it wrong and your whole site can vanish from Google — silently.

What it does

robots.txt sits at your site’s root and tells crawlers which paths they may visit:

User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
  • User-agent: * — applies to all crawlers (or name one specifically).
  • Disallow: — paths the crawler must skip.
  • Sitemap: — where your sitemap lives, so crawlers find all your pages.

The rules that bite

  • It’s a request, not a lock. Respectable crawlers obey; scrapers ignore it entirely. Never use robots.txt to hide anything sensitive — blocked pages can still appear in search results if they’re linked elsewhere.
  • One typo hides everything. Disallow: / means “crawl nothing.” A stray slash in the wrong place is the classic “my site disappeared from Google” cause.
  • Path matching is prefix-based. Disallow: /admin blocks /administrator too.

The healthy baseline

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Sitemap: https://yoursite.com/sitemap-index.xml

Block the things with no search value — admin areas, internal APIs, search result pages — and allow everything else.

Generating one

The robots.txt generator builds the file from simple inputs: pick your disallowed paths, add your sitemap URL, and copy the result to your site root. Ten minutes of setup, and your site stops both wasting crawl budget and leaking admin URLs.