Every website has a doorman for search engines, and it’s a three-line file called robots.txt. Get it wrong and your whole site can vanish from Google — silently.
What it does
robots.txt sits at your site’s root and tells crawlers which paths they may visit:
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
User-agent: *— applies to all crawlers (or name one specifically).Disallow:— paths the crawler must skip.Sitemap:— where your sitemap lives, so crawlers find all your pages.
The rules that bite
- It’s a request, not a lock. Respectable crawlers obey; scrapers ignore it entirely. Never use robots.txt to hide anything sensitive — blocked pages can still appear in search results if they’re linked elsewhere.
- One typo hides everything.
Disallow: /means “crawl nothing.” A stray slash in the wrong place is the classic “my site disappeared from Google” cause. - Path matching is prefix-based.
Disallow: /adminblocks/administratortoo.
The healthy baseline
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Sitemap: https://yoursite.com/sitemap-index.xml
Block the things with no search value — admin areas, internal APIs, search result pages — and allow everything else.
Generating one
The robots.txt generator builds the file from simple inputs: pick your disallowed paths, add your sitemap URL, and copy the result to your site root. Ten minutes of setup, and your site stops both wasting crawl budget and leaking admin URLs.