Robots.txt vs Sitemap: What Each One Actually Does

Robots.txt vs Sitemap: What Each One Actually Does

Block a page in robots.txt and Google can still show it in search results — often with the line “No information is available for this page.” The file you used to hide it is the reason Google cannot read the page and cannot see your noindex tag.

Two files, two jobs

robots.txt sitemap.xml
Location site root: /robots.txt any URL, often /sitemap.xml
Format plain text XML
Purpose which URLs crawlers may fetch which URLs exist and when they changed
Effect on indexing indirect none guaranteed
Obeyed by compliant crawlers search engines you submit to

robots.txt is a request, not a lock. Search engines honor it; scrapers and some SEO tools ignore it. It governs crawling — fetching — not indexing. A blocked URL can still be indexed if other pages link to it.

A sitemap is a list of URLs you want discovered. It helps a new site get crawled sooner and tells search engines which pages matter. It does not force indexing; thin or duplicate pages can be listed and still ignored.

A minimal pair

The robots.txt generator builds the file in your browser — no upload, no server. Choosing “Allowed” for all robots and adding two paths produces:

# robots.txt generated by ToolAsk

User-agent: *
Disallow: /admin/
Disallow: /temp/

Sitemap: https://www.example.com/sitemap.xml

The Sitemap line is just discovery: it tells crawlers where the XML lives. The XML itself looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://www.example.com/</loc></url>
  <url><loc>https://www.example.com/guides/</loc></url>
</urlset>

The generator also offers Crawl-delay and per-bot allow/block switches. Note two limits: per-bot rules are all-or-nothing for the whole site, and Google ignores Crawl-delay entirely.

Mistakes that bite

  • Blocking CSS and JS stops Google from rendering the page properly.
  • Using robots.txt to keep a page out of the index. Use a noindex meta tag and allow crawling so the bot can read it.
  • Listing disallowed URLs in the sitemap. Contradictory signals waste crawl budget.
  • Assuming a sitemap guarantees indexing. It does not.

Use robots.txt for crawl control and private paths, and a sitemap for public pages you want found. If a page must stay out of search results, noindex on the page is the tool that works.