Robots.txt is a plain text file placed at the root of a domain that tells search engine crawlers which URLs or folders they should not request. It uses simple directives like User-agent and Disallow, and it lives at a fixed address such as https://example.com/robots.txt. It manages crawl behavior, not rankings, and it does not remove pages that are already indexed.
For a local service business, robots.txt matters most when a site has thousands of thin pages, staging environments, or internal search results that shouldn’t waste crawl budget. Getting it wrong is one of the fastest ways to accidentally hide a site from Google. It works alongside crawlability settings and the sitemap, which tells crawlers what to find rather than what to avoid.
Key takeaway
Robots.txt blocks crawling, not indexing. A URL disallowed in robots.txt can still appear in Google’s index with no description if other pages link to it, so use noindex tags or password protection for anything you truly want out of search results.
How does robots.txt work?
Robots.txt works by giving crawlers a set of rules before they fetch a page, using User-agent to name a crawler and Disallow or Allow to set the path rules. Search engines that follow the robots exclusion protocol check this file first, then decide which URLs to request.
Google, Bing, and most other major crawlers respect the file, but it is a voluntary standard. Malicious bots and scrapers often ignore it entirely, so it is not a security tool. A typical rule set looks like User-agent: * followed by Disallow: /admin/, which blocks every crawler from a folder.
Why does robots.txt matter, and what doesn’t it do?
Robots.txt matters because it directs limited crawl attention away from low-value pages like filtered product views, cart pages, or duplicate parameter URLs, so crawlers spend more time on pages you want ranked. It doesn’t control indexing directly, doesn’t pass or block link juice, and doesn’t fix duplicate content on its own.
People often assume blocking a page in robots.txt guarantees it disappears from Google. It doesn’t. If external links point to a disallowed URL, Google can still index the URL with just the address showing, no title or snippet. Use a noindex meta tag or remove the page’s crawl block if you need it fully out of search results.
What are the main robots.txt directives?
The main robots.txt directives are User-agent, Disallow, Allow, and Sitemap, each controlling a different part of crawler behavior. A few sites also use Crawl-delay, though Google ignores it.
| Directive | Purpose |
| User-agent | Names the crawler the rules apply to, or * for all |
| Disallow | Blocks crawling of a path or folder |
| Allow | Overrides a Disallow for a specific subpath |
| Sitemap | Points crawlers to the XML sitemap location |
Robots.txt vs meta robots tag: what’s the difference?
Robots.txt sits at the domain level and stops a crawler from requesting a page at all, while the meta robots tag sits inside a page’s HTML and controls indexing after the page has already been crawled. Use robots.txt to save crawl budget and the meta tag to keep specific pages out of the index.
| Feature | Robots.txt | Meta robots tag |
| Location | Root file, applies site-wide or by folder | Inside each page’s head |
| Controls | Crawling | Indexing |
| Can hide an already-indexed page | No | Yes, with noindex |
What do good and bad robots.txt setups look like?
A good robots.txt blocks only genuinely low-value paths and always includes a sitemap reference, while a bad one accidentally blocks CSS, JavaScript, or entire content sections. Google needs to render pages fully to judge quality, so blocking assets it needs can hurt rankings.
- Disallow internal search result pages and admin folders, and list the sitemap URL at the bottom of the file.
- Disallow: / left over from a staging site, which blocks the entire domain from every crawler.
How do you check a robots.txt file?
Check a robots.txt file by opening yourdomain.com/robots.txt directly in a browser, then confirming the rules in Google Search Console under Settings > Crawling, which shows the last fetched version and any parsing errors.
For a specific URL, use the URL Inspection tool to see if Google reports it as blocked by robots.txt. Roofing and HVAC sites that run seasonal landing pages should recheck this file after any site migration, since developers sometimes forget to remove staging block rules before launch.
Example A Tampa moving company relaunched its site and noticed organic traffic to service-area pages dropped within days. Checking robots.txt showed Disallow: / still active from the staging server. Removing the line and resubmitting the sitemap in Search Console restored crawling within a couple of weeks.
What mistakes and myths surround robots.txt?
Costly mistake Blocking a folder in robots.txt to remove it from Google’s index. This only stops crawling; the URLs can remain indexed indefinitely without a noindex tag or removal request.
Another myth is that robots.txt improves rankings by itself. It only redirects crawl attention; it doesn’t add ranking signals. Some site owners also believe every bot obeys the file, but scrapers and bad actors frequently disregard it, so sensitive data should never rely on robots.txt for protection.
How do you improve your robots.txt file?
- Audit current rules. Open the live file and list every Disallow line, checking each against pages you actually want indexed.
- Remove overly broad blocks. Replace wildcard folder blocks with specific paths so you don’t catch pages you need crawled.
- Allow CSS and JS. Confirm rendering assets aren’t blocked, since Google needs them to evaluate Core Web Vitals and page layout.
- Add the sitemap line. Include the full sitemap URL so crawlers can find new pages faster.
- Test and monitor. Recheck the file in Search Console after every site change or migration.
