What is robots.txt?
Robots.txt is a plain-text file at your site root that instructs search engine crawlers which URLs they may or may not crawl. It is the site-level gatekeeper of crawl access, separate from the page-level robots meta tag.
Why robots.txt matters
A correct robots.txt protects crawl budget and keeps irrelevant sections (admin, staging, parameter-heavy pages) out of the crawl. A wrong one can accidentally block entire sections of your site from being indexed — a common and damaging mistake.
Best practices
- Never block critical pages, CSS or JavaScript.
- Use robots.txt to block low-value paths, not to hide content.
- Reference your XML sitemap in the file.
- Test rules with Google’s robots.txt tester.
- Remember: robots.txt blocks crawling — use noindex to block indexing.
Robots.txt and GEO
AI crawlers often respect robots.txt too. Overly aggressive rules can exclude your content from AI training and citation pipelines.
Related terms
How Social SEO Team can help
We configure robots.txt correctly and safely. Explore our technical SEO services.