Robots.txt is a plain text file placed at the root of a website, such as yourdomain.com/robots.txt, that tells search engine crawlers and other bots which parts of the site they are allowed to fetch. It uses simple Allow and Disallow rules per user agent and usually points to the XML sitemap.
Why it matters for an Indian small business
A single wrong line in robots.txt can remove a business from Google. Disallow: / blocks every crawler from every page, and it is a common leftover from a developer's staging setup that gets copied to the live site. Owners discover it months later when a customer says the site cannot be found. The file also controls access for AI crawlers from OpenAI, Anthropic and others, which matters if you want your business to be mentioned in AI answers or specifically do not.
Used correctly it keeps crawlers out of admin screens, search result pages and cart URLs that waste crawl budget, and it tells every bot where the sitemap lives.
How it works
Crawlers fetch robots.txt before anything else. The file has groups starting with User-agent (a bot name or * for all), followed by Disallow and Allow lines with URL path prefixes, and a Sitemap line with the absolute sitemap URL. Rules apply to crawling, not indexing: a blocked page can still appear in results if other sites link to it, just without a description. To keep a page out of Google, use a noindex meta tag on a page that crawlers can fetch.
Googlebot ignores crawl-delay; wildcards and the dollar sign are supported for pattern matching. The file must be at the root of the exact host, so www and non-www each need one, or redirects must point to the canonical host.
Example: a Raipur steel fabricator
Chhattisgarh Steel Works launches a new site for its warehouse sheds and pre-engineered buildings. Traffic stays at zero for two months. The owner types the domain followed by /robots.txt and sees User-agent: * and Disallow: /, copied from the agency's staging server. Changing it to allow all crawlers, blocking only /admin/ and /cart/, and adding the Sitemap line fixes discovery; the site is indexed within ten days. The owner also adds explicit rules allowing the main AI crawlers, and within a quarter the company is named in AI answers for PEB manufacturer Raipur.
Common mistakes
- Disallow: / left over from staging, blocking the whole site.
- Using robots.txt to hide pages that should have a noindex tag instead.
- Blocking CSS and JavaScript folders, which prevents Google from rendering the page properly.
- Forgetting the Sitemap line.
- Blocking AI crawlers by default and then wondering why the business never appears in AI answers.
How Neweb handles it
Neweb serves a correct robots.txt for every site that allows search and major AI crawlers, blocks only private paths, declares the sitemap and is checked on every publish so a staging rule can never reach the live domain.
Robots.txt questions
Where is robots.txt located?
At the root of the domain, for example https://yourdomain.com/robots.txt. Any other location is ignored by crawlers.
Does robots.txt stop a page appearing in Google?
No. It stops crawling, not indexing. A blocked page can still be listed if linked elsewhere. Use a noindex meta tag to keep a page out of results.
What should a small business robots.txt contain?
Allow all crawlers, disallow admin and private paths, and include the sitemap URL. For most sites that is four or five lines.
Should I block AI crawlers?
Only if you do not want your content used or cited in AI answers. For most businesses being mentioned in AI Overviews and chat assistants is useful, so allow them.
How do I test my robots.txt?
Open it in a browser, then use the robots.txt report in Google Search Console, which shows errors and lets you test whether a URL is blocked.
Related terms.
Related pages.
Robots.txt generator
Build a safe, correct robots.txt with sitemap and AI crawler rules.
Robots.txt generator guide
Line-by-line explanation of the rules and what each bot honours.
Sitemap generator
Create the sitemap your robots.txt should point to.
Set up Google Search Console
Includes the robots.txt report and URL blocking tester.
Handle it for you? From ₹249/month.
Website, free domain, Google Business Profile, SEO autopilot and WhatsApp buttons, set up for your business by Neweb.