A robots.txt file is a plain text file at the root of your website that tells crawlers which parts of the site they may fetch. Every site should have one, most small business sites in India either do not or have one that quietly blocks something important, and in 2026 the file has picked up a second job: deciding whether AI crawlers from OpenAI, Anthropic, Perplexity, and Google's Gemini get to read your pages. This guide covers the syntax you actually need, the mistakes that hurt rankings, what to do about AI bots, what WordPress, Shopify, Wix, and Neweb generate by default, how to test the file in Search Console, and five templates you can copy as they are.
If you would rather answer a few questions and get a finished file, the free robots.txt generator produces one you can paste into your site. This post explains what it is generating and why.
The syntax, in full
A robots.txt file is a set of groups. Each group names one or more crawlers and lists rules for them.
User-agent: *
Disallow: /admin/
Allow: /admin/public/
Sitemap: https://www.example.in/sitemap.xml
- User-agent names the crawler the rules apply to.
*means every crawler that does not have its own group. Googlebot, Bingbot, GPTBot, and ClaudeBot are all user-agents. - Disallow gives a path prefix the crawler must not fetch.
Disallow: /admin/blocks everything under /admin/. An empty Disallow means nothing is blocked. - Allow carves an exception out of a Disallow. Google and Bing follow the most specific rule (longest path wins).
- Sitemap points to your XML sitemap and can appear anywhere in the file. It is the single most useful line in the file for a small site.
- Wildcards:
matches any sequence of characters,$anchors the end of the URL.Disallow: /?blocks every URL containing a question mark;Disallow: /*.pdf$blocks PDFs. - Crawl-delay is ignored by Google. Bing honours it. Leave it out unless a specific bot is hammering your server.
Rules are case-sensitive, the file must be at the root (https://yourdomain.in/robots.txt, not in a subfolder), it must be served as plain text, and Google reads only the first 500 KB. Blank lines separate groups; comments start with #.
Five mistakes that cost real traffic
- *
Disallow: /underUser-agent:.** This blocks the entire site. It is the correct setting for a staging site and a disaster on a live one. Developers copy staging files to production more often than anyone admits; check the live file the day you launch. - Blocking CSS, JavaScript, or image folders. Google renders pages before ranking them. Block /assets/ or /wp-content/ and Google sees an unstyled page it may judge as poor mobile experience.
- Using robots.txt to hide a page from search results. It does not work that way. A blocked page can still be indexed from links pointing to it, and it then shows in results with no description. To keep a page out of Google, let it be crawled and add a
noindexmeta tag. Google stopped honouringnoindexinside robots.txt in 2019. - Blocking URLs with parameters wholesale.
Disallow: /*?is common advice, but if your product filters, search results, or pagination live on parameters, you have just hidden a chunk of the site. Block the specific parameters instead. - A typo in the sitemap URL or a sitemap line pointing to the old domain. Search Console reports "couldn't fetch" and nobody looks for a month.
A sixth, more subtle one: a group for a specific bot that omits rules the default group has. Once a bot matches its own group, it ignores User-agent: * entirely. If you add a User-agent: GPTBot group, repeat every Disallow from the default group inside it.
AI crawlers: allow or block?
Each AI company now runs one or more named crawlers, and each honours robots.txt. The ones you will see in your server logs as of September 2026:
- GPTBot (OpenAI, used for model training), OAI-SearchBot (powers ChatGPT search results and citations), and ChatGPT-User (fetches a page when a user asks ChatGPT about it).
- ClaudeBot (Anthropic, training and indexing) and Claude-User or Claude-SearchBot for user-initiated fetches.
- PerplexityBot and Perplexity-User (Perplexity's index and on-demand fetches).
- Google-Extended, which controls whether your content trains Gemini. It does not affect Google Search or AI Overviews; those follow Googlebot's rules.
- Applebot-Extended, CCBot (Common Crawl), Bytespider (ByteDance), Amazonbot, and meta-externalagent.
For a small business site the honest answer is: allow them. Your pages exist so that a customer searching "best CA in Indore" or "bakery near Koramangala" finds you, and a growing share of those questions are now being asked inside ChatGPT, Perplexity, and Google's AI Mode. Blocking the search-side bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) removes you from those answers, which is the opposite of what a business website is for.
The reasonable middle ground, if you publish original content you do not want used for training, is to block the training-only bots (GPTBot, Google-Extended, CCBot, Bytespider, Applebot-Extended) and allow the search and user-fetch bots. Template 4 below does exactly that. Publishers with paid content or large original datasets have a stronger case for blocking; a salon website does not.
One practical note: Neweb sites ship with all major AI crawlers allowed and an llms.txt file describing the site, because our customers want to be found, and we have seen no downside.
Platform defaults
- WordPress serves a virtual robots.txt that disallows /wp-admin/ and allows /wp-admin/admin-ajax.php. It does not include a sitemap line unless an SEO plugin (Yoast, Rank Math, AIOSEO) adds one. Edit it from the plugin's tools section rather than uploading a file, or the two will conflict.
- Shopify generates a robots.txt that blocks /cart, /checkout, /orders, /account, internal search, and filtered collection URLs, and includes the sitemap. Since 2021 you can edit it through a
robots.txt.liquidtemplate in your theme. Do not remove the checkout and cart blocks. - Wix generates one automatically with the sitemap line and lets you edit it under SEO settings, Robots.txt editor. It blocks a few Wix system paths by default.
- Neweb generates robots.txt and the sitemap for every site, allows all search and AI crawlers, and blocks only the admin and API paths. Customers do not need to touch it; if you want custom rules, support adds them.
Whatever the platform, view the live file once at yourdomain.in/robots.txt after launch and once after every redesign.
Testing it in Search Console
Google retired the standalone robots.txt tester in 2023. The current checks:
- In Search Console, open Settings, then under Crawling click Open report next to robots.txt. It shows the last fetched version, when Google fetched it, and any parse errors or warnings line by line.
- Use URL Inspection on a specific page. The Crawl section reports "Crawl allowed? Yes/No" and names the blocking rule if there is one. Do this for your home page, one service page, and one blog post after any change.
- After editing, use the report's request-a-recrawl option so Google picks up the new file within hours instead of waiting a day.
For a broader on-page check, the meta tags analyzer reads a page's title, description, canonical, and robots meta tags together, which catches the case where robots.txt allows a page but a stray noindex tag hides it anyway.
Five copy-paste templates
Replace example.in with your domain. Regenerate the sitemap with the sitemap generator if you do not already have one.
1. Small business website, allow everything
User-agent: *
Allow: /
Sitemap: https://www.example.in/sitemap.xml
2. WordPress
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://www.example.in/sitemap_index.xml
3. Online store (Shopify-style paths)
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /orders
Disallow: /account
Disallow: /search
Disallow: /*?sort_by=
Disallow: /*?filter=
Sitemap: https://www.example.in/sitemap.xml
4. Allow search and AI answers, block AI training
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://www.example.in/sitemap.xml
OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Googlebot fall under the * group here and stay allowed.
5. Staging or under-construction site, block everything
User-agent: *
Disallow: /
Remove template 5 before launch. Set a calendar reminder if you have to.