robots.txt Generator.
Generate a correct, SEO-safe robots.txt file for your website in seconds. Pick what to allow or block for Google, Bing, and other crawlers — no coding needed.
Generated entirely in your browser — nothing is sent to our servers.
What is robots.txt and why it matters for Indian business websites
Every time Google or Bing crawls the web, its bots visit millions of websites looking for pages to index. When a crawler arrives at your site, the very first file it requests is robots.txt — a plain text file sitting at the root of your domain. This file tells the crawler which pages it may visit and which it should skip.
For an Indian small business website, a correct robots.txt file matters for two reasons. First, it protects pages you genuinely do not want indexed — your WordPress admin panel, your internal search results, your cart and checkout pages. Google has no business putting your WooCommerce checkout page in search results; it will confuse customers and waste your crawl budget. Second, it ensures the important pages — your homepage, your services, your menu or catalogue — get crawled fully and frequently.
A missing or incorrect robots.txt does not stop Google from crawling your site. Google will simply crawl everything, including pages that should never appear in search results. A malformed file can accidentally block all crawlers from your entire site, which is one of the most damaging SEO mistakes a small business can make. This tool generates a syntactically correct file based on your actual choices, so you do not have to guess at the format.
How to upload robots.txt to your website
Generate your file here. Fill in the form above, tick the rules that apply to your site, and click Generate robots.txt. The file will appear in the output box.
Copy or download the text. Use the Copy button to copy it to your clipboard, or click Download robots.txt to save the file directly to your computer as robots.txt.
Open your hosting file manager or cPanel. Log in to your web hosting account. Most Indian hosts — Hostinger, BigRock, GoDaddy India, Bluehost — provide a cPanel interface with a built-in File Manager.
Navigate to public_html. Inside File Manager, open the public_html folder (sometimes called www or httpdocs). This is your website root — the same folder where index.html or index.php lives.
Create or edit robots.txt. If a robots.txt file already exists, right-click it and choose Edit. If it does not exist, click New File, name it exactly robots.txt (all lowercase), and open it for editing.
Paste your content and save. Paste the generated content, click Save, and you are done. Visit yoursite.com/robots.txt in a browser to confirm the file is live and contains the correct text.
Common robots.txt mistakes to avoid
- Accidentally blocking all crawlers by writing
Disallow: /underUser-agent: *without any Allow rules. This tells every bot to skip your entire site — your traffic will fall to zero within days as Google de-indexes your pages. - Wrong Disallow syntax. The path must start with a forward slash.
Disallow: wp-adminwill not work; it must beDisallow: /wp-admin/. Even one missing slash means the rule is ignored. - Missing the Sitemap declaration. Adding your sitemap URL in robots.txt is one of the cheapest ways to help Google discover all your pages. It takes one line and costs nothing.
- Using robots.txt as a security measure. It provides zero protection against malicious actors. Admin panels and sensitive data must be protected by authentication — not by a text file that anyone can read at yoursite.com/robots.txt.
- Putting multiple robots.txt files in subdirectories. Only the file at the root matters. Crawlers do not look inside subfolders for additional robots.txt files.
- Forgetting to test after upload. Always verify the file is live by visiting yoursite.com/robots.txt directly, and then use Google Search Console's robots.txt Tester to confirm your intended pages are allowed.
Frequently asked questions
What is a robots.txt file?
A robots.txt file is a plain text file placed at the root of your website (e.g., yoursite.com/robots.txt) that tells search engine crawlers which pages or sections they are allowed or not allowed to visit. It follows the Robots Exclusion Standard, a protocol respected by Google, Bing, and virtually every major crawler.
Will robots.txt stop hackers from accessing my private pages?
No. robots.txt is a courtesy instruction to well-behaved bots, not a security measure. Any malicious actor or non-compliant bot can simply ignore it. If you have truly private content — admin panels, customer data, invoices — protect it with a password, authentication, or server-level access control. Do not rely on robots.txt for security.
Where do I put the robots.txt file on my website?
It must go in the root directory of your domain — the same folder where your homepage lives. On a cPanel host, that is usually public_html. The file must be accessible at exactly https://yoursite.com/robots.txt. If it lives anywhere else, crawlers will not find it.
Can I have multiple robots.txt files for different subdomains?
Yes, but each subdomain needs its own separate robots.txt file. The file at www.yoursite.com/robots.txt only applies to the www subdomain. If you also run blog.yoursite.com, you need a separate robots.txt at blog.yoursite.com/robots.txt. One file cannot cover multiple domains or subdomains.
Does Google always obey my robots.txt instructions?
Google respects robots.txt for crawling — it will not fetch pages you have disallowed. However, Google may still index a disallowed page if other sites link to it. If you want a page completely removed from search results, use the noindex meta tag or the X-Robots-Tag HTTP header instead of robots.txt.
How do I test my robots.txt is working?
The easiest way is to use Google Search Console. Go to Settings, then Crawl stats, and you will see which pages Googlebot is crawling. You can also use the robots.txt Tester tool inside GSC to check whether specific URLs are allowed or blocked by your current file.
Your entire online presence, on one subscription.
Website, free domain, Google Business and SEO autopilot from ₹249/month. No plugins, no hosting headaches.