Robots.txt for Small Business Websites: A Safe Starter Guide

robots.txt for small business websites is a small text file that tells search crawlers which URLs they may request. It controls crawling, not indexing, so it cannot hide a page from Google.

This guide shows where the file lives, a safe starter example, the mistakes that block a whole site and a 6-step check for a Ghaziabad shop. If you want hands-on help, see our technical SEO help in Vaishali from Micro Supple. Last updated: 1 November 2026 (IST).

What robots.txt does and does not do

A robots.txt file is a plain text file that tells search crawlers which URLs on your site they may request. It manages crawler traffic so your server is not overloaded. Google states that it is not a mechanism for keeping a web page out of Google.

Think of the file as a sign on the shop door, not a lock. Reputable crawlers such as Googlebot follow it, but other bots may ignore it. A blocked URL can still appear in Google results without a description when other pages link to it. So the file helps with crawl load, and it does not protect private pages. Pick the right tool for each job, as the table shows.

Tool	What it controls	Use it when
robots.txt file	Crawling	You want crawlers to skip low-value URLs, such as internal search pages.
noindex tag or header	Indexing	A page may be crawled but should stay out of Search.
Password protection	Access	The page holds private content.
Removing the page	Existence	The page should not exist anymore.

Google’s Introduction to robots.txt lists noindex, password protection and removal as the ways to keep content out of Search. It also warns that mixing crawling and indexing rules can cause conflicts, so use one clear rule per page.

Where the file lives and how it is built

The robots.txt file is a UTF-8 text file named robots.txt that sits at the root of your host. Each group starts with a User-agent line, followed by Disallow and Allow rules, and an optional Sitemap line points crawlers to your sitemap.

Location matters more than most owners expect. A file inside a subfolder does not work. Each subdomain can have its own file, and rules apply only to the exact protocol, host and port where you post them. Use a plain text editor, because a word processor can insert characters that confuse crawlers. Google describes these rules in its guide to creating a robots.txt file.

• Name the file robots.txt, and keep only one per site.
• Start every group with a User-agent line.
• Add Disallow lines for paths to skip and Allow lines for exceptions.
• Start each path with a slash, and remember that paths are case-sensitive.
• Use # for comments and * as a wildcard in most rules.
• Add a Sitemap line anywhere in the file if you have a sitemap.

Robots.txt for small business websites: a safe starter file

A starter file is a short robots.txt file that allows all crawlers and names your sitemap. It works because crawlers read it first and then follow your sitemap to your pages. Google’s own example uses the same pattern: one group for all crawlers, plus a sitemap line.

Most small shops need very little. If every page should appear in Search, the first example below is enough. Add a Disallow line only for a folder that adds no value in Search, such as a cart or an internal search page. The paths in the second example are illustrations, so replace them with your own folders before you publish anything.

# Example 1: allow everything, point to the sitemap
User-agent: *
Allow: /

Sitemap: https://www.yourdomain.com/sitemap.xml

# Example 2: skip two low-value folders
User-agent: *
Disallow: /cart/
Disallow: /search/

Sitemap: https://www.yourdomain.com/sitemap.xml

Keep the file short. Every extra line is one more place for a typing mistake to hide a good page.

Mistakes that can block your whole site

A blocking mistake is a single wrong line in the robots.txt file that stops compliant crawlers from reaching important pages. The most damaging example is Disallow: /, which blocks the entire site. Owners often leave it behind after testing a new design.

Most of these errors happen during a redesign or a move to a new host. The old test setting travels with the site, and nobody checks the file again. Search traffic then drops, and the cause stays hidden for weeks. Use the table to compare each risky line with its safer version, and read the file after every site change.

Risky setup	Why it hurts	Safer setup
Disallow: / left from a test	Compliant crawlers skip every URL	Remove the line, then re-test the file
Using the file to hide a private page	Google says it is not a way to keep a page out of Google	Use noindex or a password
File placed in a subfolder	The file does not work outside the root	Place it at the root of the host
Wrong letter case in a path	Paths are case-sensitive, so the rule misses the page	Copy paths exactly from your URLs
Blocking CSS or JavaScript folders	Google may not see the page as visitors do. We could not confirm this for every setup.	Check Google’s rendering guidance before blocking any asset folder

We could not confirm how the WordPress.com platform builds the file for hosted sites. Check your host’s support page before you assume you can edit it, and note that some privacy settings may change what the file says.

How to test the file in Search Console

Testing a robots.txt file means confirming that it loads publicly and that Google reads it as you intended. Google suggests opening the file in a private browsing window first. You then check it with the robots.txt report in Search Console, which shows fetch and parsing problems.

Testing takes about ten minutes and prevents the costly mistakes above. Do it twice: once before you upload a change, and once after. Google finds and uses the file automatically after you upload and test it. Developers who want a local check can use Google’s open-source robots.txt library. Most shop owners only need the browser check and the report, plus a calm re-read.

• Open yourdomain.com/robots.txt in a private browsing window.
• Confirm the page loads as plain text, with no error.
• Open the robots.txt report in Search Console and look for fetch or parsing issues.
• Read each Disallow line and ask which pages it affects.
• Re-check the file after every redesign, plugin change or host move.

A 6-step check for a Ghaziabad small shop

A 6-step check is a short routine that confirms your robots.txt file blocks nothing important. It works by comparing the file with your sitemap and your key pages. One shop with one site can finish it in under an hour, and you need no coding skills.

Run this check once a quarter and after any site change. Keep a note of the date and what you saw, so you can spot changes later. The steps below fit a local shop in Ghaziabad that depends on a handful of service or product pages, where one blocked page costs real enquiries. For wider local checks, read our local SEO checklist for Ghaziabad shops.

1. Open the file at the root of your domain and read every line.
2. Search for Disallow: / and remove it unless you truly want the whole site blocked.
3. List your five most important pages and confirm no rule matches their paths.
4. Confirm the Sitemap line shows a working, full URL.
5. Check the robots.txt report in Search Console for errors.
6. Fix one thing at a time, and re-test after each change.

Related reading: our guide to local business structured data covers the next technical layer, and the free schema markup generator builds the code for you.

Questions small shop owners ask

Does robots.txt hide a page from Google?

No. Google says robots.txt is not a mechanism for keeping a web page out of Google. A blocked URL can still be indexed if other pages link to it. Use noindex, a password, or remove the page.

Where do I put the robots.txt file?

Place it at the root of your host, for example yourdomain.com/robots.txt. A file inside a subfolder does not work. A subdomain can have its own separate file.

How do I keep a page out of Google Search?

Use a noindex meta tag or header, password protection, or remove the page. These are the options Google lists. Do not rely on robots.txt for this job.

How do I test my robots.txt file?

Open the file in a private browsing window to confirm it loads. Then check it with the robots.txt report in Search Console. Developers can also use Google’s open-source robots.txt library.

What happens if I block a page that other sites link to?

Google can still index the URL without crawling the content, so the page may show up in results without a description. Blocking the crawl does not remove the listing.

Can I edit robots.txt on a WordPress site?

Many hosts and SEO plugins offer a robots.txt editor, so check your host’s support page. We could not confirm how WordPress.com manages the file for hosted sites.

Get a second pair of eyes on your file

Micro Supple, an AI Digital Marketing Company in Vaishali, offers a free consultation and audit for small businesses that want their technical SEO checked in plain words. We read your file, your sitemap and your key pages, then tell you what to fix first. We never promise rankings.

Learn more on the Micro Supple about page, or return to the Micro Supple home page.