Learning how to write and submit a robots.txt file is an important step in managing how search engine crawlers access your website. A correctly configured robots.txt file can help you control crawler traffic, prevent certain areas from being crawled, and make your site’s technical SEO instructions clearer. However, a robots.txt file must be written carefully because a small mistake can block important pages from search engines or create a false sense of security.
This guide explains what robots.txt does, how to create one, where to place it, how to test it, and what “submitting” the file actually means. The instructions are based on Google’s official documentation about creating robots.txt files, including the guidance available from the Google Search Central robots.txt documentation.
What Is a Robots.txt File?
A robots.txt file is a plain-text file that provides instructions to automated crawlers, also called robots or bots. These crawlers visit websites to discover and process content for search engines, analytics platforms, commercial databases, and other services. The file tells eligible crawlers which URL paths they may request and which paths they should avoid.
The file uses the Robots Exclusion Protocol. It is normally placed in the root directory of a website, such as:
https://www.example.com/robots.txthttps://example.com/robots.txt
Robots.txt instructions apply only to the host, protocol, and port where the file is located. For example, a file at https://www.example.com/robots.txt does not automatically control https://example.com/robots.txt, a subdomain, such as blog.example.com, or a different protocol, such as HTTP.
It is also important to understand what robots.txt does not do. It is not an access-control system, password protection method, or reliable way to keep private information secret. A blocked URL may still be discovered by search engines through links, and other crawlers may ignore the instructions entirely. Sensitive content should be protected with authentication, authorization, or server-level access controls.
Why Use a Robots.txt File?
Most websites do not need a complicated robots.txt file. Search engine crawlers can generally discover and crawl public pages without special instructions. Nevertheless, a carefully written file can be useful in several situations.
- Manage crawl activity: Prevent crawlers from repeatedly requesting low-value or resource-heavy areas.
- Exclude technical paths: Block internal search results, temporary files, staging directories, or administrative sections from crawling.
- Reduce duplicate crawling: Limit access to URL patterns that generate many variations of the same content.
- Identify your sitemap: Include the location of an XML sitemap to help crawlers find important URLs.
- Protect server resources: Discourage unnecessary requests to folders containing scripts, logs, or large files.
Robots.txt should be part of a broader technical SEO strategy rather than a replacement for page-level controls. If a page should not appear in search results, consider using an appropriate noindex directive, while ensuring that search engines can access the page to see that directive. Blocking a page in robots.txt can prevent crawlers from seeing its noindex instruction.
How to Write and Submit a Robots.txt File
The process involves five main stages: deciding what needs to be controlled, writing valid directives, saving the file as plain text, uploading it to the correct location, and checking that crawlers can retrieve it. The most important step is planning. Do not begin by copying a restrictive template without understanding which pages your website needs search engines to access.
1. Review Your Website Structure
Before writing rules, review your website’s directories and URL patterns. Identify areas that are useful to search engines and areas that are primarily technical or administrative.
Common examples of paths that may require restrictions include:
/admin/or/login/areas- Internal search pages such as
/search/ - Temporary or development directories
- Shopping cart and checkout paths
- Duplicate URL parameters or filter combinations
- Private scripts, logs, or backup folders
Be cautious with folders such as /css/, /js/, and /images/. Search engines may need access to these resources to render pages and understand how they work on mobile devices. Blocking essential resources can make it harder for crawlers to evaluate the page correctly.
2. Use the Correct Robots.txt Syntax
A robots.txt file consists of groups. Each group begins with one or more User-agent lines, followed by directives such as Disallow or Allow. A group applies to the user agents listed above its directives.
The simplest structure looks like this:
User-agent: *
Disallow: /private/
In this example, the asterisk is a wildcard representing all crawlers. The instruction asks crawlers to avoid URLs beginning with /private/.
3. Understand the Main Directives
User-agent
The User-agent field identifies which crawler a group targets. The value * applies broadly to all crawlers that follow the protocol. You can also specify a particular crawler, although rules for a specific user agent and rules for the wildcard user agent are handled as separate groups.
For example:
User-agent: *
Disallow: /internal/
User-agent: ExampleBot
Disallow: /research/
The first group applies to crawlers generally, while the second group is written for a crawler identified as ExampleBot. Do not assume that naming a user agent will force a crawler to obey the file. Robots.txt is based on voluntary compliance.
Disallow
The Disallow directive identifies URL paths that a matching crawler should not access. A path must begin with a forward slash when it refers to a path on the current host.
User-agent: *
Disallow: /admin/
Disallow: /tmp/
This tells crawlers not to request URLs beginning with /admin/ or /tmp/. A blank Disallow value means that no paths are disallowed in that group:
User-agent: *
Disallow:
An empty rule is different from blocking the entire website. To block all paths for a compliant crawler, use:
User-agent: *
Disallow: /
Use this instruction only when you fully understand the consequences. It can prevent search engines from crawling every page on the site.
Allow
The Allow directive can permit access to a path that would otherwise be covered by a broader Disallow` rule. For example:
User-agent: *
Disallow: /files/
Allow: /files/public/
This configuration blocks the broader /files/ path while allowing the more specific /files/public/ path. When rules conflict, crawlers use the most specific matching rule according to the protocol and their implementation. Because behavior can become difficult to interpret when many overlapping patterns are used, keep your rules as simple as possible.
Sitemap
A Sitemap line identifies the complete URL of an XML sitemap. It is not restricted to a particular user-agent group and can appear anywhere in the file. The URL should be absolute, including the protocol and domain.
Sitemap: https://www.example.com/sitemap.xml
You can list more than one sitemap when necessary:
Sitemap: https://www.example.com/sitemap-posts.xml
Sitemap: https://www.example.com/sitemap-products.xml
A sitemap does not override a disallow rule. It tells crawlers where your preferred URLs are located, while robots.txt rules determine which paths crawlers should avoid requesting.
4. Use Comments and Simple Formatting
Comments begin with a number sign and can make a file easier to maintain:
# Block account and temporary directories
User-agent: *
Disallow: /account/
Disallow: /temporary/
# Main XML sitemap
Sitemap: https://www.example.com/sitemap.xml
Save the file as plain text with the exact filename robots.txt. The filename is lowercase, and the file should not be saved as robots.txt.html, robots.txt.txt, or a formatted document such as a Word file. Avoid adding unnecessary HTML, XML, or rich-text formatting.
Google’s documentation also explains that robots.txt files can use URL path matching and wildcard patterns. An asterisk can match sequences of characters, while a dollar sign can indicate the end of a URL. Since advanced patterns can have unexpected effects, test every pattern and avoid using them unless there is a clear reason.
Robots.txt Examples
Allow All Crawlers to Access the Site
If you do not need to block any paths, you can use:
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
Alternatively, you may omit the robots.txt file entirely if no special instructions are needed. However, keeping a simple file with a sitemap can make your crawler guidance easier to find.
Block One Directory
User-agent: *
Disallow: /private/
Sitemap: https://www.example.com/sitemap.xml
This is a common and easy-to-understand configuration. It blocks URLs that begin with /private/ while leaving other website content available for crawling.
Block Internal Search Results
User-agent: *
Disallow: /search/
Disallow: /?s=
The exact rule depends on how your website generates search URLs. Always inspect real URLs before adding a query-string pattern. A poorly chosen rule could block valuable pages if the pattern is broader than intended.
Block the Entire Site Temporarily
User-agent: *
Disallow: /
This may be appropriate for a private development site that is not ready to be crawled. It should be removed before launch. Also remember that a robots.txt rule is not a security boundary; a private development environment should still use access controls.
Where to Submit a Robots.txt File
Unlike an XML sitemap, a robots.txt file is not normally “submitted” through a form in Google Search Console. To make the file available, upload it to the root of the correct website host. If your website is https://www.example.com, the file should be available at:
https://www.example.com/robots.txt
Depending on your hosting setup, you can upload it using an FTP client, a hosting control panel, a content management system plugin, a deployment platform, or your website’s file manager. If your site uses a CDN, reverse proxy, or server framework, confirm that the file is served from the public root rather than from a subdirectory.
The server should return the file over HTTP or HTTPS with a successful response. Open the URL in a browser to confirm that the contents are visible. A robots.txt file at https://www.example.com/blog/robots.txt does not control the entire domain; it is in the wrong location for that purpose.
Should You Submit Robots.txt in Google Search Console?
Google automatically checks the robots.txt file for a host when it crawls the site. Search Console can help you inspect some crawl and indexing problems, but there is generally no standard “submit robots.txt” process comparable to submitting a sitemap. After publishing an update, allow time for Google to fetch the new file. Crawlers may cache the file for a period of time, so changes are not always reflected instantly.
If you urgently need a temporary robots.txt change to be recognized, consult the current Google Search Central documentation and Search Console features available for your property. Do not repeatedly edit the file without verifying the result, because rapid changes can make troubleshooting more difficult.
How to Test Your Robots.txt File
Testing is essential before and after publishing. First, visit the live file directly and check that the expected content appears. Next, review each important URL and ask whether the current rules allow or block it as intended.
- Confirm the file is located at the root of the correct host.
- Check that the filename is exactly
robots.txt. - Verify that the server returns a successful response.
- Look for accidental rules such as
Disallow: /. - Test important pages, assets, category pages, and XML sitemaps.
- Check for spelling errors in directory names and URL paths.
- Review Search Console crawl and indexing reports for unexpected changes.
Google has provided robots.txt testing tools in various products and interfaces over time, so the exact availability may change. Use current tools in Search Console or Google’s official documentation where applicable. Third-party validators can be useful for syntax checks, but they cannot guarantee that every crawler will interpret a file identically.
Common Robots.txt Mistakes to Avoid
Blocking Important Content
The most serious error is accidentally blocking pages that should appear in search results. A rule such as Disallow: / or a broad directory rule can stop crawlers from accessing the entire site or a major section. Review the file whenever your site structure changes.
Trying to Hide Confidential Information
Do not place confidential URLs, passwords, customer information, or private documents behind robots.txt rules. The file is publicly accessible, so it can reveal the existence of paths that you intended to conceal. Use authentication and server permissions for sensitive material.
Blocking a Page and Expecting Noindex to Work
If a crawler cannot access a page because of robots.txt, it may not be able to read a page-level noindex directive. If the goal is to remove a page from search results, use the appropriate indexing control and ensure that crawlers can access the page when necessary.
Using Incorrect URL Paths
Robots.txt paths are matched against URL paths, not local computer folders. A rule for /private/ will not necessarily affect a URL such as /account/private/ unless the pattern matches that location. Include the leading slash and check the actual public URL.
Assuming Every Bot Obeys the File
Reputable search engines generally follow robots.txt instructions, but malicious bots may ignore them. If a crawler is causing harm, use rate limiting, firewall rules, authentication, bot management, or other server-level defenses.
Best Practices for Maintaining Robots.txt
Keep your robots.txt file short, readable, and focused on crawl management. A small number of clear rules is usually safer than a long collection of patterns added over time. Store the file in your source-control system if your website uses a deployment workflow, and review changes before they reach production.
Whenever you redesign your website, migrate domains, change URL structures, add a JavaScript application, or launch a new content section, review the robots.txt file. Maintain an up-to-date XML sitemap and make sure the sitemap contains canonical, indexable URLs that you want search engines to discover.
Finally, remember that robots.txt controls crawling, not guaranteed indexing. A page that is allowed to be crawled may still not be indexed, and a blocked URL may sometimes appear as a reference without its content being crawled. Use robots.txt for crawler instructions and combine it with canonical tags, redirects, access controls, and indexing directives as appropriate.
Conclusion
Knowing How to write and submit a robots.txt file helps you give search engine crawlers clear instructions about your website. Start by identifying technical or low-value paths, write simple user-agent and disallow rules, include your sitemap when useful, and save the file as plain text. Then upload it to the root of the correct host and verify the live URL.
There is usually no separate submission form required. Once the file is publicly available at /robots.txt, compliant crawlers can retrieve it automatically.