Order a free seo audit

Robots.txt – your website’s first line of defense against search engine spiders

7 minutes of reading
Robots.txt – your website’s first line of defense against search engine spiders
Category SEO

The robots.txt file, which you may not have heard of, is one of the most important tools in the webmaster’s arsenal. Many marketers are aware of it, and most website owners have heard about it, but few actually know how to use it and what role it plays in search engine optimization. This article aims to provide a basic introduction to this powerful tool, as well as some ideas on how you can utilize it for your own website’s optimization strategy and increased visibility in search engine results pages.


The future of bots

The Web crawlers that search engine bots are the technological tools used by search engines to index content on the World Wide Web. The crawlers follow links and make decisions about which pages they should include in the index, or discard because they’re not relevant enough to the keywords being searched for, duplicate pages that have already been indexed elsewhere, or expired pages that can’t be crawled any more.

Right now, most people are barely conscious of how bots are impacting their lives on a daily basis. That will change soon as bots get smarter and start doing things like teaching kids at school or acting as customer service representatives for busy companies. If you want to stop the robots from taking over, make sure your business is ready for them by using robots.txt file.

What is a robots.txt file?

A robots.txt file is an instruction file telling search engine crawlers what files to crawl, and what not to crawl on your website. A crawler is a program that moves around the web from site to site gathering data for search engines like Google, Yahoo!, and Bing.

When a crawler arrives at your website, it scans its contents for URLs pointing to other pages or files on the server (website). The robot will then check if you have set up any instructions in the robots.txt file about which content should be indexed and crawled.

The anatomy of a robots.txt

What should I include in my robots.txt file?

The robots.txt file is located in the top-level directory of your website and is a text file containing commands for web crawlers about how to crawl your site. When executed, this text file dictates whether bots are allowed to access certain parts of your site or not.

  • The Allow and Disallow directives: tell the robot whether they can access the URL and verify it (Allow) or whether it is unavailable (Disallow). By default, each robot has the ability to scan each subpage, unless its access is restricted.
  • User-agent: instructions are usually followed by one or more rules which apply to those instructions in parenthesis, but this is not mandatory.
  • Why should you include a link to your XML site map in the robots file? Google bots visit robots.txt often – it’s one of the first places they start their journey through your website. Placing a link to the sitemap will make it easier for them to navigate and it will take them less time to scan it.
  • Host – thanks to this directive, you can choose your preferred domain from among its copies on the Internet.
  • Crawl delay tells search engine spiders how long they have to wait before crawling the page, if it is not a dynamic page. The crawl delay will give you a time-out for when a certain site is unreachable and not responding to requests. To set the crawler timeout limit in robots.txt, use this syntax: crawl-delay:30. That translates to wait up to 30 seconds.

Why do you need one?

Website creators use robots.txt files to control how search engine spiders crawl their site, the purpose being to either allow or prevent them from indexing different pages and parts of a site that may be in need of repair or that are not applicable for public consumption. It is also used by website administrators to avoid crawlers indexing live parts of their website while they’re still working on it and not ready for public access.

How to set up a robots.txt file?

There are several text editors that you can use to create a robots.txt file, such as Notepad, TextEdit, vi, and emacs. Instead of using a sophisticated word processing program, use something less complicated. Many such programs, by default, save files in a proprietary format and include any unexpected characters, such as printer quotes, which complicate tasks for automated bots. To avoid this problem, if you are prompted to specify an encoding in the file saving dialog, select UTF-8.


After saving the robots.txt file on your computer, you can make it available to the search bots. There is no single tool that can help you with this, as how you upload your robots.txt file depends on the architecture of your site and server. Contact the hosting company or view their documentation. After uploading your robots.txt file, check that it is publicly available and that Google can analyze it.

How to test a file?

Google Search Console is the most popular tool to ensure that the robots file you created does not contain errors. Enter or select your domain, select “Download,” and click “Robots.txt Tester.” Check that the list of available and inaccessible subpages is correct. Those that are available will be green, and those that are blocked will be red.

Common mistakes when creating your robots.txt file

  • One mistake some people make is to use the txt extension instead of txt after the first dot. This makes your robots file inaccessible by web crawlers, so make sure you are using .txt after your first .robots or it will not work properly.
  • Another common error is when a site owner specifies that a page should be blocked, but doesn’t specify which one. It can get confusing if there are multiple pages with the same name on your site, and it may take time for Google to figure out which one you want blocked.
  • Always be sure to specify your page by URL rather than its title, as titles can change while URLs don’t. If you use a specific location on your website to house user-generated content and want Googlebot to visit those pages, you must use special syntax in your robots file. You can learn more about how to do that here.
Make an appointment for a free audit of your website
Order a free seo audit


Robots.txt is a text file that provides instructions for web crawlers like Google, Bing, and Yahoo about which parts of your website to crawl and which ones not to touch. The crawling typically refers to the ability for search engine spiders (the code on the search engine end that does all the reading) to find, access, read, and index your content for search results pages you show up in their various searches such as when someone wants info on a company or specific product name, among other things.

If you want to keep your website in pristine condition and deter search engine spiders from indexing parts of your site that are not ready for public viewing, create a robots.txt file and upload it to the root directory of your site. Doing so is a quick, easy way to protect pages on your site from Google, Bing, Yahoo! and other crawlers that follow links and index sites for crawling purposes.

Aleksandra Pietrzak
Curator at the National Museum in Poznań, graduate of Art History at the Jagiellonian University and Contemporary Art at the Pedagogical University of Krakow, curator of exhibitions and author of scientific and popular texts. A lover of contemporary art, literature and travel.
Also check
Are you wondering why your website is NOT SELLING?
Schedule a free SEO consultation and find out how we can improve your sales results.

Rate the article
Average rating 5/5 - Number of ratings: 5
Add comment

Your email address will not be published. Required fields are marked *


Would you like to see what else we have written about?

What is Outreach and its effect on SEO?
What is Outreach and its effect on SEO?
Boost your website's visibility with the power of Outreach! Learn how it drives organic traffic & elevates your SEO strategy in our latest blog.
SEO Optimization: How to Increase Your CTR?
SEO Optimization: How to Increase Your CTR?
Purpose: clicks. How to encourage users to interact with the website? What is CTR and how does it affect your ranking? Optimization of the most important elements.
SEO for an interior design store
SEO for an interior design store
Are you looking to boost your interior design store's online presence? Look no further than optimizing your website for search engines with these simple SEO tips.

Get started

with the comprehensive
SEO audit

Invest in a detailed SEO audit and understand your online performance. We analyze your website to get a clear view of what you can improve.

  • I Please send us a message first for the introduction.
  • II Then, our SEO Expert gets back right to you with a phone call.
  • III We schedule a consultation in time that works for you.
  • IV The SEO Expert audits your website and provides strategic recommendations on how to improve your performance.
  • V You'll get the SEO report with a comprehensive look at numerous search ranking factors such as technical items, on-page, content, and off-page metrics.