Robots.txt – your website’s first line of defense against search engine spiders

The future of bots

The Web crawlers that search engine bots are the technological tools used by search engines to index content on the World Wide Web. The crawlers follow links and make decisions about which pages they should include in the index, or discard because they’re not relevant enough to the keywords being searched for, duplicate pages that have already been indexed elsewhere, or expired pages that can’t be crawled any more.

Right now, most people are barely conscious of how bots are impacting their lives on a daily basis. That will change soon as bots get smarter and start doing things like teaching kids at school or acting as customer service representatives for busy companies. If you want to stop the robots from taking over, make sure your business is ready for them by using robots.txt file.

What is a robots.txt file?

A robots.txt file is an instruction file telling search engine crawlers what files to crawl, and what not to crawl on your website. A crawler is a program that moves around the web from site to site gathering data for search engines like Google, Yahoo!, and Bing.

When a crawler arrives at your website, it scans its contents for URLs pointing to other pages or files on the server (website). The robot will then check if you have set up any instructions in the robots.txt file about which content should be indexed and crawled.

The anatomy of a robots.txt

What should I include in my robots.txt file?

The robots.txt file is located in the top-level directory of your website and is a text file containing commands for web crawlers about how to crawl your site. When executed, this text file dictates whether bots are allowed to access certain parts of your site or not.

• The Allow and Disallow directives: tell the robot whether they can access the URL and verify it (Allow) or whether it is unavailable (Disallow). By default, each robot has the ability to scan each subpage, unless its access is restricted.
• User-agent: instructions are usually followed by one or more rules which apply to those instructions in parenthesis, but this is not mandatory.
• Why should you include a link to your XML site map in the robots file? Google bots visit robots.txt often – it’s one of the first places they start their journey through your website. Placing a link to the sitemap will make it easier for them to navigate and it will take them less time to scan it.
• Host – thanks to this directive, you can choose your preferred domain from among its copies on the Internet.
• Crawl delay tells search engine spiders how long they have to wait before crawling the page, if it is not a dynamic page. The crawl delay will give you a time-out for when a certain site is unreachable and not responding to requests. To set the crawler timeout limit in robots.txt, use this syntax: crawl-delay:30. That translates to wait up to 30 seconds.

Why do you need one?

Website creators use robots.txt files to control how search engine spiders crawl their site, the purpose being to either allow or prevent them from indexing different pages and parts of a site that may be in need of repair or that are not applicable for public consumption. It is also used by website administrators to avoid crawlers indexing live parts of their website while they’re still working on it and not ready for public access.

How to set up a robots.txt file?

There are several text editors that you can use to create a robots.txt file, such as Notepad, TextEdit, vi, and emacs. Instead of using a sophisticated word processing program, use something less complicated. Many such programs, by default, save files in a proprietary format and include any unexpected characters, such as printer quotes, which complicate tasks for automated bots. To avoid this problem, if you are prompted to specify an encoding in the file saving dialog, select UTF-8.

Implementing

After saving the robots.txt file on your computer, you can make it available to the search bots. There is no single tool that can help you with this, as how you upload your robots.txt file depends on the architecture of your site and server. Contact the hosting company or view their documentation. After uploading your robots.txt file, check that it is publicly available and that Google can analyze it.

How to test a file?

Google Search Console is the most popular tool to ensure that the robots file you created does not contain errors. Enter or select your domain, select “Download,” and click “Robots.txt Tester.” Check that the list of available and inaccessible subpages is correct. Those that are available will be green, and those that are blocked will be red.

Common mistakes when creating your robots.txt file

• One mistake some people make is to use the txt extension instead of txt after the first dot. This makes your robots file inaccessible by web crawlers, so make sure you are using .txt after your first .robots or it will not work properly.
• Another common error is when a site owner specifies that a page should be blocked, but doesn’t specify which one. It can get confusing if there are multiple pages with the same name on your site, and it may take time for Google to figure out which one you want blocked.
• Always be sure to specify your page by URL rather than its title, as titles can change while URLs don’t. If you use a specific location on your website to house user-generated content and want Googlebot to visit those pages, you must use special syntax in your robots file. You can learn more about how to do that here.

Summary

Robots.txt is a text file that provides instructions for web crawlers like Google, Bing, and Yahoo about which parts of your website to crawl and which ones not to touch. The crawling typically refers to the ability for search engine spiders (the code on the search engine end that does all the reading) to find, access, read, and index your content for search results pages you show up in their various searches such as when someone wants info on a company or specific product name, among other things.

If you want to keep your website in pristine condition and deter search engine spiders from indexing parts of your site that are not ready for public viewing, create a robots.txt file and upload it to the root directory of your site. Doing so is a quick, easy way to protect pages on your site from Google, Bing, Yahoo! and other crawlers that follow links and index sites for crawling purposes.