{"id":687,"date":"2016-11-07T00:00:00","date_gmt":"2016-11-07T00:00:00","guid":{"rendered":"http:\/\/blog.hostinger.io\/hostinger-tutorials\/uncategorized\/websitehow-to-block-search-engines-using-robotstxt\/"},"modified":"2025-04-13T08:25:26","modified_gmt":"2025-04-13T08:25:26","slug":"how-to-block-search-engines-using-robotstxt","status":"publish","type":"post","link":"\/tutorials\/how-to-block-search-engines-using-robotstxt\/","title":{"rendered":"Robots.TXT disallow: how does it block search engines"},"content":{"rendered":"

Are you looking for a way to control how search engine bots crawl your site? Or do you want to make some parts of your website private? You can do it by modifying the robots.txt<\/strong> file with the disallow<\/strong> command.<\/p>

In this article, you will learn what robots.txt<\/strong> can do for your site. We’ll also show you how to use it in order to block search engine crawlers.<\/p>

\"\"<\/a><\/figure>

\n\n\n\n\n\n\n<\/p>

What Is Robots.txt<\/h2>

Robots.txt <\/strong>is a plain text file used to communicate with web crawlers. The file is located in the root directory of a site.<\/p>

It works by telling the search bots which parts of the site should and shouldn’t be scanned. It’s up to robots.txt <\/strong>whether the bots are allowed or disallowed to crawl a website.<\/p>

In other words, you can configure the file to prevent search engines from scanning and indexing pages<\/a> or files on your site.<\/p>

Why Should I Block a Search Engine<\/h2>

If you want to block crawlers from accessing your entire website, or if you have sensitive information on pages that you want to make private. Unfortunately, search engine bots can’t automatically distinguish between public and private content. In this case, restricting access is necessary.<\/p>

You can also restrict bots from crawling your entire site. Especially if your website is in maintenance mode or staging.<\/p>

Another use of robots.txt<\/strong> is to prevent duplicate content<\/a> issues that occur when the same posts or pages appear on different URLs. Duplicates can negatively impact search engine optimization (SEO)<\/a>.<\/p>

The solution is simple – identify duplicate content, and disallow bots from crawling it.<\/p>

How to Use Robots.txt to Disallow Search Engines<\/h2>

If you want to check your site’s robots.txt <\/strong>file, you can view it by adding robots.txt<\/strong> after your site’s URL, for example, www.myname.com\/robots.txt<\/strong>. You can edit it through your web hosting control panel’s file manager, or an FTP client<\/a>.<\/p>

Let’s configure the robots.txt<\/strong> file via Hostinger’s hPanel’s<\/strong> file manager. First, you have to enter the File Manager<\/strong> in the Files<\/strong> section of the panel. Then, open the file from the public_html<\/strong> directory.<\/p>

\n
\"The<\/a><\/figure>\n<\/div>

If the file isn’t there, you can create it manually. Just click the New File<\/strong> button at the top right corner of the file manager, name it robots.txt<\/strong> and place it in public_html<\/strong>.<\/p>

Now you can start adding commands to the file. The two main ones you should know are:<\/p>

    \n
  1. User-agent<\/strong> – refers to the type of bot that will be restricted, such as Googlebot<\/a> or Bingbot<\/a>.<\/li>\n\n\n\n
  2. Disallow <\/strong>– is where you want to restrict the bots.<\/li>\n<\/ol>

    Let’s look at an example. If you want to prevent Google’s bot from crawling on a specific folder of your site, you can put this command in the file:<\/p>

    User-agent: Googlebot \nDisallow: \/example-subfolder\/<\/pre>
    You can also block the  search bots from crawling on a specific web page. If you want to block Bingbot from a page, then you can set the command like this:<\/p>
    User-agent: Bingbot\n\nDisallow: \/example-subfolder\/blocked-page.html<\/pre>
    Now, what if you want the robots.txt<\/strong> file to disallow all search engine bots? You can do it by putting an asterisk (*)<\/strong> next to User-agent<\/strong>. And if you want to prevent them from accessing the entire site, just put a slash (\/)<\/strong> next to Disallow<\/strong>. Here’s how it looks:<\/p>
    User-agent: *\n\nDisallow: \/<\/pre>
    You can set up different configurations for different search engines by adding multiple commands to the file. Also, keep in mind that the changes take effect after you save the robots.txt<\/strong> file.<\/p>
    Conclusion<\/h2>
    Now you’ve learned how to modify the robots.txt<\/strong> file. This lets you manage search engine bot access to your website. Now you can rest easy knowing that only what you want to be found on search engine result pages will show up there.<\/p>
    \n\n\n        
    \n            
    \n                \n                    \n                <\/svg>\n                
    \n                    Learn More About Website Optimization                <\/p>\n            <\/div>\n            
    What Is Website Optimization<\/a>
    \n    What Is a Sitemap<\/a>
    \n    How to Submit Your Website to Search Engines<\/a>
    \n    Website Localization Complete Guide<\/a><\/p>\n                    <\/div>\n        \n<\/p>\n","protected":false},"excerpt":{"rendered":"
    Are you looking for a way to control how search engine bots crawl your site? Or do you want to […]<\/p>\n
    Read More…<\/a><\/p>\n","protected":false},"author":24,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_title":"How to Block Search Engines Using robots.txt disallow Rule","rank_math_description":"You can disallow all search engine bots to crawl on your site using the robots.txt file. In this article, you will learn exactly how to do it!","rank_math_focus_keyword":"robots txt disallow all","footnotes":""},"categories":[22617,22619],"tags":[],"class_list":["post-687","post","type-post","status-publish","format-standard","hentry","category-digital-marketing","category-seo"],"hreflangs":[{"locale":"en-US","link":"https:\/\/www.hostinger.com\/tutorials\/how-to-block-search-engines-using-robotstxt","default":0},{"locale":"pt-BR","link":"https:\/\/www.hostinger.com\/br\/tutoriais\/como-criar-robots-txt-bloquear-mecanismos-busca","default":0},{"locale":"es-ES","link":"https:\/\/www.hostinger.com\/es\/tutoriales\/como-bloquear-los-motores-de-busqueda-utilizando-robotstxt","default":0},{"locale":"en-GB","link":"https:\/\/www.hostinger.com\/uk\/tutorials\/how-to-block-search-engines-using-robotstxt","default":0},{"locale":"es-MX","link":"https:\/\/www.hostinger.com\/mx\/tutoriales\/como-bloquear-los-motores-de-busqueda-utilizando-robotstxt","default":0},{"locale":"es-CO","link":"https:\/\/www.hostinger.com\/co\/tutoriales\/como-bloquear-los-motores-de-busqueda-utilizando-robotstxt","default":0},{"locale":"es-AR","link":"https:\/\/www.hostinger.com\/ar\/tutoriales\/como-bloquear-los-motores-de-busqueda-utilizando-robotstxt","default":0},{"locale":"pt-PT","link":"https:\/\/www.hostinger.com\/pt\/tutoriais\/como-criar-robots-txt-bloquear-mecanismos-busca","default":0},{"locale":"en-IN","link":"https:\/\/www.hostinger.com\/in\/tutorials\/how-to-block-search-engines-using-robotstxt","default":0},{"locale":"en-CA","link":"https:\/\/www.hostinger.com\/ca\/tutorials\/how-to-block-search-engines-using-robotstxt","default":0},{"locale":"en-PH","link":"https:\/\/www.hostinger.com\/ph\/tutorials\/how-to-block-search-engines-using-robotstxt","default":0},{"locale":"en-MY","link":"https:\/\/www.hostinger.com\/my\/tutorials\/how-to-block-search-engines-using-robotstxt","default":0},{"locale":"en-AU","link":"https:\/\/www.hostinger.com\/au\/tutorials\/how-to-block-search-engines-using-robotstxt","default":0},{"locale":"en-NG","link":"https:\/\/www.hostinger.com\/ng\/tutorials\/how-to-block-search-engines-using-robotstxt","default":0}],"_links":{"self":[{"href":"https:\/\/www.hostinger.com\/tutorials\/wp-json\/wp\/v2\/posts\/687","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.hostinger.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hostinger.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hostinger.com\/tutorials\/wp-json\/wp\/v2\/users\/24"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hostinger.com\/tutorials\/wp-json\/wp\/v2\/comments?post=687"}],"version-history":[{"count":20,"href":"https:\/\/www.hostinger.com\/tutorials\/wp-json\/wp\/v2\/posts\/687\/revisions"}],"predecessor-version":[{"id":112206,"href":"https:\/\/www.hostinger.com\/tutorials\/wp-json\/wp\/v2\/posts\/687\/revisions\/112206"}],"wp:attachment":[{"href":"https:\/\/www.hostinger.com\/tutorials\/wp-json\/wp\/v2\/media?parent=687"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hostinger.com\/tutorials\/wp-json\/wp\/v2\/categories?post=687"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hostinger.com\/tutorials\/wp-json\/wp\/v2\/tags?post=687"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}