{"id":9744,"date":"2018-05-28T11:51:10","date_gmt":"2018-05-28T11:51:10","guid":{"rendered":"https:\/\/www.hostinger.com\/tutorials\/?p=9744"},"modified":"2026-03-09T19:20:01","modified_gmt":"2026-03-09T19:20:01","slug":"wordpress-robots-txt","status":"publish","type":"post","link":"\/ng\/tutorials\/wordpress-robots-txt","title":{"rendered":"The complete guide to WordPress robots.TXT"},"content":{"rendered":"

To ensure that your site ranks highly in Search Engine Result Pages (SERPs), you’ll need to make it easy for search engine ‘bots’ to explore its most important pages. Having a well-structured robots.txt <\/strong>file in place will help direct those bots to the pages you want them to index (and avoid the rest).<\/p>

In this article, we’re going to cover:<\/p>

    \n
  1. What a robots.txt <\/strong>file is, and why it’s important<\/li>\n\n\n\n
  2. Where the WordPress robots.txt <\/strong>file is located.<\/li>\n\n\n\n
  3. How to create a robots.txt <\/strong>file.<\/li>\n\n\n\n
  4. What rules to include in your WordPress robots.txt <\/strong>file.<\/li>\n\n\n\n
  5. How to test the robots.txt <\/strong>file, and submit it to Google Search Console.<\/li>\n<\/ol>

    By the end of our discussion, you’ll have everything you need to configure a perfect robots.txt <\/strong>file for your WordPress website. Let’s dive in!<\/p>

    Download all-in-one WordPress cheat sheet<\/a><\/p>

    \n\n\n\n\n<\/p>

    What a WordPress robots.txt <\/strong>File Is (And Why You Need One)<\/h2>
    \"An
    WordPress’ default robots.txt<\/strong> file is pretty basic, but you can easily replace it.<\/figcaption><\/figure><\/div>

    When you create a new website, search engines will send their minions (or bots)<\/a> to ‘crawl’ through it and make a map of all the pages it contains. That way, they’ll know what pages to display as results when someone searches for related keywords. At a basic level, this is simple enough.<\/p>

    The problem is that modern websites contain a lot<\/em> more elements than just pages. WordPress enables<\/a> you to install plugins, for example, which often come with their own directories. You don’t want these to show up in your search engine results, however, since they’re not relevant content.<\/p>

    What the robots.txt <\/strong>file<\/a> does is provide a set of instructions for search engine bots. It tells them: “Hey, you can look here, but don’t go into those rooms over there!” This file can be as detailed as you want, and it’s rather easy to create, even if you’re not a technical wizard.<\/p>

    In practice, search engines will still crawl your website even if you don’t have a robots.txt <\/strong>file set up. However, not creating one is inefficient. Without this file, you’re leaving it up to the bots to index all your content, and they’re so thorough that they might end up showing parts of your website you don’t want other people to have access to.<\/p>

    More importantly, without a robots.txt <\/strong>file, you’ll have a lot of bots crawling all over your website. This can negatively impact its performance. Even if the hit is negligible, page speed is something that should always be at the top of your priorities list. After all, there are few things people hate as much as slow websites.<\/p>

    \"\"<\/a><\/figure>

    Where the WordPress robots.txt File Is Located<\/h2>

    When you create a WordPress website, it automatically sets up a virtual robots.txt <\/strong>file located in your server’s main folder. For example, if your site is located at yourfakewebsite.com<\/em>, you should be able to visit the address yourfakewebsite.com\/robots.txt,<\/em> <\/em>and see a file like this come up:<\/p>

    User-agent: *\nDisallow: \/wp-admin\/\nDisallow: \/wp-includes\/<\/pre>
    This is an example of a very basic robots.txt <\/strong>file. To put it in human terms, the part right after User-agent: <\/strong>declares which bots the rules below apply to. An asterisk means the rules are universal and apply to all bots. In this case, the file tells those bots that they can’t go into your wp-admin <\/strong>and wp-includes <\/strong>directories. That makes a certain amount of sense since those two folders contain a lot of sensitive files.<\/p>
    However, you may want to add more rules to your own file. Before you can do that, you’ll need to understand that this is a virtual<\/em> file. Usually, the WordPress robots.txt <\/strong>location is within your root <\/strong>directory, which is often called public_html <\/strong>or www<\/strong> (or is named after your website):<\/p>
    \"WordPress<\/figure><\/div>
    However, the robots.txt <\/strong>file WordPress sets up for you by default isn’t accessible at all, from any directory. It works, but if you want to make changes to it, you’ll need to create your own <\/strong>file and upload it to your root <\/strong>folder as a replacement.<\/p>
    We’ll cover several ways to create a new robots.txt <\/strong>for WordPress in a minute. For now, though, let’s talk about how to determine what rules yours should include.<\/p>
    What Rules to Include in Your WordPress robots.txt <\/strong>File<\/h2>
    In the last section, you saw an example of a WordPress-generated robots.txt <\/strong>file. It only included two short rules, but most websites set up more than that. Let’s take a look at two different robots.txt <\/strong>files, and talk about what they each do differently.<\/p>
    Here is our first WordPress robots.txt<\/strong> example:<\/p>
    User-agent: *\nAllow: \/\n# Disallowed Sub-Directories\nDisallow: \/checkout\/\nDisallow: \/images\/\nDisallow: \/forum\/<\/pre>
    This is a generic robots.txt <\/strong>file for a website with a forum. Search engines will often index each thread within a forum. Depending on what your forum is for, however, you might want to disallow it. That way, Google won’t index hundreds of threads about users making small talk. You could also set up rules indicating specific sub-forums to avoid, and let search engines crawl the rest of them.<\/p>
    You’ll also notice a line that reads Allow: \/ <\/strong>at the top of the file. That line tells bots that they can crawl through all of your website pages, aside from the exceptions you set below. Likewise, you’ll notice that we set these rules to be universal (with an asterisk), just as WordPress’ virtual robots.txt <\/strong>file does.<\/p>
    Now let’s check out another WordPress robots.txt<\/strong> example:<\/p>
    User-agent: *\nDisallow: \/wp-admin\/\nDisallow: \/wp-includes\/\nUser-agent: Bingbot\nDisallow: \/<\/pre>
    In this file, we set up the same rules WordPress does by default. However, we also added a new set of rules that block Bing’s search bot<\/a> from crawling through our website. Bingbot, as you might imagine, is the name of that bot.<\/p>
    You can get pretty specific about which search engine’s bots get access to your website, and which ones don’t. In practice, of course, Bingbot is pretty benign (even if it’s not as cool as Googlebot). However, there are<\/em> some malicious bots out there.<\/p>
    The bad news is that they don’t always follow the instructions of your robots.txt <\/strong>file (they are rebels, after all). It’s worth keeping in mind that, while most bots will follow the instructions you provide in this file, you aren’t forcing<\/em> them to do so. You’re just asking nicely.<\/p>
    If you read up on the subject, you’ll find a lot of suggestions for what to allow and what to block on your WordPress website. However, in our experience, fewer rules are often better. Here’s an example of what we recommend your first robots.txt <\/strong>file should look like:<\/p>
    User-Agent: *\nAllow: \/wp-content\/uploads\/\nDisallow: \/wp-content\/plugins\/<\/pre>
    Traditionally, WordPress likes to block access to the wp-admin <\/strong>and wp-includes <\/strong>directories. However, that’s no longer considered a best practice<\/a>. Plus, if you add metadata to your images for Search Engine Optimization (SEO)<\/a> purposes, it doesn’t make sense to disallow bots from crawling that information. Instead, the above two rules cover what most basic sites will require.<\/p>
    What you include in your robots.txt <\/strong>file will depend on your specific site and needs, however. So feel free to do some more research on your own!<\/p>
    How to Create a WordPress robots.txt <\/strong>File (3 Methods)<\/h2>
    Once you’ve decided what will go in your robots.txt<\/strong> file, all that’s left is to create one. You can edit robots.txt<\/strong> in WordPress either by using a plugin or manually. In this section, we’ll teach you how to use two popular plugins to get the job done and discuss how to create and upload the file on your own. Let’s get to it!<\/p>
    1. Use Yoast SEO<\/h3>
    \"The<\/figure><\/div>
    Yoast SEO<\/a> hardly needs an introduction. It’s the most popular SEO plugin for WordPress, and it enables you to optimize your posts and pages to make better use of your keywords. Aside from that, it also provides you with help when it comes to increasing your content’s readability, which means more people will be able to enjoy it.<\/p>
    Personally, we’re fans of Yoast SEO<\/a> due to its ease of use. That applies just as much to creating a robots.txt <\/strong>file. Once you install and activate the plugin<\/a>, navigate to the SEO -› Tools <\/strong>tab in your dashboard, and look for the option that says File editor:<\/strong><\/p>
    \"The<\/figure><\/div>
    Clicking on that link will send you to a new page, where you can edit your .htaccess <\/strong>file without leaving your dashboard. There’s also a handy button that reads Create robots.txt file<\/strong>, which does exactly what you’d expect:<\/p>
    \"Creating<\/figure><\/div>
    Once you click on this button, the tab will display a new editor where you can modify your robots.txt <\/strong>file directly. Keep in mind that Yoast SEO sets its own default rules, which override your existing virtual robots.txt <\/strong>file.<\/p>
    Whenever you add or remove rules, remember to click on the Save changes to robots.txt <\/strong>button, so they persist:<\/p>
    \"Edit<\/figure><\/div>
    That’s easy enough! Now let’s see how another popular plugin does the same thing.<\/p>
    2. Through the All in One SEO Pack Plugin<\/h3>
    \"The<\/figure><\/div>
    All in One SEO Pack<\/a> is the other big name when it comes to WordPress SEO. It includes most of the features Yoast SEO does, but some people prefer it because it’s a more lightweight plugin. As far as robots.txt <\/strong>goes, creating the file with this plugin is also just as simple.<\/p>
    Once you have the plugin set up, navigate to the All in One SEO > Feature Manager <\/strong>page in your dashboard. Inside, you’ll find an option called Robots.txt<\/strong>, with a conspicuous Activate <\/strong>button right below it. Go ahead and click on that:<\/p>
    \"Activating<\/figure><\/div>
    Now, a new Robots.txt <\/strong>tab will show up under your All in One SEO <\/strong>menu. If you click on it, you’ll see options to add new rules to your file, save the changes you make, or delete it altogether:<\/p>
    \"Adding<\/figure><\/div>
    Do note that you can’t make changes to your robots.txt <\/strong>file directly using this plugin. The file itself is grayed out, unlike with Yoast SEO, which enables you to type in whatever you want:<\/p>
    \"A<\/figure><\/div>
    In any case, adding new rules is simple, so don’t let that small downside discourage you. More importantly, All in One SEO Pack also includes a feature that can help you block ‘bad’ bots, which you can access from your All in One SEO <\/strong>tab:<\/p>
    \"Using<\/figure><\/div>
    That’s all you need to do if you choose to use this method. However, let’s talk about how to create a robots.txt <\/strong>file manually, if you don’t want to set up an extra plugin just to take care of this task.<\/p>
    3. Create and Upload Your WordPress robots.txt <\/strong>File Via FTP<\/h3>
    Creating a txt <\/strong>file couldn’t be simpler. All you have to do is open up your favorite text editor (such as Notepad or TextEdit), and type in a few lines. Then you can save the file, using any name you want and the txt <\/strong>file type. It literally takes seconds to do this, so it makes sense that you might want to edit robots.txt<\/strong> in WordPress without using a plugin.<\/p>
    Here’s a quick example of one such file:<\/p>
    \"Creating<\/figure><\/div>
    For the purposes of this tutorial, we saved this file directly to our computer. Once you have your own file created and saved, you’ll need to connect to your website via FTP<\/a>. If you’re not sure how to do that, we have a guide to doing this using the beginner-friendly FileZilla client<\/a>.<\/p>
    Once you’re connected to your site, navigate to the public_html <\/strong>folder. Then, all you have to do is upload the robots.txt <\/strong>file from your computer over to your server. You can do that by either right-clicking on the file using your FTP client’s local navigator, or by simply dragging and dropping it into place:<\/p>
    \"Uploading<\/figure><\/div>
    It should only take a few seconds for the file to upload. As you can see, this method is nearly as simple as using a plugin.<\/p>
    How to Test Your WordPress robots.txt File and Submit It to Google Search Console<\/h2>
    Once your WordPress robots.txt file is created and uploaded, you can use Google Search Console<\/a> to test it for errors. The Search Console is a collection of tools Google offers to help you monitor how your content appears in its search results. One of these tools is a robots.txt <\/strong>checker, which you can use by logging into your console and navigating to the robots.txt Tester <\/strong>tab:<\/p>
    \"The<\/figure><\/div>
    Inside, you’ll find an editor field where you can add your WordPress robots.txt <\/strong>file code, and click on the Submit <\/strong>button right below. Google Search Console will ask if you want to use that new code, or pull the file from your website. Click on the option that says Ask Google to Update <\/strong>to submit it manually:<\/p>
    \"Submitting<\/figure><\/div>
    Now, the platform will check your file for errors. If there are any, it will point them out for you. However, you’ve seen more than one WordPress robots.txt<\/strong> example by now, so chances are high that yours is perfect!<\/p>
    Conclusion<\/h2>
    In order to increase your site’s exposure, you’ll need to ensure that search engine bots are crawling the most relevant information. As we have seen, a well-configured WordPress robots.txt <\/strong>file will enable you to dictate exactly how those bots interact with your site. That way, they’ll be able to present searchers with more relevant and useful content.<\/p>
    Do you have any questions about how to edit robots.txt<\/strong> in WordPress? Let us know in the comments section below!<\/p>
    \n                    
    Learn More About WordPress SEO<\/h4>\n                    
     WordPress SEO Tips<\/a>
    \n    How to Create a Sitemap in WordPress<\/a>
    \n    How to Add WordPress Schema Markup<\/a>
    \n    How to Add Meta Description and Meta Title in WordPress<\/a>
    \n    How to Setup AMP (Accelerated Mobile Pages) in WordPress<\/a>
    \n    How to Prevent Search Engines from Indexing Your WordPress Site<\/a>
    \n<\/a>    How to Add Breadcrumbs in WordPress<\/a> <\/p>\n                <\/div>\n\n\n\n<\/p>
    WordPress robots.txt FAQ<\/h2>
    Here are some frequently asked questions about WordPress robots.txt.<\/p>
    Can I Disable robots.txt in WordPress?<\/h3> 
    No, you cannot disable robots.txt in WordPress as it is a standard file used to communicate with search engines and instructs them which pages to crawl or not. However, you can modify the content of the file to control how search engines interact with your site.<\/p> <\/div> 
    How Do I Optimize My WordPress Site’s robots.txt?<\/h3> 
    To optimize your WordPress site’s robots.txt, you can use plugins like Yoast SEO<\/strong> to generate a robots.txt file with recommended settings. You can also modify the file manually to control which pages and directories to block or allow search engines to crawl.<\/p> <\/div> <\/div>\n","protected":false},"excerpt":{"rendered":"
    To ensure that your site ranks highly in Search Engine Result Pages (SERPs), you’ll need to make it easy for search engine ‘bots’ to explore its most important pages. Having a well-structured robots.txt file in place will help direct those bots to the pages you want them to index (and avoid the rest). In this article, we’re […]<\/p>\n
    Read More…<\/a><\/p>\n","protected":false},"author":76,"featured_media":145316,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_title":"The Complete Guide to WordPress robots.txt (And How to Use it for SEO)","rank_math_description":"Learn how to leverage your WordPress robots.txt file to improve your SEO and prevent unnecessary bots from making requests on your website.","rank_math_focus_keyword":"wordpress robots.txt, robots.txt for wordpress, wordpress robots.txt location, wordpress robots.txt example, edit robots.txt in wordpress","footnotes":""},"categories":[22637],"tags":[],"class_list":["post-9744","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-wordpress"],"hreflangs":[{"locale":"en-US","link":"https:\/\/www.hostinger.com\/tutorials\/wordpress-robots-txt","default":0},{"locale":"pt-BR","link":"https:\/\/www.hostinger.com\/br\/tutoriais\/arquivos-robots-txt-wordpress","default":0},{"locale":"fr-FR","link":"https:\/\/www.hostinger.com\/fr\/tutoriels\/robots-txt-wordpress","default":0},{"locale":"es-ES","link":"https:\/\/www.hostinger.com\/es\/tutoriales\/guia-completa-robots-txt-wordpress","default":0},{"locale":"id-ID","link":"https:\/\/www.hostinger.com\/id\/tutorial\/cara-setting-robot-txt-di-wordpress","default":0},{"locale":"en-UK","link":"https:\/\/www.hostinger.com\/uk\/tutorials\/wordpress-robots-txt","default":0},{"locale":"en-MY","link":"https:\/\/www.hostinger.com\/my\/tutorials\/wordpress-robots-txt","default":0},{"locale":"en-PH","link":"https:\/\/www.hostinger.com\/ph\/tutorials\/wordpress-robots-txt","default":0},{"locale":"es-MX","link":"https:\/\/www.hostinger.com\/mx\/tutoriales\/guia-completa-robots-txt-wordpress","default":0},{"locale":"es-CO","link":"https:\/\/www.hostinger.com\/co\/tutoriales\/guia-completa-robots-txt-wordpress","default":0},{"locale":"es-AR","link":"https:\/\/www.hostinger.com\/ar\/tutoriales\/guia-completa-robots-txt-wordpress","default":0},{"locale":"pt-PT","link":"https:\/\/www.hostinger.com\/pt\/tutoriais\/arquivos-robots-txt-wordpress","default":0},{"locale":"en-IN","link":"https:\/\/www.hostinger.com\/in\/tutorials\/wordpress-robots-txt","default":0},{"locale":"en-CA","link":"https:\/\/www.hostinger.com\/ca\/tutorials\/wordpress-robots-txt","default":0},{"locale":"en-AU","link":"https:\/\/www.hostinger.com\/au\/tutorials\/wordpress-robots-txt","default":0},{"locale":"en-NG","link":"https:\/\/www.hostinger.com\/ng\/tutorials\/wordpress-robots-txt","default":0}],"_links":{"self":[{"href":"https:\/\/www.hostinger.com\/ng\/tutorials\/wp-json\/wp\/v2\/posts\/9744","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.hostinger.com\/ng\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hostinger.com\/ng\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hostinger.com\/ng\/tutorials\/wp-json\/wp\/v2\/users\/76"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hostinger.com\/ng\/tutorials\/wp-json\/wp\/v2\/comments?post=9744"}],"version-history":[{"count":27,"href":"https:\/\/www.hostinger.com\/ng\/tutorials\/wp-json\/wp\/v2\/posts\/9744\/revisions"}],"predecessor-version":[{"id":145315,"href":"https:\/\/www.hostinger.com\/ng\/tutorials\/wp-json\/wp\/v2\/posts\/9744\/revisions\/145315"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.hostinger.com\/ng\/tutorials\/wp-json\/wp\/v2\/media\/145316"}],"wp:attachment":[{"href":"https:\/\/www.hostinger.com\/ng\/tutorials\/wp-json\/wp\/v2\/media?parent=9744"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hostinger.com\/ng\/tutorials\/wp-json\/wp\/v2\/categories?post=9744"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hostinger.com\/ng\/tutorials\/wp-json\/wp\/v2\/tags?post=9744"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}