{"id":16168,"date":"2026-08-20T01:44:55","date_gmt":"2026-08-20T01:44:55","guid":{"rendered":"https:\/\/www.hostinger.com\/support\/?p=16168"},"modified":"2026-08-20T02:24:01","modified_gmt":"2026-08-20T02:24:01","slug":"hostinger-cdn-search-engine-crawlers-and-seo","status":"publish","type":"post","link":"https:\/\/www.hostinger.com\/support\/hostinger-cdn-search-engine-crawlers-and-seo\/","title":{"rendered":"Hostinger CDN: Search engine crawlers and SEO"},"content":{"rendered":"

Hostinger CDN allows verified search engine crawlers, such as Googlebot and Bingbot, to access your website without browser verification. If Google Search Console reports “robots.txt unreachable” or crawl errors with 403 or 429 status codes, check four settings: the Under Attack mode, country and IP blocking rules, blocked bots, and your crawl rate.<\/p>

How Hostinger CDN identifies search engine crawlers<\/h2>

Hostinger CDN<\/a> verifies crawlers using the official IP address lists published by search engines and reverse DNS (Domain Name System) checks. Verified crawlers are not shown the browser verification page under normal settings, so enabling Hostinger CDN does not harm indexing or SEO (Search Engine Optimization).<\/p>

Requests that only pretend to be Googlebot or Bingbot fail this verification. Blocking fake crawlers protects your website from scrapers and does not affect search rankings.<\/p>

Why search engine crawlers get blocked by Hostinger CDN<\/h2>

There are four common reasons a search engine crawler receives errors through Hostinger CDN: Under Attack mode, country blocking, bot blocking rules, and crawl-rate limits.<\/p>

Under Attack mode challenges all visitors, including crawlers<\/h3>

The Under Attack mode<\/a> makes Hostinger CDN challenge every visitor with a browser verification page. Search engine crawlers do not execute the verification, so they receive errors while the mode is active. Keep Under Attack mode enabled only during an active attack:<\/p>

    \n
  1. Go to hPanel<\/a> → Websites → Dashboard → Performance → CDN<\/strong>.
    \n\"\"<\/li>\n
  2. Click Manage<\/strong> and open the Security tab<\/strong>.
    \n\"\"<\/li>\n
  3. Change the security level to Medium<\/strong> or Low <\/strong>once the attack has stopped.<\/li>\n<\/ol>

    \"\"<\/p>

    Country blocking can block crawlers<\/h3>

    Googlebot crawls primarily from IP addresses located in the United States, so blocking the United States in the Traffic blocking tab also blocks Googlebot. The same applies to allow-only setups that omit the crawler’s country. Search engines publish the IP ranges their crawlers use, so compare your country-blocking rules<\/a> against the published crawler locations, then remove or adjust rules that cover them.<\/p>

    Bot blocking rules<\/h3>

    Bots blocked explicitly in your CDN settings receive a 403 response<\/a>, even when they are verified. Review your blocked bots configuration and unblock search engine crawlers that you want to index the website.<\/p>

    Crawl bursts can hit rate limits<\/h3>

    Rapid bursts of requests to pages that are not cached, such as large XML sitemaps, can reach the rate limits that protect your website and return a 429 status code<\/a>. Confirm the 429 response came through Hostinger CDN before changing any crawler settings: run a live URL test and check the response headers for `x-hcdn-request-id`<\/code> — when the header is missing, the response came from another layer, such as the website itself. Occasional 429 responses during bursts are normal and crawlers retry automatically.<\/p>

    If Bing reports persistent 429 errors, lower the crawl rate in Bing Webmaster Tools<\/strong> under Configure My Site → Crawl Control<\/a><\/strong>. Googlebot adjusts its crawl rate automatically.<\/p>

    How to verify crawler access to your website<\/h2>

    Test crawler access after each settings change:<\/p>