{"id":21275,"date":"2026-10-22T02:09:59","date_gmt":"2026-10-22T02:09:59","guid":{"rendered":"https:\/\/www.hostinger.com\/support\/?p=21275"},"modified":"2026-10-22T02:13:13","modified_gmt":"2026-10-22T02:13:13","slug":"how-to-allow-or-block-ai-crawlers-on-your-hostinger-website","status":"publish","type":"post","link":"https:\/\/www.hostinger.com\/support\/how-to-allow-or-block-ai-crawlers-on-your-hostinger-website\/","title":{"rendered":"How to allow or block AI crawlers on your Hostinger website"},"content":{"rendered":"<p>To allow or block AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot on a Hostinger website, add rules to the <code>robots.txt<\/code> file in your website&rsquo;s root folder. On web and cloud hosting, you create or edit <code>robots.txt<\/code> with File Manager. Hostinger AI Builder creates <code>robots.txt<\/code> automatically, allows all crawlers by default, and doesn&rsquo;t support custom rules. After making changes, check the result with the Hostinger AI Visibility Checker.<\/p><h3><b>Before you start<\/b><\/h3><ul>\n<li>Editing <code>robots.txt<\/code> requires File Manager access, available on web and cloud hosting plans.<\/li>\n<li>Hostinger AI Builder websites, in manual and agentic mode, can&rsquo;t edit <code>robots.txt<\/code>.<\/li>\n<\/ul><h2 id=\"h-how-robots-txt-controls-ai-crawlers\">How robots.txt controls AI crawlers<\/h2><p>The <code>robots.txt<\/code> file tells crawlers which parts of your website they may access. It&rsquo;s always located at <code>domain.tld\/robots.txt<\/code>. Each AI crawler identifies itself with a user agent name, for example <code>GPTBot<\/code> for OpenAI.<\/p><p>A rule group starts with a <code>User-agent<\/code> line and lists what that crawler may access. <code>Allow: \/<\/code> permits the whole website, and <code>Disallow: \/<\/code> blocks it. A <code>Content-Signal<\/code> line states how your content may be used, for example in search results but not for AI training.<\/p><h2 id=\"h-create-or-edit-robots-txt-on-hostinger-web-and-cloud-hosting\">Create or edit robots.txt on Hostinger web and cloud hosting<\/h2><p>Create a <code>robots.txt<\/code> file in the <code>public_html<\/code> folder to control AI crawler access. This works for WordPress and other websites, and the file replaces any <code>robots.txt<\/code> that WordPress generates automatically.<\/p><ol>\n<li>Open <a data-click-id=\"hgr-post-content-link-the-hostinger-file-manager\" href=\"https:\/\/www.hostinger.com\/support\/4548688-basic-actions-in-the-file-manager-in-hostinger\/\">the Hostinger File Manager<\/a> for your website.<\/li>\n<li>Open the <code>public_html<\/code> folder.<\/li>\n<li>Open <code>robots.txt<\/code> if it exists. Otherwise, click <strong>N<\/strong><b>ew file<\/b>&nbsp;and name the new file <code>robots.txt<\/code>.<\/li>\n<li>Add your rules. See the examples in the next section.<\/li>\n<li>Click the save icon in the top-right corner of the editor.<\/li>\n<li>Open <code>domain.tld\/robots.txt<\/code> in your browser to confirm the new rules appear.<\/li>\n<\/ol><p>You can also <a data-click-id=\"hgr-post-content-link-edit-website-files-using-the-hostinger-file-editor\" href=\"https:\/\/www.hostinger.com\/support\/7949267-how-to-use-the-file-editor-at-hostinger\/\">edit website files using the Hostinger File Editor<\/a>.<\/p><p><b>NOTES<\/b><\/p><ul>\n<li>Start each line at the beginning, without spaces. Some crawlers and tools don&rsquo;t read lines that start with a space.<\/li>\n<li>If <code>domain.tld\/robots.txt<\/code> returns a 404 error on a WordPress website, creating the file in <code>public_html<\/code> fixes it.<\/li>\n<\/ul><h2 id=\"h-robots-txt-examples-for-ai-crawlers\">robots.txt examples for AI crawlers<\/h2><p>Use these examples as a starting point and replace <code>domain.tld<\/code> with your domain.<\/p><h3>Allow all crawlers and declare content use<\/h3><p>This example allows all crawlers, permits search use, and asks crawlers not to use your content for AI training:<\/p><pre><code>User-agent: *\r\nAllow: \/\r\nContent-Signal: search=yes, ai-train=no<\/code><\/pre><h3>Block a specific AI crawler<\/h3><p>Add a separate rule group for each AI crawler you want to block. This example blocks GPTBot and allows all other crawlers:<\/p><pre><code>User-agent: *\r\nAllow: \/\r\n\r\nUser-agent: GPTBot\r\nDisallow: \/<\/code><\/pre><p>To block another AI crawler, replace <code>GPTBot<\/code> with its user agent from the <b>Common AI crawler user agents<\/b> table.<\/p><h3>Block all crawlers<\/h3><p>This example blocks all crawlers from your whole website, including AI crawlers and search engines such as Google:<\/p><pre><code>User-agent: * \r\nDisallow: \/<\/code><\/pre><p>Use this rule only for websites that shouldn&rsquo;t be crawled at all, for example a staging or private website. With this rule, the <b>AI crawlers are not blocked<\/b> check in the Hostinger AI Visibility Checker fails with 0\/7 points.<\/p><div class=\"intercom-interblocks-callout\" style=\"background-color: #feedaf80;border-color: #fbc91633\">\n<p><b>Important:<\/b> Blocking all crawlers also stops Google and other search engines from crawling your website.<\/p>\n<\/div><h3>Add your sitemap<\/h3><p>Add a <code>Sitemap<\/code> line so crawlers find all your pages, for example <code>Sitemap: https:\/\/domain.tld\/sitemap.xml<\/code>. Open the sitemap URL first and add the line only if the sitemap loads.<\/p><h2 id=\"h-common-ai-crawler-user-agents\">Common AI crawler user agents<\/h2><p>AI companies use different user agents for different purposes. For example, OpenAI uses <code>GPTBot<\/code> for AI training and <code>OAI-SearchBot<\/code> for ChatGPT search results.<\/p><div class=\"intercom-interblocks-table-container\">\n<table role=\"presentation\">\n<tbody>\n<tr>\n<td style=\"background-color: #e3e7fa80;width: 261px\">\n<p class=\"intercom-align-center no-margin\"><b>User agent<\/b><\/p>\n<\/td>\n<td style=\"background-color: #e3e7fa80\">\n<p class=\"intercom-align-center no-margin\"><b>Company<\/b><\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 261px\">\n<p class=\"no-margin\"><code>GPTBot<\/code><\/p>\n<\/td>\n<td>\n<p class=\"no-margin\">OpenAI<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 261px\">\n<p class=\"no-margin\"><code>OAI-SearchBot<\/code><\/p>\n<\/td>\n<td>\n<p class=\"no-margin\">OpenAI<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 261px\">\n<p class=\"no-margin\"><code>ChatGPT-User<\/code><\/p>\n<\/td>\n<td>\n<p class=\"no-margin\">OpenAI<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 261px\">\n<p class=\"no-margin\"><code>ClaudeBot<\/code><\/p>\n<\/td>\n<td>\n<p class=\"no-margin\">Anthropic<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 261px\">\n<p class=\"no-margin\"><code>PerplexityBot<\/code><\/p>\n<\/td>\n<td>\n<p class=\"no-margin\">Perplexity<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 261px\">\n<p class=\"no-margin\"><code>Google-Extended<\/code><\/p>\n<\/td>\n<td>\n<p class=\"no-margin\">Google<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 261px\">\n<p class=\"no-margin\"><code>CCBot<\/code><\/p>\n<\/td>\n<td>\n<p class=\"no-margin\">Common Crawl<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div><h2 id=\"h-robots-txt-on-hostinger-ai-builder-websites\">robots.txt on Hostinger AI Builder websites<\/h2><p><b>Hostinger AI Builder creates <code>robots.txt<\/code> automatically and allows all crawlers, including AI crawlers.<\/b> You can&rsquo;t add custom rules, such as blocking a specific AI crawler, in manual or agentic mode.<\/p><ul>\n<li><b>Manual mode<\/b> &ndash; the automatic <code>robots.txt<\/code> file allows all crawlers and includes your sitemap. No setting edits it.<\/li>\n<li><b>Agentic mode<\/b> &ndash; <code>robots.txt<\/code> is created after publishing on a custom domain, as described in <a data-click-id=\"hgr-post-content-link-hostinger-ai-builder-seo-settings-in-agentic-mode\" href=\"https:\/\/www.hostinger.com\/support\/10771376-hostinger-ai-builder-agentic-mode-seo-settings\/\">Hostinger AI Builder: SEO settings in agentic mode<\/a>. The file is managed by the platform and is read-only. If you ask the AI to block a crawler, the AI can&rsquo;t apply the rule to the live <code>robots.txt<\/code> file. Open <code>domain.tld\/robots.txt<\/code> to see the rules currently in effect.<\/li>\n<\/ul><h2 id=\"h-hostinger-cdn-settings-that-block-crawlers\">Hostinger CDN settings that block crawlers<\/h2><p>Hostinger CDN security settings can block crawlers even when <code>robots.txt<\/code> allows them. Under Attack mode and blocked bots are two examples. See <a data-click-id=\"hgr-post-content-link-hostinger-cdn-search-engine-crawlers-and-seo\" href=\"https:\/\/www.hostinger.com\/support\/hostinger-cdn-search-engine-crawlers-and-seo\/\">Hostinger CDN: Search engine crawlers and SEO<\/a> for the settings to review.<\/p><h2 id=\"h-check-your-robots-txt-with-the-hostinger-ai-visibility-checker\">Check your robots.txt with the Hostinger AI Visibility Checker<\/h2><p>Scan your website with <a data-click-id=\"hgr-post-content-link-the-hostinger-ai-visibility-checker\" href=\"https:\/\/www.hostinger.com\/support\/how-to-use-the-hostinger-ai-visibility-checker\/\">the Hostinger AI Visibility Checker<\/a> after changing <code>robots.txt<\/code>. The <b>Crawl permission<\/b> category shows four checks:<\/p><ul>\n<li><b>robots.txt is present and parseable<\/b>.<\/li>\n<li><b>AI crawlers are not blocked<\/b>.<\/li>\n<li><b>Not blocked by bot detection<\/b>.<\/li>\n<li><b>Content Signals declared<\/b>.<\/li>\n<\/ul><p>To confirm a block on a specific AI crawler, open <code>domain.tld\/robots.txt<\/code> and check that its rule group is in the file.<\/p><p>Your <code>robots.txt<\/code> file now tells AI crawlers which parts of your website they can access and how they can use your content. Review it whenever you want to change which AI tools can read your website.<\/p><h2 id=\"h-faq\">FAQ<\/h2><p><b>Does robots.txt guarantee that AI crawlers stop accessing my website?<\/b><\/p><p>No. The <code>robots.txt<\/code> file is a request that crawlers choose to follow. Major AI companies, such as OpenAI and Anthropic, state that their crawlers respect <code>robots.txt<\/code> rules.<\/p><p><b>Can I block AI crawlers on a Hostinger AI Builder website?<\/b><\/p><p>No. Hostinger AI Builder manages <code>robots.txt<\/code> automatically and allows all crawlers. Custom rules aren&rsquo;t supported in manual or agentic mode.<\/p><p><b>What does the Content-Signal line do?<\/b><\/p><p>The <code>Content-Signal<\/code> line states how your content may be used. For example, <code>search=yes, ai-train=no<\/code> permits search results but asks crawlers not to use your content for AI training.<\/p><p><b>Additional resources<\/b><\/p><ul>\n<li><a data-click-id=\"hgr-post-content-link-wordpress-robots-txt-tutorial\" href=\"https:\/\/www.hostinger.com\/tutorials\/wordpress-robots-txt\/\">WordPress robots.txt tutorial<\/a>.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Control which AI crawlers, such as GPTBot and ClaudeBot, can access your website.<\/p>\n","protected":false},"author":594,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"include_on_kodee":true,"footnotes":""},"categories":[283],"tags":[],"class_list":["post-21275","post","type-post","status-publish","format-standard","hentry","category-seo-website-builder"],"hreflangs":[],"include_on_kodee":true,"_links":{"self":[{"href":"https:\/\/www.hostinger.com\/support\/wp-json\/wp\/v2\/posts\/21275","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.hostinger.com\/support\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hostinger.com\/support\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hostinger.com\/support\/wp-json\/wp\/v2\/users\/594"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hostinger.com\/support\/wp-json\/wp\/v2\/comments?post=21275"}],"version-history":[{"count":3,"href":"https:\/\/www.hostinger.com\/support\/wp-json\/wp\/v2\/posts\/21275\/revisions"}],"predecessor-version":[{"id":21279,"href":"https:\/\/www.hostinger.com\/support\/wp-json\/wp\/v2\/posts\/21275\/revisions\/21279"}],"wp:attachment":[{"href":"https:\/\/www.hostinger.com\/support\/wp-json\/wp\/v2\/media?parent=21275"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hostinger.com\/support\/wp-json\/wp\/v2\/categories?post=21275"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hostinger.com\/support\/wp-json\/wp\/v2\/tags?post=21275"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}