How to allow or block AI crawlers on your Hostinger website

To allow or block AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot on a Hostinger website, add rules to the robots.txt file in your website’s root folder. On web and cloud hosting, you create or edit robots.txt with File Manager. Hostinger AI Builder creates robots.txt automatically, allows all crawlers by default, and doesn’t support custom rules. After making changes, check the result with the Hostinger AI Visibility Checker.

Before you start

• Editing robots.txt requires File Manager access, available on web and cloud hosting plans.
• Hostinger AI Builder websites, in manual and agentic mode, can’t edit robots.txt.

How robots.txt controls AI crawlers

The robots.txt file tells crawlers which parts of your website they may access. It’s always located at domain.tld/robots.txt. Each AI crawler identifies itself with a user agent name, for example GPTBot for OpenAI.

A rule group starts with a User-agent line and lists what that crawler may access. Allow: / permits the whole website, and Disallow: / blocks it. A Content-Signal line states how your content may be used, for example in search results but not for AI training.

Create or edit robots.txt on Hostinger web and cloud hosting

Create a robots.txt file in the public_html folder to control AI crawler access. This works for WordPress and other websites, and the file replaces any robots.txt that WordPress generates automatically.

1. Open the Hostinger File Manager for your website.
2. Open the public_html folder.
3. Open robots.txt if it exists. Otherwise, click New file and name the new file robots.txt.
4. Add your rules. See the examples in the next section.
5. Click the save icon in the top-right corner of the editor.
6. Open domain.tld/robots.txt in your browser to confirm the new rules appear.

You can also edit website files using the Hostinger File Editor.

NOTES

• Start each line at the beginning, without spaces. Some crawlers and tools don’t read lines that start with a space.
• If domain.tld/robots.txt returns a 404 error on a WordPress website, creating the file in public_html fixes it.

robots.txt examples for AI crawlers

Use these examples as a starting point and replace domain.tld with your domain.

Allow all crawlers and declare content use

This example allows all crawlers, permits search use, and asks crawlers not to use your content for AI training:

User-agent: *
Allow: /
Content-Signal: search=yes, ai-train=no

Block a specific AI crawler

Add a separate rule group for each AI crawler you want to block. This example blocks GPTBot and allows all other crawlers:

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

To block another AI crawler, replace GPTBot with its user agent from the Common AI crawler user agents table.

Block all crawlers

This example blocks all crawlers from your whole website, including AI crawlers and search engines such as Google:

User-agent: * 
Disallow: /

Use this rule only for websites that shouldn’t be crawled at all, for example a staging or private website. With this rule, the AI crawlers are not blocked check in the Hostinger AI Visibility Checker fails with 0/7 points.

Add your sitemap

Add a Sitemap line so crawlers find all your pages, for example Sitemap: https://domain.tld/sitemap.xml. Open the sitemap URL first and add the line only if the sitemap loads.

Common AI crawler user agents

AI companies use different user agents for different purposes. For example, OpenAI uses GPTBot for AI training and OAI-SearchBot for ChatGPT search results.

robots.txt on Hostinger AI Builder websites

Hostinger AI Builder creates robots.txt automatically and allows all crawlers, including AI crawlers. You can’t add custom rules, such as blocking a specific AI crawler, in manual or agentic mode.

• Manual mode – the automatic robots.txt file allows all crawlers and includes your sitemap. No setting edits it.
• Agentic mode – robots.txt is created after publishing on a custom domain, as described in Hostinger AI Builder: SEO settings in agentic mode. The file is managed by the platform and is read-only. If you ask the AI to block a crawler, the AI can’t apply the rule to the live robots.txt file. Open domain.tld/robots.txt to see the rules currently in effect.

Hostinger CDN settings that block crawlers

Hostinger CDN security settings can block crawlers even when robots.txt allows them. Under Attack mode and blocked bots are two examples. See Hostinger CDN: Search engine crawlers and SEO for the settings to review.

Check your robots.txt with the Hostinger AI Visibility Checker

Scan your website with the Hostinger AI Visibility Checker after changing robots.txt. The Crawl permission category shows four checks:

• robots.txt is present and parseable.
• AI crawlers are not blocked.
• Not blocked by bot detection.
• Content Signals declared.

To confirm a block on a specific AI crawler, open domain.tld/robots.txt and check that its rule group is in the file.

Your robots.txt file now tells AI crawlers which parts of your website they can access and how they can use your content. Review it whenever you want to change which AI tools can read your website.

FAQ

Does robots.txt guarantee that AI crawlers stop accessing my website?

No. The robots.txt file is a request that crawlers choose to follow. Major AI companies, such as OpenAI and Anthropic, state that their crawlers respect robots.txt rules.

Can I block AI crawlers on a Hostinger AI Builder website?

No. Hostinger AI Builder manages robots.txt automatically and allows all crawlers. Custom rules aren’t supported in manual or agentic mode.

What does the Content-Signal line do?

The Content-Signal line states how your content may be used. For example, search=yes, ai-train=no permits search results but asks crawlers not to use your content for AI training.

Additional resources

• WordPress robots.txt tutorial.