{"id":8073,"date":"2025-09-04T10:07:21","date_gmt":"2025-09-04T10:07:21","guid":{"rendered":"https:\/\/www.hostinger.com\/blog\/?p=8073"},"modified":"2025-09-04T10:07:49","modified_gmt":"2025-09-04T10:07:49","slug":"cdn-ai-audit","status":"publish","type":"post","link":"https:\/\/www.hostinger.com\/blog\/cdn-ai-audit","title":{"rendered":"Your website, your rules: Track and control AI bots"},"content":{"rendered":"<p><strong>Artificial intelligence (AI)<\/strong> bots now scrape websites to train models and answer questions directly, often without sending traffic back to the original sources. Our new <strong>AI audit<\/strong> feature puts control in your hands: <strong>you decide which AI bots are allowed on your website &ndash; and which aren&rsquo;t<\/strong>.<\/p><h2 class=\"wp-block-heading\" id=\"h-the-new-era-of-web-crawlers\">The new era of web crawlers<\/h2><p>For years, website owners welcomed bots, also known as crawlers, from <strong>search engines<\/strong>, as they <strong>improved visibility and helped monetize traffic<\/strong>.<\/p><p><strong>AI crawlers<\/strong> are now changing the model. They collect content not just for indexing, but to deliver <strong>instant answers directly within AI tools<\/strong>.<\/p><p>For many businesses, this creates new opportunities. Mentions in AI responses can expand reach and attract new audiences.<\/p><p>For others, especially those monetizing through visits, it creates challenges. If an AI tool provides the full answer, only a few readers will click through. No traffic means no income.<\/p><h2 class=\"wp-block-heading\" id=\"h-be-in-charge-with-ai-audit\">Be in charge with AI audit<\/h2><p>With <strong>AI audit<\/strong>, the choice is yours. Keep the crawlers that bring value. Block the ones that don&rsquo;t. Adjust your settings anytime.<\/p><p>The feature is available to web hosting and cloud hosting clients using our <strong>content delivery network (CDN)<\/strong>. You can find it in your website&rsquo;s dashboard under <strong>Performance &rarr; CDN &rarr; AI Audit<\/strong>.<\/p><div class=\"wp-block-image\">\n<figure data-wp-context='{\"imageId\":\"69e7607870c9e\"}' data-wp-interactive=\"core\/image\" class=\"aligncenter size-large wp-lightbox-container\"><img decoding=\"async\" width=\"1200\" height=\"765\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/hpanel-cdn-ai-audit-highlighted.png\/public\" alt=\"CDN dashboard in hPanel with AI audit highlighted.\" class=\"wp-image-8077\" srcset=\"https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/hpanel-cdn-ai-audit-highlighted.png\/w=1200,fit=scale-down 1200w, https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/hpanel-cdn-ai-audit-highlighted.png\/w=300,fit=scale-down 300w, https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/hpanel-cdn-ai-audit-highlighted.png\/w=1024,fit=scale-down 1024w, https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/hpanel-cdn-ai-audit-highlighted.png\/w=768,fit=scale-down 768w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" \/><button class=\"lightbox-trigger\" type=\"button\" aria-haspopup=\"dialog\" aria-label=\"Enlarge\" data-wp-init=\"callbacks.initTriggerButton\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-style--right=\"state.imageButtonRight\" data-wp-style--top=\"state.imageButtonTop\">\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewbox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\"><\/path>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure><\/div><p>From there, you can:<\/p><ul class=\"wp-block-list\">\n<li><strong>See<\/strong> which AI crawlers visit your site<\/li>\n\n\n\n<li><strong>Track<\/strong> crawler activity and requests over time<\/li>\n\n\n\n<li><strong>Block<\/strong> the bots you don&rsquo;t want, while keeping the ones you do<\/li>\n<\/ul><p>Note: Blocking a crawler prevents future access, but doesn&rsquo;t erase what it already collected.<\/p><h2 class=\"wp-block-heading\" id=\"h-our-analysis-how-ai-bots-are-crawling-the-web\">Our analysis: How AI bots are crawling the web<\/h2><p>How active are these AI crawlers? To find out, we analyzed logs from <strong>5 million client websites<\/strong>.<\/p><p>Here&rsquo;s what we discovered:<\/p><ul class=\"wp-block-list\">\n<li><strong>Hundreds of crawler groups<\/strong> visit our clients&rsquo; websites<\/li>\n\n\n\n<li>About<strong> a third<\/strong> of the most active crawlers are<strong> linked to AI providers<\/strong><\/li>\n\n\n\n<li>Crawlers from <strong>Google<\/strong>, <strong>Meta<\/strong>, and <strong>OpenAI<\/strong> are the most active, reaching between <strong>60 and 85% of websites daily<\/strong><\/li>\n\n\n\n<li>Bots from <strong>Apple, ByteDance (TikTok)<\/strong>, and<strong> Huawei (Petalsearch)<\/strong> typically crawl <strong>20&ndash;25% of sites<\/strong><\/li>\n\n\n\n<li><strong>Anthropic (Claude), Amazon, <\/strong>and <strong>Perplexity<\/strong>, and other AI crawlers visit only <strong>2-5%<\/strong><\/li>\n<\/ul><p>Don&rsquo;t let the lower percentages fool you &ndash; crawlers <strong>rotate targets<\/strong> and can build a near-complete picture of the web <strong>within weeks<\/strong>.<\/p><div class=\"wp-block-image\">\n<figure data-wp-context='{\"imageId\":\"69e7607871a91\"}' data-wp-interactive=\"core\/image\" class=\"aligncenter size-large wp-lightbox-container\"><img decoding=\"async\" width=\"1110\" height=\"1210\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/website-coverage-by-ai-crawlers.png\/public\" alt=\"Website coverage by AI crawlers over time.\" class=\"wp-image-8078\" srcset=\"https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/website-coverage-by-ai-crawlers.png\/w=1110,fit=scale-down 1110w, https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/website-coverage-by-ai-crawlers.png\/w=275,fit=scale-down 275w, https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/website-coverage-by-ai-crawlers.png\/w=940,fit=scale-down 940w, https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/website-coverage-by-ai-crawlers.png\/w=768,fit=scale-down 768w, https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/website-coverage-by-ai-crawlers.png\/w=1409,fit=scale-down 1409w, https:\/\/imagedelivery.net\/LqiWLm-3MGbYHtFuUbcBtA\/wp-content\/uploads\/sites\/4\/2025\/09\/website-coverage-by-ai-crawlers.png\/w=1879,fit=scale-down 1879w\" sizes=\"(max-width: 1110px) 100vw, 1110px\" \/><button class=\"lightbox-trigger\" type=\"button\" aria-haspopup=\"dialog\" aria-label=\"Enlarge\" data-wp-init=\"callbacks.initTriggerButton\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-style--right=\"state.imageButtonRight\" data-wp-style--top=\"state.imageButtonTop\">\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewbox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\"><\/path>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure><\/div><p>To make crawler activity clearer, here are some examples of <strong>common AI bots<\/strong>:<\/p><ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/developer.amazon.com\/amazonbot\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Amazonbot (Amazon)<\/strong><\/a> &ndash; Indexes web content for various purposes, including support for Amazon Alexa<\/li>\n\n\n\n<li><a href=\"https:\/\/support.apple.com\/en-us\/119829\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Applebot (Apple)<\/strong><\/a> &ndash; Crawled data helps power Apple&rsquo;s features, including the company&rsquo;s AI assistant Siri<\/li>\n\n\n\n<li><a href=\"https:\/\/platform.openai.com\/docs\/bots\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Openai-GPTBot (OpenAI)<\/strong><\/a> &ndash; Crawls content that may be used in training generative AI foundation models<\/li>\n\n\n\n<li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/google-common-crawlers\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Googlebot (Google)<\/strong><\/a> &ndash; Google&rsquo;s main crawler. While its official description doesn&rsquo;t mention AI training, public data and evidence suggest it may be used for that purpose<\/li>\n\n\n\n<li><a href=\"https:\/\/developers.facebook.com\/docs\/sharing\/webmasters\/web-crawlers\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Meta-Externalagent (Meta)<\/strong><\/a> &ndash; Crawls the web for use cases such as training AI models or improving products<\/li>\n<\/ul><h2 class=\"wp-block-heading\" id=\"h-ready-to-take-control\">Ready to take control?<\/h2><p>The internet is changing fast, but your website is still your space. With AI audit, our CDN extension, you decide who can access your content &ndash; aligning visibility with your business goals.<\/p><p>Take control of your content today with the AI audit.<\/p><p>And if you want to increase visibility in AI-generated responses, consider adopting <a href=\"https:\/\/www.hostinger.com\/tutorials\/generative-search-engine-optimization\" target=\"_blank\" rel=\"noreferrer noopener\">generative engine optimization<\/a> (GEO) strategies: create well-structured content, <a href=\"https:\/\/www.hostinger.com\/tutorials\/website-speed-optimization\" target=\"_blank\" rel=\"noreferrer noopener\">improve website loading speed<\/a>, and add an <a href=\"https:\/\/www.hostinger.com\/tutorials\/what-is-llms-txt\" target=\"_blank\" rel=\"noreferrer noopener\">llms.txt<\/a> file to make it easier for AI tools to navigate your site.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Artificial intelligence (AI) bots now scrape websites to train models and answer questions directly, often without sending traffic back to the original sources. Our new AI audit f\u2026<\/p>\n","protected":false},"author":405,"featured_media":8079,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2391],"tags":[],"hashtags":[],"class_list":["post-8073","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-product-updates"],"hreflangs":[],"_links":{"self":[{"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/posts\/8073","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/users\/405"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/comments?post=8073"}],"version-history":[{"count":2,"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/posts\/8073\/revisions"}],"predecessor-version":[{"id":8081,"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/posts\/8073\/revisions\/8081"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/media\/8079"}],"wp:attachment":[{"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/media?parent=8073"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/categories?post=8073"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/tags?post=8073"},{"taxonomy":"hashtags","embeddable":true,"href":"https:\/\/www.hostinger.com\/blog\/wp-json\/wp\/v2\/hashtags?post=8073"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}