Free Tool · LLM Visibility
Paste any URL. The tool fetches your robots.txt and the page itself, then runs the exact decision path that GPTBot, ClaudeBot, PerplexityBot, Googlebot, and others would follow. When a bot is blocked, you see the exact rule and line number that did it.
10 requests per minute per IP · Always fresh, no caching · Fail-closed on 5xx robots.txt
Enter a domain or path — e.g. avinashvagh.com or avinashvagh.com/blog.
/robots.txt from the URL’s origin and parse it per RFC 9309 (groups, longest-match, Allow wins ties). User-agent group, then run the URL path against its Allow / Disallow rules. X-Robots-Tag response header and <meta name="robots"> tag for noindex directives.X-Robots-Tag noindex header is silently keeping the page out of indexes.robots.txt that triggered each block — so you can fix it in seconds.Eight categories covering search engines, AI training and search bots, social previews, SEO crawlers, scrapers, cloud-provider AI grounding, Google’s specialised bots, and archive/research agents.
Traditional search engine crawlers. Blocking these removes you from organic search.
AI training and live-retrieval bots. Allow these to be cited in AI answers; block them to keep your content out of model training.
Generate link previews on social networks and chat apps when your URL is shared.
Third-party SEO crawlers and audit tools used by competitors and agencies.
Aggregators, datasets, and general-purpose scraping frameworks.
Cloud-native AI grounding and security crawlers from major cloud providers.
Google's specialized crawlers: images, video, news, inspection, and AI Overviews.
Archives, plagiarism checkers, and niche crawlers that don't fit the other buckets.
Paste any URL above. The tool fetches your origin’s /robots.txt, parses it per RFC 9309, and runs the URL path against each bot’s product token. If a Disallow rule matches, the report shows you the exact line — including which User-agent group it came from — so you can edit the right line.
Googlebot crawls for Google Search results. Google-Extended is a separate signal that opts your content in or out of Gemini training and grounded AI answers. You can allow one and block the other — this tool shows both verdicts side by side.
Yes. After Gate 1 (robots.txt), the tool fetches the page and inspects the X-Robots-Tag response header (including per-bot scoped directives like googlebot: noindex) and the page’s <meta name="robots"> tag. If a bot is allowed by robots.txt but blocked by a header or meta tag, you’ll see that flagged.
Three common causes: a wildcard Disallow under User-agent: * that catches more URLs than intended, a more-specific group that overrides your default Allow, or a 5xx error on /robots.txt itself — major search engines treat a 5xx as “block everything”. The report surfaces the exact rule and line number so you can pinpoint the cause.