LLM SEO Index Crawler Check

LLM SEO Index let's you check if your website(s) are visible to AI search assistants like ChatGPT, Perplexity, Claude as well as to traditional search engines like Google and Bing.

I recently noticed that the robots.txt for one of my websites actually stopped ChatGPT from crawling the website. Why? A few days ago I accidentally enabled a bot crawling feature on a web hosting platform that was configured stricter than I expected.

That's why we built LLM SEO Index, to easily identify problems like these.

We currently cover:

• ChatGPT

• Perplexity

• Claude

• Amazon (Alexa)

• Apple (Siri)

• Meta AI

and

• Google

• Microsoft Bing

• DuckDuckGo

• Yandex

• Brave

With more and more users going directly to ChatGPT to get answers and recommendations it matters a lot whether AI search assistants can crawl your websites or not.

Unfortunately many websites block ChatGPT and other bots by accident. Serving of websites has become more and more complex in the last few days and often many different systems have to play well together to make sure your web properties are actually crawlable the way you want them to be crawlable.

With LLM SEO Index we built a simple check that you can run for your website(s) or for websites of your clients to get a fast high level result that helps you to dig deeper if there is anything unexpected going on.

We want to make this as good as it possibly can be. Any feedback you have is very welcome!

Some top publishers blocking AI and SEO bots signals a strategic shift in controlling content access. As AI-driven search grows, website owners must rethink their crawling policies to capture new traffic sources beyond Google. An awesome tool as expected. Carry on guys.

Nice job team, being able to quickly check if ChatGPT, Claude, and Google can crawl a site in just 10 seconds is super handy for spotting robots.txt issues!