Know a great AI tool or AI news source? Submit it to AiByts

What AI crawlers fetch from a small open-source site: 31 days of logs

We checked every request that claimed to be GPTBot, ClaudeBot, PerplexityBot and the rest against the operators' published IP lists. Then we looked at what the real ones read. The data is downloadable.

By Relic TeamRelic Blog
What AI crawlers fetch from a small open-source site: 31 days of logs

Read the full story

Originally published by Relic Blog on Sep 14, 2026.

Read on Relic Blog(opens in a new tab)

Comments

Sign in to join the discussion.

No comments yet — be the first to share your thoughts.