Radar - Bot & Crawler Insights

Web crawlers insights

Radar now offers expanded insights into web crawlers, giving you greater visibility into aggregated trends in crawl and refer activity.

We have introduced the following endpoints:

• /bots/crawlers/summary/{dimension}: Returns an overview of crawler HTTP request distributions across key dimensions.
• /bots/crawlers/timeseries_groups/{dimension}: Provides time-series data on crawler request distributions across the same dimensions.

These endpoints allow analysis across the following dimensions:

• user_agent: Parsed data from the User-Agent header.
• referer: Parsed data from the Referer header.
• crawl_refer_ratio: Ratio of HTML page crawl requests to HTML page referrals by platform.

Broader bot insights

In addition to crawler-specific insights, Radar now provides a broader set of bot endpoints:

• /bots/: Lists all bots.
• /bots/{bot_slug}: Returns detailed metadata for a specific bot.
• /bots/timeseries: Time-series data for bot activity.
• /bots/summary/{dimension}: Returns an overview of bot HTTP request distributions across key dimensions.
• /bots/timeseries_groups/{dimension}: Provides time-series data on bot request distributions across the same dimensions.

These endpoints support filtering and breakdowns by:

• bot: Bot name.
• bot_operator: The organization or entity operating the bot.
• bot_category: Classification of bot type.

The previously available verified_bots endpoints have now been deprecated in favor of this set of bot insights APIs. While current data still focuses on verified bots, we plan to expand support for unverified bot traffic in the future.

Learn more about the new Radar bot and crawler insights in our blog post ↗.