Known Agents AI & Bot Traffic Trends - November 2025

Known Agents lets websites track and control their bot traffic with features like Automatic Robots.txt, Agent Analytics, and LLM Referral Tracking. You can connect your website using any CDN, backend, or the WordPress plugin, for free.

Why You Should Care

With the AI wave, understanding and optimizing for bot traffic has become a baseline requirement for any online business that wants to stay competitive. AI search crawlers determine whether your website appears in AI-powered search results, AI assistants are gathering intelligence on your brand, and AI agents are booking reservations and making purchases on behalf of real customers. Maintaining visibility and control over bots scraping your content is essential to protect your IP.

These reports are designed to help you navigate this transition successfully by revealing underlying AI traffic trends that remain a blind spot for everyone else. The data comes from thousands of websites already leveraging Agent Analytics to their advantage, representing a diverse cross-section of industries and traffic levels. Sources and methodology can be found in the appendix.

This Month's Highlights

• Bots are now 38% of all website traffic. Of that, 29% is related to AI use cases such as data collection for model training (AI Data Scrapers), indexing for search results (AI Search Crawlers), and fetching content to generate answers (AI Assistants).
• Data collection for AI model training remains the largest segment of AI-related traffic. AI data scrapers visit websites across all categories but focus particularly on "online communities".
• Content fetching from AI assistants like ChatGPT, Perplexity, and Gemini exceeded 3% of all website traffic for the first time ever. This has doubled in the last quarter alone (up from 1.5%).
• Human referrals from AI-generated answers continue to grow steadily. ChatGPT dominates as the primary referrer, accounting for 79% of them. However, the click-through rate to cited websites is just 2.37%.

Bot Activity Trends

AI Assistants

• Many new "Deep Research" AI assistants have been identified. These bots crawl websites to generate formatted, in-depth analyses from a large number of sources. One example is Google-NotebookLM, which powers Google's NotebookLM product that uses their Gemini model to synthesize podcast-style study material. These new bots have been added to the agent database and are now trackable in Agent Analytics.
• Website traffic from AI assistants continues to grow. For the first time ever, website traffic from AI assistants exceeded 3%. This growth is driven by the increasing user adoption of AI chat platforms like ChatGPT.

See the insights page for realtime data. You can find a breakdown by individual agent in the appendix.

AI Data Scrapers

• AI data scrapers have been targeting "online community" websites. This makes sense, since question-and-answer content is ideal for training LLMs because it's similar to the desired output format of responding to user prompts. This is also reflected in data showing that platforms like Reddit are among the most frequently cited sources in AI-generated answers.
• However, traffic remains heavy across all website types. AI data scrapers generate the highest traffic volume of all AI bot types at about 5% overall. These scrapers target websites across diverse categories, and the number of websites that are adding them to their robots.txt files has been increasing. If you'd like to do the same, we recommend using Automatic Robots.txt rather than adding individual bots manually.

See the insights page for realtime data. You can find a breakdown by individual agent in the appendix.

AI Search Crawlers

• A surprising number of websites are blocking AI search crawlers. 23% of top websites are blocking AI search crawlers, while 29% are blocking AI data scrapers. The fact that these numbers are similar is unexpected, given the difference in use cases. AI data scraping primarily benefits AI companies, while AI search crawlers can directly benefit websites by referring human traffic from AI-powered search results (known as GEO or AIO). We hypothesize that websites aren't properly distinguishing between the two types, and have clarified the documentation in our agent database to help correct this possible misconception.
• AI search crawler traffic has remained steady for the past few months. Unlike other AI bots like AI assistants whose traffic correlates with AI product adoption, volume from AI search crawlers hasn't increased significantly in recent months. It's likely that they've already discovered the vast majority of existing websites and have transitioned to performing incremental fetches for updated content of known websites. As expected for this use case, "reference" websites are heavily crawled.

See the insights page for realtime data. You can find a breakdown by individual agent in the appendix.

AI Agents

• Viable browser-using agents are just starting to launch publicly. In the past 3 weeks, Microsoft, Amazon, and Google have all announced new AI models and infrastructure capable of using a browser autonomously. These agents are capable of performing multi-step tasks on websites like making reservations, booking travel, and comparing products. We expect website traffic from these types of agents to increase significantly in the coming months.
• Many AI agents are now identifying themselves with Web Bot Auth. Thanks to a push by Cloudflare, many AI agents are now adopting the Web Bot Auth standard (also known as HTTP Message Signatures) to cryptographically identify themselves to the websites they visit. This makes it impossible for "bad" bots to pretend they're "good" bots. Agent Analytics now includes Web Bot Auth support, so you can see verified visits from AI agents like ChatGPT-User when they visit your website.

See the insights page for realtime data. You can find a breakdown by individual agent in the appendix.

All Other Bots

Realtime insights into hundreds of other bots of all different types (e.g. Intelligence Gatherers, Archivers, SEO Crawlers, Search Engine Crawlers) can be found on the insights page.

LLM Referral Trends

• AI is referring humans to websites in every category. There's no standout website type that benefits most from AI referral traffic. Instead, websites across the board have been receiving this traffic relatively consistently.
• ChatGPT refers the most traffic by far. ChatGPT is responsible for 79% of all AI referrals. However, every other AI chat platform that we track (e.g. Perplexity, Gemini, Copilot, Claude, etc.) has been steadily increasing referral volume.

See the insights page for realtime data.

We Want Your Feedback

We're constantly working to make our analyses as helpful as possible for the web community. If you have questions, suggestions, requests, or would like to discuss these findings, please reach out to us. If you want these insights for your own website, simply sign up and connect your website.

Appendix

Methodology

• The data in this report comes from an analysis of 150 million visits across a diverse set of 2,500 websites.
• All bot names, descriptions, and categories are defined in the agent database, which is updated every day.
• The "top websites" ranking is based on Similarweb's list.
• Website categories are based on those defined by Google AdSense.

Breakdown by Agent