Known Agents 2025 Year In Review
Known Agents lets websites track and control their bot traffic with features like Automatic Robots.txt, Agent Analytics, and LLM Referral Tracking. You can connect your website using any CDN, backend, or the WordPress plugin, for free.
The Known Agents 2025 Year in Review reveals how AI bots reshaped the web in 2025, and what these shifts mean for website owners in 2026 and beyond. We encourage you to explore the findings below. Detailed sources and methodology can be found in the appendix.
1. Bots represented ~40% of all website traffic
Non-human traffic reached a tipping point in 2025. Traditional crawlers like Googlebot remained the most active, but new AI-related bots made up 29% of all that traffic. Altogether, bots accounted for approximately 40% of visits to the average website. This shift fundamentally changes how website owners need to approach infrastructure costs, content optimization, and IP protection strategies. Website owners can use Agent Analytics to track all bot activity across categories.
Traffic Percentage By Agent Type
Traffic Percentage By Top Agent
2. AI bots fell into 4 major behavioral categories
AI bots weren't monolithic. They served distinct purposes with real business implications:
- AI data scrapers gathered content to train LLMs
- AI search crawlers determined whether sites appeared in AI-powered search results
- AI assistants gathered intelligence on brands
- AI agents performed multi-step tasks on behalf of real humans
Each category exhibited different traffic patterns depending on whether they were automated or user-initiated.
3. Publishers inconsistently blocked AI bots of the same category
Robots.txt analysis across the top 1,000 domains exposed a fragmented approach to bot management. Publishers blocked certain AI data scrapers, while allowing others that served the same purpose. The block rates for functionally identical bots varied dramatically, revealing that most sites lacked a coherent strategy or were unable to keep up with new bots. To maintain a consistent blocking strategy, we recommend using Automatic Robots.txt rather than adding individual bots manually.
More troublingly, many publishers blocked AI assistant and search crawler bots that could have driven real traffic from AI platforms like ChatGPT and Gemini. Publishers appeared unable to distinguish beneficial crawlers from extractive training bots. This confusion likely cost them significant referral traffic as AI search grew. Again, we recommend using Automatic Robots.txt to solve this problem.
AI Data Scraper Blocked Percentage
4. AI agents became a potential new customer
Autonomous AI agents such as ChatGPT User and Manus User emerged. They browsed, compared, and transacted on behalf of human users. Companies like Browserbase built infrastructure to help businesses develop their own AI agents. With this early foundation, paired with advancements in model accuracy, AI agent activity was expected to pick up significantly in 2026. Website owners who want to capture this growing conversion channel should use Agent Analytics to see how AI agents are navigating their pages and improve conversion rates.
AI Agent Traffic Percentage
5. Initiatives to build trust accelerated, and adoption grew
The industry responded to bot proliferation with better mechanisms for transparency. Standards like HTTP Message Signatures (web bot auth) emerged to cryptographically verify bot identity, while bots increasingly included metadata in their requests to provide detail about their purpose and operator. This helped websites verify legitimate bots and optimize their experience.
Appendix
Methodology
- The data in this report comes from an analysis of 150 million visits across a diverse set of 2,500 websites.
- All bot names, descriptions, and categories are defined in the agent database, which is updated every day.
- The "top websites" ranking is based on Similarweb's list.
- Website categories are based on those defined by Google AdSense.
We Want Your Feedback
We're constantly working to make our analyses as helpful as possible for the web community. If you have questions, suggestions, requests, or would like to discuss these findings, please reach out to us. If you want these insights for your own website, simply sign up and connect your website.