HeadlessChrome
What is HeadlessChrome?
About
Headless Chrome is a tool provided by Google's open source Chromium project. It gives other software the ability to render web pages in full detail. One of the most common use cases is enabling AI agents to browse the internet through libraries like Playwright and Puppeteer. You can see how often HeadlessChrome visits your website by setting up Dark Visitors agent analytics.
Detail
| Documentation | https://developer.chrome.com/docs/chromium/headless |
Type
Expected Behavior
The behavior of headless agents generally depends who is using them and what exactly they're using them for. Patterns could vary significantly depending whether the operator is human-guided software (e.g. performance testing), or an autonomous AI agent acting like a human.
Agent Analytics
Visits to Your Website
Half of your traffic probably comes from artificial agents, and there are more of them every day. Track their activity with agent analytics.
Top Websites
Robots.txt
Should I Block HeadlessChrome?
Probably not. Headless agents normally don't follow robots.txt rules anyway.
How Do I Block HeadlessChrome?
You can block HeadlessChrome or limit its access by setting user agent token rules in your website's robots.txt. Set up Dark Visitors agent analytics to check whether it's actually following them.
| User Agent Token | Description |
|---|---|
HeadlessChrome |
Should match instances of HeadlessChrome |
# robots.txt
# This should block HeadlessChrome
User-agent: HeadlessChrome
Disallow: /
Recommended Solution
Instead of doing this manually, use automatic robots.txt to keep your rules updated with the latest AI scrapers, crawlers, and assistants automatically.