Set Up Automatic Robots.txt

Overview

Serve a robots.txt that continuously updates with rules for new crawlers, scrapers, AI agents, and other bots as they're discovered. Please contact us if you need help getting set up.

Option 1: Connect Using Your Platform

Looking for Akamai, Azure, Google Cloud, Vercel, or Netlify? Let us know.

Option 2: Connect From Your Backend Code