Author(s): Ran Isenberg

Originally published on Towards AI.

My Blog’s Biggest Fans Are AI Bots. So I Fought Back.

For several days, I thought I had gone viral on a whole new level. My daily blog reads jumped from around 600 to nearly 2,800, with a single post carrying almost the entire spike. Then the skepticism crept in: my daily reports could not tell me where the new readers were coming from, and a jump that large with nothing to explain it felt less like a win than a question. It pulled me into an investigation across AWS WAF, the CloudFront console, and a long back and forth with Claude.

After noticing “viral” traffic that didn’t match normal engagement patterns, the author confirms the spike is driven by AI bots rather than real readers, using GA4 metrics alongside AWS WAF data and verification via Claude. They explain how AWS WAF’s AI activity dashboard reveals large volumes of both AI scrapers/tools/agents and a major share of unverified (anonymous) traffic, including weekly “archiver” spikes that copy the site. The post distinguishes verified vs. unverified bots, shows how bots disguise themselves to evade simple identification, and notes that bots can also inflate errors by probing nonexistent URLs. In response, the author tests common mitigations—updating robots.txt (which they argue is voluntary and unreliable), trying JS challenges (which can be bypassed), and experimenting with monetization via HTTP 402/x402 for bots that identify themselves. Their main breakthrough comes from enabling CloudFront Bot Protection/WAF category actions to block unverified bots in monitor-vs-block modes, then rebuilding analytics so bot sessions are surfaced accurately. They report measurable improvement in reads/users, longer average session durations among remaining traffic, and reduced costs under flat-rate plans because blocked requests don’t consume allowance. They conclude with practical guidance for content creators: rely on edge-side bot visibility, sanity-check analytics against real behavior, use per-category protections experimentally, and accept that imperfect blocking may be necessary until tooling improves.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI