Author(s): Jonathan Harbourne

Originally published on Towards AI.

Meta’s AI keeps scraping my community website, and I get charged £25 a month for the privilege

How the AI industry has externalised its costs onto the community websites that individuals build and pay for out of their own pockets — and what the numbers actually look like when it happens to you.

At around nine o’clock on Monday evening, my phone started buzzing with server alerts. A website I run — a small, volunteer-built community wiki that I pay for out of my own pocket — was straining: processors pinned, pages loading at a crawl. My first thought was the obvious one: I was under attack.

I wasn’t. When I dug into the logs, I found something that made me angrier than an attack would have. The site was being crawled — methodically, relentlessly — by a bot belonging to Meta, the company that owns Facebook, Instagram and WhatsApp and is valued at well over a trillion dollars. And it was costing me real money, out of my own pocket.

The site is the LGBT History Project — one of the UK’s largest online archives of LGBT history, built on MediaWiki, the same software that runs Wikipedia. I started it fifteen years ago, after attending an LGBT History Month event and finding the “history” part surprisingly thin. I wanted somewhere to capture people’s knowledge and memories of a rapidly changing time. Since then, its pages have been viewed more than 47 million times, the site has been archived by the British Library, and the project earned me a Points of Light award from the Prime Minister. Recently, I gave it its first major overhaul in all those years, migrating it from an old server to Amazon’s cloud with the latest software, to make it faster and more secure. Instead, visitors complained it had become slower. Now I knew why.

And while this archive happens to document LGBT history, that detail is incidental to what follows. It could be a wiki about steam trains, or parish churches, or moth species. What matters is what it is: a community website, built by volunteers, paid for by one individual. Me.

Here is exactly what I found, because the specifics matter.

What the logs showed

Over about a day and a half, one Meta crawler — identifying itself as ‘meta-webindexer’ — made 230,743 requests to my site. That was more than a quarter of all the requests my server handled in that time. And because it was pulling whole page histories rather than light pages, it accounted for 47 per cent of all the data the server sent out — nearly half the bandwidth, from just over a quarter of the requests. It pulled 12.15 gigabytes off the server in the process.

I checked who owned the addresses, because a user-agent string can be faked. It wasn’t faked. The internet registry lists the range as netname ‘FB-BLOCK’, description “Meta Route”, announced by AS32934 — Facebook, Inc. It is unmistakably Meta.

For scale: the wiki has 4,814 actual articles. Not a huge site — the kind of thing a couple of volunteers maintain in the evenings. So why 230,000 requests?

Why a tiny website looks infinite to a bot

This is the part that deserves to be better understood, because it explains why small, well-meaning sites are hit so disproportionately hard.

A wiki doesn’t just have articles. It keeps the full editing history of every page. Those 4,814 articles carry 51,239 saved revisions between them — and every single old revision is its own web address. So is the “difference” view between any two of them. So is every page’s edit form, its history, its “what links here” list. Multiply it out and a few thousand articles explode into hundreds of thousands of unique URLs.

A careful crawler recognises these for what they are — low-value duplicates and machine pages — and leaves them alone. Meta’s did not. It went after diff after diff, old revision after old revision, edit page after edit page: 27,504 requests for diffs, 18,863 for superseded versions of pages, 12,321 for edit screens no reader ever visits. But the page it hammered hardest was stranger still: my login screen, requested 113,016 times — nearly half of everything it asked for. Every article carries a “log in” link that points back to itself, and the crawler followed every one, minting a fresh login URL for each page on the site.

The polite request it never read

There is a decades-old convention for asking crawlers to stay out of exactly these areas — a plain text file called ‘robots.txt’. When I moved the site to its new server, I put one in place that asks every crawler to keep off the ‘/index.php’ addresses — the very ones that generate those old revisions, diffs and edit screens — and, by a widely used if non-binding convention, to wait ten seconds between requests. It was sitting at ‘lgbthistoryuk.org/robots.txt’ throughout the period these figures cover.

Meta’s crawler sent 226,391 requests to those disallowed addresses — 98 per cent of everything it asked for — at a rate of hundreds per minute, not one every ten seconds. In the logs for that period, not one of its 230,743 requests was for the ‘robots.txt’ file. Whether it read the rules and disregarded them, or never bothered to look, the effect on my server was identical: the polite “please don’t” was simply steamrollered.

The bill lands on me, not on them

Here is the injustice in one sentence: Meta decides to copy an entire website for its own commercial benefit, and the individual who runs that website is the one charged for the privilege of being copied.

My site runs on Amazon’s cloud, and Amazon — like every host — charges for data sent out to the internet. Every page that crawler pulled, I paid to deliver. At current rates, Meta’s share alone works out to roughly $30 a month — around £25 — and climbing, before you count the other AI crawlers doing the same thing. And because the load was heavy enough to slow the site for real human visitors, I had to pay to upgrade the server to cope. Money out of one person’s pocket, keeping a free, volunteer-built reference site online, is instead subsidising the training data of one of the richest companies on earth.

I want to be precise about what this is and isn’t. It isn’t hacking. It isn’t illegal. Meta is copying material that is, in the main, publicly visible. But “legal” and “fair” are not the same word. Meta could have asked. It could have throttled itself. It could have honoured the file that exists specifically to say please don’t strip-mine my edit history. It did none of these, because it didn’t have to — the cost of its choices doesn’t land on Meta. It lands on me.

This is happening everywhere the web is small and unpaid

I am not a special case, and that is rather the point.

In April 2025 the Wikimedia Foundation reported that the bandwidth it uses to serve multimedia had risen by 50 per cent since the start of 2024 — almost entirely because of automated scrapers harvesting content to train AI, not because of any rise in human readers. Crawlers, it found, generated 65 per cent of its most expensive traffic while accounting for just 35 per cent of pageviews. The documentation host Read the Docs described a single crawler pulling 10 terabytes in one day and 73 terabytes in a month — a bandwidth bill north of $5,000 — and found that blocking AI crawlers cut their daily traffic by roughly three-quarters. Numerous open-source projects have resorted to putting puzzle-gates in front of their own websites just to keep AI bots from knocking them over. One developer’s exasperated post about the phenomenon — “please stop externalising your costs directly into my face” — became a rallying cry precisely because so many small operators recognised it.

And it isn’t only Meta. OpenAI’s ‘GPTBot’, Anthropic’s ‘ClaudeBot’, Amazon’s and ByteDance’s crawlers have all drawn the same complaints. An entire industry raced to scrape the open web for training data, and the cumulative effect has been to make the bandwidth bills, the server strain and the cleanup somebody else’s problem. Usually somebody small — and usually somebody paying out of their own pocket.

What can actually be done

For my own site, the immediate fixes are unglamorous but effective. I blocked the offending crawler at the front door, so its requests are now turned away cheaply before they can tie up the server. I’m putting the site behind Cloudflare’s free tier, which absorbs this kind of traffic on their network instead of on my bill, and which can now block AI scrapers wholesale.

But the development that actually points somewhere better came in July 2025, when Cloudflare launched something called “Pay Per Crawl”: a marketplace that lets a website charge AI companies for the right to crawl it — and began blocking AI scrapers by default. In other words, the thing I instinctively wanted the moment I saw that bill — if this is for your benefit, you can pay for it — is finally being built. It is early, and aimed mostly at large publishers. But the principle is exactly right, and it needs to reach down to the volunteers, the hobbyists and the one-person archives, because that is where much of the web’s real knowledge lives, and where these costs bite hardest.

The ask

I don’t think Meta set out to bully a one-person website. I think something worse is true: it simply didn’t notice. A rounding error in a crawl budget. And that is precisely why this needs saying out loud — because the entire arrangement depends on the people footing the bill staying quiet about it.

So here is the record, with the numbers attached. A company valued at well over a trillion dollars took 12 gigabytes from a volunteer-run community site in a day and a half, brushed past the one polite request the site makes of every crawler, and left me the invoice. If enough of the people who run small, self-funded websites say the same thing, with the same receipts, “somebody else’s problem” stops being a viable business model.

The open web was built as a gift. It should not become an unpaid supplier to the richest companies in history.

—

The author runs the LGBT History Project, a volunteer-built community wiki, and pays for its hosting personally. All figures in this article are taken directly from the site’s own server logs and public internet registry records. Drafted with AI assistance; all figures and events are the author’s own.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI