A year of fighting scrapers on my 1.5 million-page website

345 points - today at 2:51 PM

Comments

jwr today at 3:54 PM

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see.

A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.

And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.

johnorourke today at 3:16 PM

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software.

[1] https://anubis.techaro.lol/

tarr11 today at 3:28 PM

> My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%.

This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.

GodelNumbering today at 4:28 PM

I just checked Cloudflare for SignalBloom (https://www.signalbloom.ai, which I own and operate).

Over the last 72 hours, Claude-searchbot [1] alone fetched ~205,000 pages. Sent exactly 1 referral. There is a lot of free financial data on the site, hoping for real users to benefit from it. It is hard to not feel a little cheated out that Claude gets to claim "Found it!" to its users without me getting no credits or compensation whatsoever.

[1] Exact user agent `Claude-SearchBot/1.0; +searchbot@anthropic.com)`

Proof: https://i.postimg.cc/Pqc3SS8T/Screenshot-2026-08-07-at-5-33-...

qbane today at 3:08 PM

> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.

thomashabets2 today at 5:24 PM

Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly.

Is it really a bot if it's in response to a human asking for some aggregate information about charities, triggering a web search and then following the result links to get details for the human? Well, clearly yes it is, but it's a very different proposition from this article's implication that "they have no throttling on their scrapers"[1].

> Challenge 46 datacenter ASNs. Humans don't browse from AWS.

People who have workstations in the cloud do.

> The bots use 99% of the bill and I pay 100% of it.

Running a site this way is always a wallet-DDoS risk.

[1] though yes, by far most will be pure automation with no human in the loop. It's an assumption on my part, but feels like a safe one.

ddxv today at 3:11 PM

I'm in a similar boat. Probably 99.999% is bots. I have nearly 1m unique "visitors" according to cloudflare and my real users are in the dozens a day. That being said, I love the open internet and am holding on to keeping as much open as I can.

varenc today at 4:01 PM

Can someone help me understand the underlying motivation behind this?

It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely.

What's the motivation behind the same entity re-fetching a page it just fetched less than an hour ago?

grigio today at 10:00 PM

LLM are the best way to read the web. right to the point, no ADS

mgbmtl today at 3:24 PM

I run scripts on my servers on an hourly basis to check which are the top 25 IPs visiting the server (aggregated by /24). If anyone in those top 25 IPs are from China, Vietnam, etc, or from Alibaba/Amazon/etc, the /24 gets blocked by iptables.

It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam.

However, on a Gitlab instance I manage (500 users), we have to restrict viewing of git logs and pretty much everything except issues. The bots were too aggressive. Chinese crawlers have access to a huge range of IPs and they often do only 10-20 requests per day, while generating in total over 50k requests per day. Our server load went from 99% down to 0.1% after that (and it's a fairly big server).

rsolva today at 8:59 PM

I made a small booking site for a local dutch canal boat, which has a calender function. A simple PHP app. I checked the Apache logs recently, and it had THOUSANDS of claudebot and other AI UserAgents flooding the logs every day, apparently because the scrapers keep hitting the 'next month' button on the calendar in a an infinite loop, all day, everyday!

This is a small booking app without any useful information at all, it surprises me that the AI boots have no discernment about what the are scraping, just wasting their own and other peoples resources. And their own reputation! You would thing they could spare a few tokens on a classifier model to do a quick evaluation of their scraping efforts, but apparently they do not.

Anyway, I have done my best to block these UAs and so far it seems to have improved the situation.

andai today at 3:11 PM

> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it.

Kinda sounds like we're missing a peer to peer network here.

Instead of downloading the same data over and over again we can just download it once and then share it.

Wouldn't that be better for everyone involved?

It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)

Bender today at 3:21 PM

Seems about right. I rotated my logs this morning. Most humans go to access.log and most bots go to botpoop.log. This is the line count:

     2 access.log [1]
    40 botpoop.log [2]

2 is really 1 since a human will grab the CSS file. Most bots do not bother with the style-sheet so that's a 40:1 bots to humans. I could cut that down by blocking data-centers but then I inadvertently block a lot of VPN's which I really don't need to do for a static compressed blog served from ram. The bots just get a TCP Reset but it's still fun to log and study them. The most interesting one I've seen recently is ReadYou which may be a reader but it appears to be much more, possibly acting as a cell phone distributed bot collecting data for a centralized site.

[1] - https://nochan.net/logs/access.log

[2] - https://nochan.net/logs/botpoop.log

storus today at 3:10 PM

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

falcor84 today at 4:41 PM

It's a sequence of thin lines, between using Chrome out of the box, to using something like Brave, to using a highly customized Zen Browser, to having ChatGPT look up a particular page for you, to using a small BeautifulSoup/scrapy script to scrape dozens of pages, to scraping the entire web. And at every point on this spectrum it's humans driving a "user agent" tool to make requests and process responses on their behalf.

tananaev today at 4:35 PM

I also see quite a bit of traffic from China and Singapore. I wonder if it's some scraping for AI training. It doesn't really bother me too much because traffic is still fairly low, but it skews all the analytics for me.

ashu1461 today at 3:24 PM

I wonder if the author tried out the recently released feature by cloudfare to block ai bots

https://developers.cloudflare.com/bots/additional-configurat...

thorsson12 today at 4:34 PM

The experience of browsing the web has really suffered lately. The mandatory 3-4 second "verifying that you're a human" block from Cloudflare seem to show up on more and more websites. Seems like a questionable choice from Cloudflare to teach everyone to associate Cloudflare's logo with high latency...

dzonga today at 3:08 PM

blocking by geo yeah might work - but what happens when someone is traveling abroad ? they've to use a VPN to access your site ?

my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.

butz today at 6:18 PM

Cloudflare's "Verify you are human" captcha is the new cookie banner.

travisgriggs today at 5:24 PM

I wonder what the web would be like if we priced bandwidth at the requester point and as you go.

somebudyelse today at 4:47 PM

I visited someone's blog and didn't have to solve a cloudflare challenge! That's amazing work

brownieman1325 today at 4:34 PM

I recently got a surge from Singapore and heard a lot more peeps in my circle of friends saying the same thing...

DoesntMatter22 today at 9:05 PM

I'm surprised that the solution wasn't to poison the data so they will train on junk data

luciana1u today at 5:11 PM

at this point the web is mostly machines politely asking other machines for permission to read each other's content, and humans are the ones triggering the captchas.

deleted today at 4:08 PM

r0b0tan today at 4:26 PM

I think it’s wild how AI and data bots are putting certain business models under pressure. We’ve already seen the same thing happen with Tailwind.

1vuio0pswjnm7 today at 8:09 PM

Original HN title: "99% of My Website Traffic Is Bots"

Not clear how the author arrived at the precise 99% figure; perhaps "99%" is a figure of speech

"And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers."

"I'm trying to run a business here."

What's the business

(where "business" is defined as "buying and selling")

From https://patronview.com/robots.txt

   # As a condition of accessing this website, you agree to abide by the following
   # content signals:
   
   # (a)  If a Content-Signal = yes, you may collect content for the corresponding
   #      use.
   # (b)  If a Content-Signal = no, you may not collect content for the
   #      corresponding use.
   # (c)  If the website operator does not include a Content-Signal for a
   #      corresponding use, the website operator neither grants nor restricts
   #      permission via Content-Signal with respect to the corresponding use.
   
   # The content signals and their meanings are:
   
   # search:   building a search index and providing search results (e.g., returning
   #           hyperlinks and short excerpts from your website's contents). Search does not
   #           include providing AI-generated search summaries.
   # ai-input: inputting content into one or more AI models (e.g., retrieval
   #           augmented generation, grounding, or other real-time taking of content for
   #           generative AI search answers).
   # ai-train: training or fine-tuning AI models.
   # use:      how AI systems may consume the content (immediate, reference, or full).
   
   # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
   # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
   # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
   
   # BEGIN Cloudflare Managed content
   
   User-agent: *
   Content-Signal: search=yes,ai-train=no,use=reference
   Allow: /

Perhaps this could be construed as a license, e.g., permitting or prohibiting certain uses of the "content"

If, for example, the website operator had enforceable intellectual property rights in the "content", such as copyrights, then perhaps the operator could restrict access to the "content" under the threat of litigation to enforce those rights

Basic questions

1. Is the "content" protected by intellectual property rights, e.g., copyrights

2. Does the website operator have intellectual property rights in the "content", e.g., copyrights

3. Does the website operator have agreements with the rights holders, e.g., granting the operator authorization to restrict access to the "content"

DaveZale today at 4:33 PM

This is like technological cannibalism.

I stopped posting to my website. Why should it be so much work to stop this theft?

Would it be helpful to have geofencing and regulation?

wandr today at 3:58 PM

I'm working on a web app right now, with the intention of it going to be 100% paywalled. It's 95% complete, but the remaining 5% is just implementing the paywall. In the meantime, the app is live and operational with a fully functional signup. I am deleting about 100 new bot signups per day right now, it is crazy out there.

Venn1 today at 3:23 PM

I'm blocking the Amazon search crawler, anything coming from Googleusercontent, and limiting AI crawlers to search rather than allowing AI assistants. The residential proxy waves are something to behold, but Cloudflare does an okay job catching those in the AI labyrinth. Still, it's all a bit silly, and I can't imagine what large sites deal with when I'm tangoing with this much nonsense on a small tech blog.

FerretFred today at 3:32 PM

Sigh .. same here. I don't write blog posts often (enough) but the ones I do write are from personal experiences and I take a lot of care with them. I look at my logs snd see bots everywhere, but now I just let them get on with it. AI scrapers are different though; they get to read my content which, just for them contains a smsttering of finest digital toxin. A pox on your datasets!

sp1982 today at 3:30 PM

The annoying part is a large percentage of misbehaving bots (not obeying robots.txt for example) are via end user proxies across the world. However most of these aren't doing full-browser loop, so if you are behind cloudflare, you can do non-interactive challenge and that can help quite a bit.

deleted today at 3:22 PM

knuckleheads today at 3:46 PM

Previously, I had done a fair amount of research into how Google's monopoly on web crawling further entrenches their monopoly in the search engine market. You can read more about this here, https://knuckleheads.club, there is a long report from ~2020 or so that explains how it worked at the time. The club is mothballed, I am doing other things with my life, and I'm happy to say that we played a very small role in the DOJ ordering Google to share their crawl data with qualified competitors (a work in progress, but it's progressing).

Chatbots have super charged this dynamic though, to the point that it is showing up in the robots.txt data. The last few weeks I've been having Claude rerun some old analysis of Common Crawl from back then, when I have spare usage and time. What I've found is that you can see pretty clearly the rise in people outright blocking AI chatbot related crawlers likely because of how aggressive they have become.

  Quarter  Crawl    GPT     Claude  CC      G-Ext   Byte    Bing    Google
  2023 Q1  2023-06  0.00%   0.00%   0.16%   0.00%   0.06%   0.47%   0.39%
  2023 Q2  2023-14  0.00%   0.00%   0.18%   0.00%   0.06%   0.45%   0.38%
  2023 Q3  none       —       —       —       —       —       —       —
  2023 Q4  2023-40  2.21%   0.00%   2.12%   0.04%   0.11%   0.39%   0.27%
  2024 Q1  2024-10  0.53%   0.05%   0.31%   0.09%   0.18%   0.34%   0.31%
  2024 Q2  2024-18  0.55%   0.09%   0.32%   0.11%   0.24%   0.32%   0.31%
  2024 Q3  2024-30  0.68%   0.22%   0.36%   0.20%   0.38%   0.24%   0.33%
  2024 Q4  2024-42  1.10%   0.50%   0.44%   0.32%   0.50%   0.25%   0.40%
  2025 Q1  2025-05  1.14%   0.66%   0.54%   0.42%   0.66%   0.25%   0.44%
  2025 Q2  2025-18  1.37%   0.93%   0.63%   0.70%   0.92%   0.29%   0.19%
  2025 Q3  2025-30  1.42%   1.07%   0.74%   0.62%   1.01%   0.31%   0.27%
  2025 Q4  2025-43  1.92%   1.51%   1.23%   1.15%   1.52%   0.27%   0.19%
  2026 Q1  2026-04  2.13%   1.76%   1.68%   1.58%   1.77%   0.22%   0.15%
  2026 Q2  2026-17  2.80%   2.38%   2.26%   2.13%   2.50%   0.22%   0.14%
  2026 Q3  2026-30  3.45%   3.01%   2.89%   2.71%   3.16%   0.21%   0.14%

GPTBot is OpenAI, ClaudeBot is Anthropic, CCBot is Common Crawl, Google-Ext is a way for website owners to indicate they don't want their content to be used for AI, Bytespider is Bytedance, Bing and Google are the last two. Take these numbers with a truck of salt, haven't had time to verify them.

It's very clear that website owners do not like getting their content scraped and are indicating to GPTBot et al. that they are not welcome. It's a shame that CCBot is caught in the cross fire, but that's life. Bing and Google are doing just fine though, almost like having significant power in the search engine market gives you an advantage in other markets too. Who knew!

nromiun today at 3:44 PM

This is a static website running on Cloudflare infra. What on earth costs $90 per month? First optimize your infra before throwing up rules in front of your visitors. I have several websites on Cloudflare too and I don't even check how many million requests I get. Because it does not cost me anything.

> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.

Being self aware does not make it okey. Either you are okey with scraping (like me) or against it. Don't use it yourself and block your site at the same time.

These same people will be crying about how Cloudflare ruins the internet because they get these captchas.

AdrianB1 today at 3:31 PM

I checked the comments to see if anyone pointed to this: I can imagine so many memes with this line :)

l72 today at 4:56 PM

I don't look at the logs of my personal site very often as it is a static site, but just went to check, and yeah, it's almost all ai crawlers. Note sure what is going on here, but I hope this isn't really anthropic:

  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /secrets.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /credentials.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /secrets.yml HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /service-account.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /key.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /config/.env HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /service_account.json HTTP/2.0" 404 366 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /serviceAccountKey.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /firebase-adminsdk.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /Dockerfile HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.github/.env HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /firebase-service-account.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.docker/config.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.npmrc HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.boto HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.s3cfg HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.svn/entries HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.htpasswd HTTP/2.0" 404 346 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /terraform.tfstate HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /docker-compose.yaml HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.vscode/launch.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_rsa HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_ed25519 HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_ecdsa HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/authorized_keys HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/known_hosts HTTP/2.0" 404 343 "-" "anthropic-ai"

oaw93j4oij today at 4:27 PM

I despise cloudflare. They've decided that my home IP address is bad, so I have to capchas for most websites. Sometimes on infinite loop and I never get to the website. I even reset my home IP address more than once, but it instantly continues. Especially if I use any VPN, even my work VPN.

bediger4000 today at 3:55 PM

I think that asymmetry is why scraping keeps getting worse. The scrapers' costs fell faster than everyone's defenses improved.

There it is. Just like the fckn spammers who ruined SMTP email, scrapers externalize the costs.

Who finances the effort to use residential proxies? That takes a lot of effort, even if it's shoddy

righthand today at 3:15 PM

> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer. That's usually the easiest way I spot a bot.

> And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch.

> But those 4,000 bots were just the warm-up.

I just hate this style of writing like you're on Twitter. Why does the above need to be 3 different paragraphs? A paragraph break indicates a separate thought but the author is still talking about the same data and still making their point. The sentence "But those 4,000 bots were just the warm-up." is effective when still the last line of a paragraph and it signals respect for your readers. I stopped reading after this because it's just a terrible reading experience.

Here's a correct version that doesn't read like the author left for a week to think about what the next sentence would be or having some sort of anxiety-induced mental pause:

> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer, which is usually the easiest way I spot a bot. They were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. But those 4,000 bots were just the warm-up.

ticktock today at 10:02 PM

[dead]

imthenitto today at 4:02 PM

[flagged]

hmokiguess today at 3:12 PM

[dead]

szundi today at 3:47 PM

[dead]

lazerg today at 2:55 PM

[flagged]

zuzululu today at 3:10 PM

pretty crazy that this article is seemingly written by an AI used a detector and it is 95% confident its generated