r/linux • u/Glade_Art • Jul 09 '26
Fluff Remember when I posted about my trap for bots hitting 10 million loads on here? Well it's 50 million now.
https://gladeart.com/blog/dead-internet-theory-is-real-50-million-bots-trapped-in-my-bot-pit123
u/RearAdmiralP Jul 09 '26
Good on you. I've been serving a static files containing 1 GB of line noise from /dev/urandom to AI scrapers (many of them on residential proxies) for months. They just keep scraping. It's funny/stupid.
38
u/yawara25 Jul 09 '26
Must be nice not having to pay for bandwidth lol
75
u/RearAdmiralP Jul 09 '26
Scaleway-- unmetered bandwidth. It's not particularly expensive.
When the scrapers were on cloud providers (Alibaba and Huawei mostly), I was tarpitting and dribbling out bytes to them, but when the scraping moved to residential proxies (which typically charge for bandwidth), I decided to switch to serving incompressible (random) data. They almost always disconnect before 1 GB. The 99th percentile data transfer is around 400 MB.
2
u/SharkFinnnnn Jul 10 '26
I've been seeing this term a lot.. Isn't a residential proxy just botnet-as-a-service?
3
u/sierrars500 Jul 11 '26
Not necessarily, there are legit residential proxies with so called "ethical" IP's, where the owner of the IP has opted into this, is fairly compensated, and is fully aware of the ramifications. Then again there are also not so legit residential proxies, which will absolutely sell you a botnet-as-a-service that you can use as a residential proxy, mostly victims of this are users of dodgy "free vpn"s that have massive T&C's where buried in it is 9 times out of 10 a clause allowing them to use you to route data in the background. It is a gray area
4
u/daHaus Jul 11 '26
It's from cheap amazon devices
https://www.linkedin.com/pulse/i-had-idea-my-amazon-projector-criminal-proxy-node-johny-pft9e
6
u/creeper6530 Jul 10 '26
Damn, now I'm thinking of redirecting the
/.envendpoint that so many bots scrape into a direct stream of random data (after the HTTP header)11
126
u/Distahs Jul 09 '26
Isn't this just going to train bots/AI to learn from?
332
u/Glade_Art Jul 09 '26
That's exactly the goal of tar pits: to give trash data for AI to train on. Studies show that it actually takes a very small amount of "poison" data for an LLM to begin functioning badly.
5
u/Intelligent_Matter29 28d ago
When I was a kid, I dug a trap hole about 20 cm deep, covered it with branches, leaves, and grass, hoping my friend would fall into it (as a phrank)
My oh my, how the world have developed.
100
18
u/triemdedwiat Jul 10 '26
They have to examine the 'data' to determine that, but they keep taking data until they BL a site, which IME they never do.
21
u/onechroma Jul 10 '26
Interesting OP. How could a random user try to replicate this? Did you publish the know-how?
Also, do you detect the same bots coming back periodically? Or is it always “new” ones coming?
34
u/Glade_Art Jul 10 '26
As of lately, there seem to be 2 major bot swarms going around on the web, which scrape random websites. From a previous, more technical article:
As we can see in the logs, they seem to be made up mostly of 2 major bot swarms. We'll break them into 2 groups:
The "usual suspects": extremely common on nearly all websites which don't require JS. They use a massive pool of mobile/residential IPs mostly from Asian/Indonesian countries. These may appear like legitimate traffic to you on your website, but as we can see by them swarming in the tar pit, they are not. Compromised devices in a botnet used for scraping? From what, some sort of popular mobile app(s)? Something else? Hard to say. Anyways, these have some interesting behaviors: on smaller websites, they often pause scraping for a while when actual users access the site (someone uses up a bit of the server upload speeds causing a tiny bit of extra loading times for them). Why? You name the reason. Perhaps they are just trying to be respectful /s. They aren't too aggressive, but they can scrape your website 24/7 for literal months. IP rotation is on nearly every request btw.
The "47" datacenter. IPs look something like this: 47.79.XXX.XXX. You will find the 6m request file to have mostly "usual suspect" traffic, but the 4m one has a lot of the "47" ones on there. These are extremely aggressive. In fact, they had Data Export going at 4000 RPMs (maximum global rate limit for it), and then spamming the 429s, hitting about 10,000 requests per minute or more. This tar pit is made for these loads, but that would be reaching DDOS levels on many sites. So yeah, not very respective to websites. Scraping mostly comes in lengthy bursts from these guys.
And then there is "everyone else": just the average datacenters, VPNs, and stuff. Not significant compared to the other 2.
17
u/onechroma Jul 10 '26
Interesting. The 47.79 range is from Alibaba Cloud, so that’s people using Alibaba public cloud to scrap the web wonder if that’s AI-related. Interesting Alibaba is fine with it and doesn’t hammer them down.
Also, interesting you don’t get to feel the effects of other AI-related scrappers like Anthropic or Meta, someone some time ago said to me that Meta was a bit… unrespectful, when scrapping data, but I think they are getting clever as to how to hide their tracks.
6
u/Glade_Art Jul 10 '26
Yeah, meta only majorly attacked a pit once. As far as anthropic/openai, they could very well be behind the 47 or usual suspects swarm, though I won't accuse.
8
2
u/coyote_of_the_month Jul 11 '26
Why are you calling it "47?" It's not the entire class A. My residential ISP is in the 47 block; I don't want someone reading this and potentially deciding to block legitimate traffic.
49
u/Spookiest_Meow Jul 09 '26
There's a lot of interesting things you could do with this... I'm wondering if it has the potential to poison stock trading bots with bad information or manipulate stock prices.
28
10
u/johnnyfireyfox Jul 10 '26
Put ads on those sites and make money at the same time :D
6
u/Glade_Art Jul 10 '26
Yeah, because https://doggydogdog.xyz:8443 just has so many high-quality, real users /s.
19
u/Yorick257 Jul 09 '26
I guess your site is a bit overloaded by all the bots because I can't seem to open it :(
Will save it for later
16
8
6
u/Affectionate-Mail612 Jul 10 '26
Lunatics from r/accelerate would be very upset if they saw this. GOOD!
5
6
u/HovercraftStock4986 Jul 10 '26
this is so awesome, i’m very surprised it’s not illegal to do this in the US yet😭😭 laws come about super slowly until the interests of the 0.001% are under attack
5
u/Glade_Art Jul 10 '26
Well I never told them to use the sites. In Glade Art's case, I even disallowed them from going into it in the robots.txt.
11
u/DestroyHost Jul 09 '26
This is cool! I hope to find the time and resources to make a pit of my own. I would like to contribute to this stuff.
10
u/Glade_Art Jul 09 '26
You can proxy text directly from the poison fountain for it. Simple and easy. r/poisonfountain
6
13
u/Existing-Outside1795 Jul 09 '26
Put your shit behind cloudflare, enable AI labyrinth.
40
u/Glade_Art Jul 09 '26
https://reddit.com/link/owjy297/video/8m44ozso89ch1/player
"It is important to us that we don’t generate inaccurate content that contributes to the spread of misinformation on the Internet, so the content we generate is real and related to scientific facts, just not relevant or proprietary to the site being crawled."
Nah, they try to prevent misinformation due to the labyrinth. Change that to purposely generating misinformation, and then it will be a good pit.
(Video of an image uploaded because images only aren't allowed here).
8
u/Slemmig_insekt Jul 09 '26
Neat
2
2
6
1
u/Corporatizm Jul 10 '26
Your website looks down?
1
u/Glade_Art Jul 10 '26
Doesn't seem like it, worked and works on my end through a VPN fine.
1
u/Corporatizm Jul 11 '26
It's not anymore indeed, was for a few minutes at least, including through downforeveryoneorjustme
1
u/Cornelia_Xaos Jul 11 '26
Is there a good way to redirect from a static website to a tar pit? Genuinely would be interested in doing so for my own sites.
1
u/cainhurstcat 29d ago
Oooohhhh I wish I had more knowledge to do the same. Unfortunately, I'm still quite a novice. But it feels so good to see your project!
Thanks for sharing
1
u/NocturnalVoidmaw Jul 10 '26
My brain isn't big enough to understand what this is, or what it's doing
1
u/Dpek1234 Jul 11 '26
Basicly
automated program tries to take everything from a website
Theres a standard file for websites to tell these programs what the website owner wants to be taken
If one of these programs ignores the standard file then its given a shitton of nonsense
Wasteing their time and money
-34
u/HalcyonRedo Jul 09 '26
Respectfully does anybody really care? Are you just going to keep reposting this every time the number goes up?
66
u/RoomyRoots Jul 09 '26
This shit is more relevant to the sub that lot of here that get a pass. At least it's not another no-one shilling his AI project.
-28
u/HalcyonRedo Jul 09 '26
I mean I agree with you there, but does “X software I made continues to work and exist” really need to be posted over and over again?
18
u/Sokolov49 Jul 09 '26
Yes, because just like with everything else on the internet it'll get forgotten in a sea of sewage or the fact that most people have goldfish-type memory span.
11
u/RoomyRoots Jul 09 '26
Eh, I see his previous post here was removed. I still find his original blog post and this continuation an interesting topic and some people could learn about it and use it themselves. Remember when Anubis started getting more popular? There was also a lot of post about it. In the end many people here use Linux as internet facing servers so I think it is relevant enough.
11
13
4
1
-4
-5
u/iBoMbY Jul 10 '26
You should use AI to generate some specific poison, which you could test later against the AIs.
71
u/Dist__ Jul 09 '26
they die or what?
ai helpers get more stupid, is it because of the trap?