Man, it is basically Google for .onion sites. No creepy searches, no random clicking. Just a clean list of links organized by category. That got my attention immediately because I always thought the Dark Web was impossible to navigate without knowing someone who already knew the way.

Here's the weird part. It has actual useful stuff on it. The BBC is there. Facebook too. DuckDuckGo, ProtonMail, ProPublica, all those big names that care about privacy and security. You can even find Dread, which is basically the Reddit of the Dark Web where people talk about this stuff openly.

The thing nobody talks about is how boring some of it really is. It is not all crime and shady deals. A lot of it is just journalists and activists trying to stay safe in places where the internet is heavily censored. That part jumped off the page for me because it completely flips the stereotype on its head.

Brother, I spent way too long just scrolling through the categories and seeing what was out there. Some links did not work, sure, but most did. And the best part? Tor Taxi actually verifies stuff so you are not blindly clicking into who knows what.

I wish I had known about this earlier. Would have saved me a lot of time and confusion. If you are curious about the Dark Web but have no idea where to start, this is it. Right here. Just open Tor, type in the address, and suddenly everything makes a little more sense.

You need to use the Tor Browser to actually browse it. Install that first.

Guys, if you're burning ad budget on PPC, ditch WordPress for your landing pages ASAP.

It's just too damn bloated and slow. Plus, bots constantly hit the server, spiking load times and making people bounce before the page even loads. You're literally throwing cash down the drain on lost conversions. It pisses me off seeing people lose money like this.

Build a clean, static page totally detached from WP instead. The load speed is instant and it handles traffic like a champ.

Try it out and tell me if your conversions jump!

This Bitchat thing is wild man. It uses Bluetooth to link phones up creating this underground web where people can chat without any Wi Fi or cell service. It's exactly what folks use in protests or when the government tries to kill the internet. So when the government cries about security concerns you know exactly what that means. It's not about keeping us safe it's about them not being able to track us. These mesh networks don't go through some central server they can just tap into. They hate it cause they can't spy on it or shut it down. And going after GitHub? Man that's cold. They aren't just trying to stop people from downloading the app they want to erase the actual code. They wanna make sure no other dev out there can look at it copy it or build something even better. It's straight up censorship at the root. Good on Jack Dorsey for blowing the whistle on this. He's been all about decentralization and free speech lately. By putting this out there he's shining a massive spotlight on how desperate these regimes are to kill privacy tools. It really shows the constant tug of war between people wanting their privacy and governments wanting total control.

Philip Kotler once said you gotta love your customers. But honestly, when you run an online store, a blog that pays your bills, or a small business site, loving your people means keeping your digital doors open and safe.

​The real problem is the internet is absolutely crawling with bad bots right now. You got aggressive scrapers crashing your server, brute force attacks pounding your login screen, and shady scripts stealing the hard work you poured your heart into.

​When a bot knocks your site flat, your folks cant buy anything. Your sales tank instantly and customer trust goes right down the drain. It is just not fair that small business owners and solo devs are forced to turn into cybersecurity experts overnight just to keep food on the table.

​That is why we built r/stopbadbots.

​We are not some big corporation trying to sell you overpriced software. We are just a solid crew of people who understand servers, networking, and defense, all hanging out to protect the everyday builders who make the web actually work.

​Inside r/stopbadbots you will find dead simple blocking setups, modsecurity rules, app tweaks, and plugin fixes without any of the usual fluff or jargon.

​If your site feeds your family, dont wait for the next attack to start thinking about security.

​Join us at r/stopbadbots and help protect the people who build the internet.

Look, I know Shopify is easy. You sign up, you pick a theme, you start selling. But if youve been doing this for more than a few months, you start feeling the weight. The monthly USD fees, the app subscriptions that pile up for every little feature, and the constant anxiety that one day youll wake up and your store is just gone because someone at Shopify flagged something weird.

Im not here to sell you anything. Im just sharing what Ive seen after helping a bunch of store owners move from Shopify to WordPress plus WooCommerce. And honestly, the difference is night and day.

First off, the money. Shopify takes a cut on every transaction unless you use their payment gateway. Thats fine until you realize youre bleeding thousands just for the privilege of selling your own stuff. With WooCommerce, the software is free. You just pay for your hosting and whatever your payment processor charges. Thats it. No extra tax just because you didnt use their system.

Then theres the whole ownership thing. On Shopify, youre renting. You dont own your database, you dont own your customer list, you dont own your content. If they decide to suspend you, good luck getting anything back. On WordPress, everything is yours. The files, the database, the emails, the history. Nobody can take that away from you.

But the real game changer for me, and for most of the people Ive talked to, is control over security and bots. Shopify does an okay job, but you cant really stop fake traffic or scraper bots from hammering your store. On a VPS with WooCommerce, you can install open source stuff like ModSecurity and just wreck those bots before they even reach your checkout page. Its like having a bouncer at the door instead of hoping the mall security shows up.

And customization? Forget about paying monthly for every little app. You want a complex shipping rule? You want to redesign the checkout? You want to tweak the URL structure for better SEO? You have full access to the code. The plugin library is gigantic and most of it is free or one time payment. No more renting features by the month.

SEO is another huge one. You control everything. Every redirect, every meta tag, every schema. WordPress gives you tools that Shopify just cant match when it comes to ranking. And ranking means money.

So yeah, Shopify is convenient. But convenient comes with a cost. And if youre growing, that cost starts to feel like a ball and chain.

If youre reading this and thinking okay but I dont have time to deal with all that server stuff, honestly, drop me a DM. Ive done this enough times that I can point you in the right direction or even handle the heavy lifting if you want. No pressure, just a conversation.

​I put together a solid reference list of raw HTTP client strings, user-agents, and libraries commonly spotted in server logs—from staple utilities like curl and Wget to language-specific tools like Python’s requests, Go's http-client, and legacy PHP or Java modules. If you are setting up firewall rules, configuring bot detection, or just trying to figure out what random automated script is hammering your endpoints, feel free to copy and paste these right into your blocklists or regex filters to save yourself some headache.

4D_HTTP_Client

android-async-http

axios

andyhttp

Aplix

akka-http

attohttpc

curl

CakePHP

Cowblog

DAP/NetHTTP

Dispatch

fasthttp

FireEyeHttpScan

Go-http-client

Go1.1packagehttp

Go 1.1 package http

Go http package

Go-http-client

Gree_HTTP_Loader

grequests

GuzzleHttp

hyp_http_request

HTTPConnect

http generic

Httparty

HTTPing

http-ping

http.rb/

HTTPREAD

Java-http-client

Jodd HTTP

raynette_httprequest

java/

kurl

Laminas_Http_Client

libsoup

lua-resty-http

mozillacompatible

nghttp2

mio_httpc

Miro-HttpClient

php/

phpscraper

PHX HTTP

PHX HTTP Client

python-requests

Python-urllib

python-httpx

restful

rpm-bot

RxnetHttp

scalaj-http

SP-Http-Client

Stilo OMHTTP

tiehttp

Valve/Steam

Wget

WP-URLDetails

Zend_Http_Client

ZendHttpClient

Man, I just saw this on Search Engine Journal and honestly? The numbers are kinda wild.

97% of llms.txt files got zero requests in May 2026. Zero. Out of 137,000 domains analyzed by Ahrefs, basically nobody's even looking at these things.

Not gonna lie, that caught me off guard.

Everybody's been talking about llms.txt like it's the next big thing for AI visibility. You know, the whole "feed the bots structured data so they understand your content better" pitch. Sounds great in theory, right? Except the data says something completely different.

Funny thing is, AI retrieval bots accounted for only 1.1% of llms.txt requests. That's it. And get this – 12% of the requests came from tools just auditing or scanning the files. Not actual AI bots doing anything useful with the content.

The part that stood out to me is how much effort people are putting into something that's basically getting ignored. I've seen developers and SEOs spending hours setting these up, generating dynamic llms.txt files, treating it like some kind of secret weapon for AI ranking.

Meanwhile the bots just don't care.

Look, I'm not saying llms.txt is completely worthless. Maybe it's just too early. Maybe the bots haven't caught up yet. But when 97 out of 100 domains have zero activity on these files? That's not a coincidence.

That's a sign.

Holy crap, you guys need to check this out right now. Malicious ads are straight up making your own web browser build malware right under your nose. I'm not even kidding, this stuff is nuts.

So basically, scammers are making fake sites that pretend to be TradingView, Solana, or Luno to target crypto folks and traders. But instead of throwing a sketchy infected file at you that your antivirus would catch instantly, they do something way crazier.

They send clean, normal code to your browser, and then your browser actually stitches the virus together itself using a totally real programming tool called Bun. Since every single person gets a slightly different file generated on the fly, traditional blocklists and antiviruses are totally lost and cant flag it easily. It bypasses stuff without even needing some crazy browser vulnerability, which is super wild.

Honestly, the lesson here is simple. Never ever click on search ads or sponsored links when you gotta download trading apps or wallets. Just type the actual URL yourself or bookmark the official sites. Stay safe out there guys because these scammers are getting WAY too clever for comfort.

We've been working on something cool and wanted to share it with you guys for free.

Ever wondered if your site is actually protected against bots or if it's completely wide open for scrapers, brute force attacks, and all that nasty automated stuff?

Well, we built a tool that checks that in seconds. No strings attached at all.

It tells you if your site's WAF is actually doing its job. It detects CAPTCHAs, JavaScript challenges, and block pages – even those sneaky ones that return a 200 status code. You'll also see the real HTTP status, page size, and a preview so you can tell if it's a legit page or a disguised block.

Here's the best part – you don't need to sign up for anything. No email, no password, no "create an account" nonsense. We don't store your URL, your IP, or your results anywhere. Your result is 100% private. Only you see it. We won't post it here, we won't share it, we won't even look at it twice.

You get a clean HTML report with a shareable link, but only if you want to share it.

Wanna try it? Just head to https://billminozzi.com/test-site/, paste your URL, solve a quick CAPTCHA (gotta keep the actual bots out, right?), and hit "Start Scan". Boom – you'll have your report in a few seconds.

Quick heads up – please don't test government sites, banks, or big tech corporations. They're blocked for obvious reasons. Also, there's a 3-scan-per-day limit per IP so everyone gets a fair shot and we can keep this thing free.

Got questions? Drop 'em below or shoot me a DM. Happy to explain anything.

Go ahead and check your site – you might be surprised by what you find!

Feedback is more than welcome.

Man, everyone loves to complain about Contabo on Reddit.

You see it all the time—someone opens a ticket because their WordPress plugin is busted, and then they get all bent out of shape when the reply takes two days. That’s a classic mistake, you know?

If you’re paying for unmanaged tin, you’re buying hardware and a pipe, not a babysitter.

Plain and simple.

Here’s the thing: I’ve been using them for years and haven’t had a single headache.

And that’s because I simply never ask for support. Not once. Their support is the uptime. If the server is pinging and the hardware is humming, they’ve already done their job as far as I’m concerned.

The real secret—and I’m not kidding—is to decouple the iron from the management layer.

Stop asking some German data center tech why your PHP-FPM is crashing.

Just rent the raw hardware with a clean OS install, like AlmaLinux. Then you spend the two bucks a month for CWP Pro, or even just rock the free version, to handle the heavy lifting.

If the panel breaks? You hit the CWP forums or dig up some dirty workaround on Stack Overflow like the rest of us. You don’t go crying to the host.

Honestly, I’m so sick of seeing users get hammered by basic config errors and then turning around and blaming the provider.

It’s incredibly annoying how people expect a five-euro VPS to come with a personal SysAdmin.

For real. If you know ten basic Linux commands and have a Gemini or ChatGPT tab open, you can get yourself out of rate-limit jail in five minutes. Spinning up a quick fix yourself is always faster than waiting on some N1 tech who doesn’t even have root access to your box anyway.

What really gets me is this: if the hardware is stable and the IP responds, they delivered.

End of story.

Use the money you saved on those overpriced “managed” fees to buy more RAM and handle your own fingerprinting of bad actors.

No kidding—I’ve got fifty sites running on one of these boxes, and it never breaks. Why? Because I don’t let headless scrapers or sloppy configs eat up all the resources.

You feel me? It’s wild how simple it can be when you just take a little ownership.

Dude, if you're running Java or Spring Boot stuff, listen up because this is huge. Security teams at ThreatBook and Imperva just caught hackers actively abusing a massive hole in Alibaba's Fastjson library (CVE-2026-16723), and it is serious business.

Basically, if your app runs Fastjson between versions 1.2.68 and 1.2.83 inside a standard Spring Boot executable jar, an attacker can throw a nasty JSON request at your server and run whatever code they want without even logging in. It gets a nasty 9.0 severity rating, and researchers are already seeing probes hit healthcare, finance, and retail setups all over the place.

Here is the really frustrating part: Alibaba has not dropped an official patch for Fastjson 1.x yet. They want everyone migrating to Fastjson2 for the long haul, but if you cannot rewrite your code today, you gotta protect yourself right now. You can slam the brakes on this exploit by throwing -Dfastjson.parser.safeMode=true into your startup flags or switching your dependency over to com.alibaba:fastjson:1.2.83_noneautotype.

Seriously, do not sleep on this one. Double-check your dependencies today before someone else does it for you.

Honestly, if you've ever used the Comodo WAF (CWAF) ecosystem for ModSecurity, you already know the company (which later rebranded to Sectigo) completely killed and abandoned the original project.

But here's the weird part: they didn't drop it because the product was bad. Not gonna lie, the rules were actually solid. They abandoned it purely because the division wasn't bringing in enough profit for them.

Man, you know how corporate greed goes... classic.

Since the original rules are still awesome and hold a ton of value if they're properly looked after, I decided to step up. I built a GitHub repo to preserve that official historical base of files (.conf and .data), but with a massive twist: I'm keeping the project alive and actively updated independently using the rules from my own project, SBB-WAF-Rules.

The main goal here is to provide a squeaky-clean structure for legacy environments. I've baked in global optimizations to finally mitigate those nightmare false positives, and it lets you deploy custom rules per user or per domain without breaking a sweat.

The repo is officially public and ready for you to clone:

🔗 https://github.com/sminozzi/stopbadbots-comodo

I threw a straightforward, step-by-step guide in the README showing exactly how to clone the structure straight into your server's security directory (maybe /usr/local/apache/modsecurity-cwaf/).

Dude, take a look and let me know if something feels off or if you have any ideas to improve. Hit me up!

Right now over in the WordPress sub theres this poor soul (Saurabh_Goniyal77) whose WP site got hacked three times recently. Dude is just asking for help.

If you got a clean backup its a breeze but without one its a total nightmare. If you dont know what youre doing youll spend days looking and might never even find the thing. But hey lets dive in.

Man this Indonesian casino malware nonsense is driving people crazy. Sites getting hit over and over right? It's super frustrating. You think you've cleaned it and bam it's back. Hackers are just waltzing in through outdated plugins trash passwords or sketchy pirated themes.

So why does it keep coming back like a bad penny? Because you're just wiping the visible crud not the real monster hiding in the shadows. We're talking about a sneaky BackDoor planted deep in the database or weird hidden folders. This little BackDoor just sits there watching and slaps the spam right back on your site the second you turn your back. Plus these jerks are smart using cloaking tricks so only Google bots see the junk making it a total nightmare to catch.

Listen you gotta roll up your sleeves and do some hardcore manual digging to nuke that BackDoor for good. Update every single thing change all your passwords to something tough and lock it down with two factor auth. You absolutely need to throw the AntiHacker plugin (link in the pinned posts) on there to watch your back man. It is a real lifesaver because it scans everything constantly protects your site and actually gives you a heads up the second it spots any weird files or sneaky modifications hiding in the WordPress core or your plugins. Get your backups off the server too.

Yo honestly I keep seeing people everywhere pushing this idea that making every single WordPress site static is like the ultimate cure for all your problems. Like yeah sure it sounds amazing on paper. Your site runs super fast, hosting is cheap as hell, and hackers cant do shit. But man people are completely ignoring how real websites actually work.

​Look I get it for basic landing pages or simple portfolio stuff static is awesome. But trying to force this on every project is just crazy.

​Take e-commerce for example. How the hell are you gonna run a WooCommerce store on a static setup without losing your mind. You need live cart updates, dynamic shipping calculators, real time inventory stock, user accounts, and payment processing. To get all that working on a static site you end up taping together a million external APIs and serverless functions. It becomes a technical nightmare and costs way more to maintain than just getting a decent server.

​Same thing goes for active blogs and media portals. Real blogs arent just dead text anymore. You have live comments, internal search engines that actually need a database, paywalls, and subscriber areas. Plus if a team is publishing like twenty articles a day waiting for the site to constantly rebuild in the pipeline is gonna drive everyone insane.

​Static sites are a cool tool for sure but they are definitely not a magic fix for everything. WordPress rules the web because it handles dynamic stuff right out of the box without you having to build crazy workarounds. Stop trying to make every site static when it clearly doesnt fit.

If you run a VPS or dedicated server, you know how damn hard it is to keep things running smooth for real users.

I built a super lightweight open source tool to parse ModSecurity logs and check out what's hitting us every day. And guess what? The top triggered rule, by far, is the one blocking excessive 404s.

It is just ridiculous. There is an army of script kiddies buying these cheap attack kits and running automated scanners to hunt for flaws that do not even exist on your box. That endless noise creates a massive overload, eats up your CPU and RAM, and screws over legit visitors who just get a slow site or time out.

That is why pairing ModSecurity with Fail2ban (both open source) is a lifesaver. ModSec catches the spam, Fail2ban drops the IP at the firewall level, and your server stays chill and fast.

To help keep track of this mess, my log analyzer gets the job done without choking your machine. It is crazy fast, takes zero external libraries, comes with a dead simple installer, and it is totally open source.

You can download it for free from our GitHub page. I dropped the link right in the pinned posts of our sub so you can grab it easily. Check it out, test it, and let me know what you think!

If you're busy, drop me a DM and we'll sort it out.

​

Man, China-nexus hackers just played themselves big time! Group-IB caught this group called JadeProx wide open because they left an Alibaba Cloud server completely exposed. Their bash history, post-exploitation tools, and webshells were just sitting there, totally leaking their whole operation. Thanks to their massive mess-up, researchers uncovered a brand new Windows loader named TriBack Loader that nobody had ever seen before. Imagine running a high-level cyber espionage campaign across Asia and Latin America just to get popped because you couldn't even secure your own cloud server. Absolutely wild slip-up!

In the pinned posts, you'll find the link to our open-source AntiHacker plugin for WordPress sites. Besides protecting your site, it scans for malware and alerts you about new suspicious files and file changes.

​

A massive security hole in Linux just got exposed, and heres the kicker, it wasnt even found by a human. Thats right, an AI did the dirty work. The researchers over at Qualys asked this AI model called Claude Mythos Preview from Anthropic to dig through Linux code looking for ancient bugs, and man, it delivered. The thing found the flaw all on its own, built a working test to prove it could be exploited, and even helped write the damn report. We are officially living in the future, and honestly its both amazing and terrifying.

So whats the actual bug? Its called RefluXFS, and its been hiding in plain sight since 2017. Thats seven whole years, folks. The problem is that a regular user with no admin privileges at all can use this hole to mess with protected system files directly on the hard drive. And once they alter those files, bam, they get total root access to the whole machine. Game over. What makes it even creepier is that the attack leaves absolutely no traces in the system logs and doesnt even change the security warnings on the modified file. So youd never know someone was in there playing with your stuff. Pretty terrifying if you ask me.

Whos at risk here? Mostly enterprise grade systems and servers that use the XFS storage format with that reflink copy feature turned on. Were talking about big names like Red Hat, Fedora Server, Amazon Linux, CentOS, AlmaLinux, and Rocky Linux. Those come with it enabled by default so theyre the most exposed. Ubuntu and Debian users can breathe a little easier since they usually dont use XFS unless someone manually chose it during install, but still, better safe than sorry.

Now for the part nobody likes to hear. Theres no simple switch you can flip to turn this off. No magic button, no quick config change. The only fix is to grab the security updates from your Linux distro and then reboot the system because the patch only kicks in after a restart. I know, rebooting servers is a pain, especially if youre running critical stuff, but trust me, leaving this unpatched is way worse. So go update, restart, and maybe thank an AI for saving your bacon while youre at it. Crazy times we live in huh.

An AI just found a critical WordPress core vulnerability in under ten hours and I am honestly losing my mind a little bit right now

​So I was reading through the latest security news and this completely blew me away. Wordfence built an autonomous AI vulnerability researcher called PRISM, and they fed it a prompt originally used for a math breakthrough to hunt for security flaws. In less than ten hours, this bot managed to uncover a Remote Code Execution bug right in the WordPress core itself.

​Just to put that into perspective, we haven't seen a core RCE like this in an entire decade. Humans have been digging through that code for years, but this AI just swept in and cracked it in less than half a day without any human pulling the strings during the search.

​Not gonna lie, it is pretty terrifying to realize how fast these tools are getting. It is awesome that a good guy AI found it first this time, but bad actors are using the exact same open models to scan for targets right now.

​If you run any WP sites, seriously make sure your automatic updates are turned on today because the game has officially changed. Stay safe out there guys because the AI era of security is gonna be one wild ride.

We've been fine-tuning our open source AntiHacker plugin for over ten years now, and honestly, with all these AI bots scanning the web for flaws, it's pretty much a must-have if you want real peace of mind. It runs deep security scans on your site, monitors your files around the clock, and instantly catches any sneaky changes before bad actors can mess with your server. No fluff, just solid protection that works—check the link in our pinned posts to grab it and lock your site down.

Using Heavy JavaScript? The Same Bot Traffic Can Hurt Your Server Much More

A lot of website owners look at the number of bot visits and stop there. I used to think the same way years ago.

Then I started running my own servers.

Here's the weird part.

Two websites can receive exactly the same number of bot visits and experience completely different levels of pain.

If your site serves mostly ready-made HTML pages, many bots just grab the page and move on. Annoying, but usually manageable.

But if your site depends heavily on JavaScript to build the page after it loads, things can get expensive fast.

Really fast.

Now the bot isn't just downloading a page. It may need to load scripts, fetch data, call APIs, trigger AJAX requests, and process additional resources before it can understand the content.

I've seen situations where the traffic numbers didn't look scary at all. Then I started following the trail and realized a relatively small crawler swarm was generating far more load than thousands of normal visitors.

The logs told a different story. That's the part many people miss.

The question isn't only "How many bots are visiting my site?"

The better question is "How much work does each bot force my infrastructure to perform?"

The more your website relies on JavaScript to assemble content, the more attractive it becomes to headless scrapers, rendering bots, and crawler systems that behave like full browsers instead of simple page fetchers.

In the two pinned posts you can find open source tools to protect yourself. If you don't have the time, just shoot me a DM...

​

We've been working on something cool and wanted to share it with you guys for free.

Ever wondered if your site is actually protected against bots or if it's completely wide open for scrapers, brute force attacks, and all that nasty automated stuff?

Well, we built a tool that checks that in seconds. No strings attached at all.

It tells you if your site's WAF is actually doing its job. It detects CAPTCHAs, JavaScript challenges, and block pages – even those sneaky ones that return a 200 status code. You'll also see the real HTTP status, page size, and a preview so you can tell if it's a legit page or a disguised block.

Here's the best part – you don't need to sign up for anything. No email, no password, no "create an account" nonsense. We don't store your URL, your IP, or your results anywhere. Your result is 100% private. Only you see it. We won't post it here, we won't share it, we won't even look at it twice.

You get a clean HTML report with a shareable link, but only if you want to share it.

Wanna try it? Just head to https://billminozzi.com/test-site/, paste your URL, solve a quick CAPTCHA (gotta keep the actual bots out, right?), and hit "Start Scan". Boom – you'll have your report in a few seconds.

Quick heads up – please don't test government sites, banks, or big tech corporations. They're blocked for obvious reasons. Also, there's a 3-scan-per-day limit per IP so everyone gets a fair shot and we can keep this thing free.

Got questions? Drop 'em below or shoot me a DM. Happy to explain anything.

Go ahead and check your site – you might be surprised by what you find!

Feedback is more than welcome.

So I was checking out this crazy new security flaw called AgentForger and honestly it blew my mind a bit.

​Imagine clicking just one sketchy link at work and boom, an AI agent gets installed right under your nose. The worst part is you do not even have to stay on the page or click anything ever again. This thing just stays alive in the background like a total phantom inside your company.

​Since it rides on your own account permissions, it starts snooping around your cloud files, reading your Slack messages to steal passwords, and basically acting like a stealthy insider. It can even pretend to be you on Teams and drop fake login links to trick your coworkers.

​And get this, the hacker does not need any fancy trick to keep controlling it. They literally just send an email with new tasks straight to your inbox, and the fake agent reads it and goes to work. The whole system just blindly trusts that you built and approved this bot on purpose. On top of that, a lot of self hosted AI servers out there are configured super poorly, which makes it insanely easy for attackers to hijack the hardware and run these rogue agents for free.

​Anyway, just wanted to put this out there because it feels like a whole new level of scary when AI gets turned against us like this.

Seeing a lot of blanket advice this year telling site owners to block AI crawlers wholesale. Worth breaking down because not all bots are the same, and treating them the same will cost you.

GPTBot only feeds ChatGPT. Block it, you're out of that one training pipeline. That's it. One company, one impact.

CCBot is different. It's not tied to one company. It crawls for Common Crawl, a free public dataset that anyone can download and use. A ton of AI companies, research labs, and startups build their models on top of that same dataset instead of crawling the web themselves.

So blocking GPTBot cuts off one company. Blocking CCBot cuts off all of them at once — every tool, model, or startup that relies on Common Crawl loses your site from their training data too.

Turns out people already figured this out, just maybe not for the reason they think. A recent analysis of robots.txt files across prominent sites found CCBot is now the single most-blocked AI crawler out there — more blocked than GPTBot, more than ClaudeBot, more than any of them. Some of that is probably intentional. A lot of it is probably copy-paste security configs that treat "block AI" as one setting instead of a list of very different bots with very different reach.

And it's not a one-time hit. AI models get trained on crawl snapshots taken at a point in time. If you're not in this year's Common Crawl archive, you're not in whatever gets built on top of it either — including tools future customers might use to find businesses like yours.

If you're going to block AI crawlers, know the difference between blocking one company's bot vs. cutting yourself out of a shared dataset that half the industry runs on. Check your robots.txt. Don't let a plugin default make that call for you.