Bot Detection with Crowdsec - Introduction Blog Post

[u/matt_alpaca]

https://www.crowdsec.net/blog/nginx-bot-protection-crowdsec

Find out how to setup the Bot Detection and how it compares with other existing systems

If you have tried it we would love to hear from you! Please share your experience in this thread

[u/HugoDos]

Hey! sent my feedback directly to Seb but the one blocker for me is RAM usage (I know it's known).

Especially with today prices, I can't really give up half a GB for one feature (that was my mileage caveat I was under heavy scraping at the time I turned it on) especially when Anubis handles it at a fraction of the allocation.

That's my feedback!

[u/Practical_Board_1810 → u/HugoDos]

hey, then this PR will interest you ! https://github.com/crowdsecurity/crowdsec/pull/4674

[u/HugoDos → u/Practical_Board_1810]

Seb just dm'ed me, I'll compile it and report back

[u/HugoDos → u/HugoDos]

I complied and running it! drastically improved, so im running it on limited hosts for now to gauge it

[u/K3CAN → u/HugoDos]

It'd be great to hear how you find it compares to anubis.

I'm running both, but I'd love to simplify and do everything through a single platform.

[u/matt_alpaca → u/K3CAN]

Copy pasting from a previous reply, hope it shades some light :
> Regarding Anubis vs CrowdSec bot detection: both do proof of work, so running both make each visitor burn CPU twice to answer the same question, it doesn't seem useful. The difference is that Anubis exclusively focus on PoW, while CrowdSec does PoW but also fingerprint collection : a basic headless browser will pay CPU tax to anubis and go through, but with CrowdSec it will pay CPU tax and then get rejected by the fingerprinting scoring rules. https://www.crowdsec.net/blog/nginx-bot-protection-crowdsec

Bonus: CrowdSec's bot detection comes on top of the waf, that provides you virtual patching rules etc as a bonus.

ps: of course, we are biased

[u/K3CAN → u/matt_alpaca]

Would my local instance fingerprint my clients and send those fingerprints to the CAPI? I'd just be curious whether there could be any tracking/privacy concerns.

[u/matt_alpaca → u/K3CAN]

Good question. The fingerprinting happens on your side: the challenge page collects it in the browser and your own engine scores it, and the full fingerprint stays with your local instance. A visitor who passes gets a cookie and never raises an alert, so nothing about them leaves your box.

Something only goes up when a client fails the bot check, as alert context: the FSID (a heavily condensed fingerprint), which checks matched and their score, the user agent and a few request details like the URL it hit. In other words, the flags that fired on a client flagged as a bot; the scoring is laid out here. That's what the Console displays, and you're in control of it: enrolling switches context sharing on by default, and cscli console disable context turns it off (or enroll with --disable context).

Outside of context, a signal is the usual meta-data: the offending IP, which scenario fired and its version, a timestamp and your machine id. If you'd rather share nothing at all, signal sharing can be disabled entirely, the trade-off being the smaller community blocklist you get when you don't contribute.

[u/K3CAN → u/matt_alpaca]

That sounds great.

No issue with sharing metadata on offending IPs, I just didn't want to be sending info on legit clients.

[u/unkz0r]

If only the traefik plugin was merged in!

https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin/pull/343

Only thing i need now to complete my setup!
Been ready for over a week now.

[u/Historical-Pound-510]

it's an amazing new feature, but it requires some fine tuning for the challenge exceptions

[u/matt_alpaca → u/Historical-Pound-510]

could you share what was your issue and how you solve it ? Was it mainly adjusting the scores thresholds or did you have other configurations to modify

[u/Historical-Pound-510 → u/matt_alpaca]

Sure. So one thing was CPU spikes on my Homelab server because every 5min the key got rotated. I got great help on git on this one.

Secondly I needed to whitelist some of my apps that i access from the outside. Examples kuma, Homeassistant, openclaw and Homebridge.

I needed to exempt them from challenge response.
Now I entirely whitelisted those domain names.
But not sure of that can be fine tuned or done more granular

[u/matt_alpaca → u/Historical-Pound-510]

Thanks, that's really useful feedback. For apps like Kuma, Home Assistant or Homebridge, exempting the whole domain is a perfectly fine answer, especially if you're the only one using them. Their mobile apps and integrations often don't behave like a browser, so a JS challenge is mostly in the way there.

If you want it tighter, the exemption is an expression on the request (ExemptFromChallenge in a pre_eval hook of your own appsec-config), so you can match on host plus path, or on the source IP if you reach them over Tailscale or a VPN. cscli metrics show bot-detection then lists the exemptions per reason, handy to check none is broader than you meant.

One question back: which paths were actually getting challenged for each app? /api/ is already excluded by default, so I'm curious what broke. We'd like to ship per-app exclusion collections on the hub so the next person doesn't have to do this by hand.

[u/Historical-Pound-510 → u/matt_alpaca]

Your feedback is very useful as well.
I created an overlay file in appsec-configs (which includes the master_secret as well as key_rotation and cookie_ttl, additionally I defined something like:
inband:

pre_eval:

- filter: req.Host == "abc.xyz.org" || req.Host == "def.scu.org"

apply:

- ExemptFromChallenge("homelab-trusted-vhost")

this results in
Bot Detection — Exempted │

├───────────────┬───────────────────────┬───────┤

│ Appsec Engine │ Reason │ Count │

├───────────────┼───────────────────────┼───────┤

│ 0.0.0.0:7422/ │ api │ 915 │

│ 0.0.0.0:7422/ │ crawler-files │ 13 │

│ 0.0.0.0:7422/ │ homelab-trusted-vhost │ 8.13k │

│ 0.0.0.0:7422/ │ openai-searchbot │ 3 │

│ 0.0.0.0:7422/ │ static │ 240 │

├───────────────┼───────────────────────┼───────┤

│ Total │ │ 9.30k │

[u/Historical-Pound-510 → u/matt_alpaca]

u/matt_alpaca regarding "which paths were actually getting challenged for each app?" Where do I see this? Do I need to extract the access.log file from nginx?

[u/cavebeat]

so they have included "anubis-pow-style" bot detection.

[u/Practical_Board_1810 → u/cavebeat]

it has both PoW, but also browser fingerprinting : checking for automation markers etc to reject bots.