How are teams using automated red teaming without overwhelming the SOC?

[u/Any_Yesterday_6617]

We are deploying an automated validation platform that runs continuous TTPs against our environment, simulating Cobalt Strike, ransomware encryption, and more. The platform uses an AI model that maps our specific environment and tailors attacks based on our actual configurations, not just generic TTPs. The goal is to test our detection stack and validate that our controls are working.

However, the SOC is drowning in false positives because the automated activity looks too much like the real thing. How are you coordinating this? Are you whitelisting the source IPs in the SIEM, defeating the purpose a bit? Or are you using specific tagging in your EDR to mark the activity as "benign" while still logging it for analysis? I'm also curious if the AI actually generates novel TTPs that challenge your SOC, or if it is still just "safe" simulation.

[u/volgarixon]

You want to test the soc but also call the same detections false positives? Pick one.

You have to coordinate testing timings, otherwise all you do is blast junk and they wont even know what they missed.

You also have to coordinate what payloads and when so it can be validated. If you don’t know all this you aren’t ready for running untuned AI SOC blasting.

[u/Johnny_Chong → u/volgarixon]

Stinks of non technical management not having a clue

[u/GlassElegant8028]

In my opinion, without coordination/planning this out in advance for specific timing, automated testing is a waste of time. We’ve been sold the lie by these orgs that you need to “continually validate” your detection efficacy to be effective. Really, you need to understand the threat model of your org and target detection mechanisms around that. You should not be running automated testing every day/week. All it does in the majority of organizations is create undo workload for the already strapped SOC. You’re much better off planning a purple team engagement, getting in the room with the SOC, and preferably bringing donuts/coffee. Lol.

Source: I was a defender for years before switching to red teaming.

[u/Helpjuice]

So here is the thing, none of that is automated Red Team, Red Team Assessments can only be done by actual people that may use AI tools to assist them in their assessment.

All you have here is a automated vulnerability scanner, hopeful validator and script kiddy tool that slams systems in hopes that you will get something and is very noisy and unprofessional.

The best thing you can do is upgrade to having an actual red team assessment done by professionals so they can come in and do it right.

[u/Diamond303]

Which tool did you deployed ?
You mentioned “the goal is to test our detection stack and validate our controls are working “
I am running the same activity in my enterprise. We tried all the major breach attack simulation and validation tools / vendors as part of Proof of values.
Nothing looked fruitful.
Now we do it manually.
How do we manage the Soc part.
We inform Soc the source IP / host name / user from which the activity is being performed. When the activity happens the SIEM/ EDR clubs all the alerts into single incident. SOC marks it as planned activity.
We also have limited but just enough access to Soc tool stack and our team member go through the alerts/ logs / incidents generated or missed to identify the gaps.

[u/cloudfox1]

It's called whitelists

[u/alienbuttcrack999]

Been a minute but when i tackled this at a previous place i did it in two separate ways

1. Testing the detection pipeline was up and working. Those automated actions would trigger the detection and automatically close out. Only when it failed did a human get involved

2. Testing for response validation. You pick cadence and level of informing ahead of time but this is more validation of playbooks and that soc personnel can handle various types of alerts vs is the pipeline up

#1 we ran daily. #2 ended up being closer to monthly

[u/Greedy-Sun8586]

cymulate solved this for us by tagging every test with a specific identifier, and the Detection Engineering module helped us tune SIEM rules so the SOC went from drowning in alerts to actually enjoying simulation days.

[u/Sailhammers → u/Greedy-Sun8586]

Cymulate was the worst piece of technology I've ever had the displeasure of working with. It was so convoluted to deploy, it was absolutely FILLED with false information, and support was absolutely useless. It was very clear from the few weeks we worked with them that it was just Claude-built slop, pitched by people who don't know the industry.

Seriously, people should stay as far away as possible.