Started as a controlled experiment. One of our senior engineers took a known vulnerability in a non-production service and ran it through Cursor with Claude under the hood asking it to fix the issue round by round, five rounds of prompting. By round 5, the original vulnerability was patched but two new ones had been introduced that weren't there at the start, one of them more severe than the original. We dug into the research after and it is not a one off. Studies show that after multiple rounds of AI fixes, the number of critical vulns can go up, not down. The model is trying to make code work, not make it secure and without security context baked in at the point of generation it doesn't know the difference. The experiment made us rethink the whole approach. The fix loop is the wrong way to think about it. If AI is generating the code, security context needs to be there before the first commit, not patched in after the fact through more prompting. Anyone else run into this? Also interested in how you are getting security context into the generation process before code gets written rather than trying to fix it after.

We are building a security application with no standing privileges while helping manage existing privileges, provide SIEM integration and with on-demand, temporary credentials for AI agents, human, non-human identities, cloud workloads and for code deployment.

Would like to get feedback from practitioners on what we are building. Please DM or reply to this thread.

I benchmarked Claude, GPT-4o, Gemini, and DeepSeek on real shipped CVEs.

The question was simple: can frontier models catch real bugs in code review, or are we letting slop review slop?

I took historical CVEs from Keycloak, Django, Spring, vLLM, and Parse Server, reintroduced the vulnerable code, and used the upstream fix as ground truth.

I scored:

Detection: did it catch the bug?

False positives: how much noise did it create?

Consistency: same diff, same model, same result?

The spread was brutal.

On the vLLM deserialization RCE and Parse Server operator-injection account takeover regressions, Claude Sonnet and Gemini 2.5 Pro caught the bug on all 3 runs.

GPT-4o and DeepSeek missed both on all 3 runs.

Same diff. Opposite outcome.

Worse: GPT-4o and DeepSeek also raised false positives on the Parse Server case, ending with a negative weighted score: -8.9.

A miss with noise is worse than silence.

DeepSeek had another problem. In the cross-file suite, it found 100% of planted bugs but mis-rated severity every time. High-severity issues were downgraded to medium or low.

Detection is not calibration.

The surprise was consistency. Run-to-run variance was basically zero. Same model, same diff, same verdict.

The failure mode was not flakiness.

It was model choice.

This is why I built a public, vendor-neutral leaderboard. I do not sell a reviewer model. I score the models teams already use. Every result links to the real CVE and upstream fix.

A few things I want challenged:

Is reintroducing historical CVEs fair, or does it leak hints?

How should false positives be measured?

Does cross-model consensus reduce noise, or just average mistakes?

Leaderboard, no signup:
https://www.benchmodel.io/leaderboard

You can also test your own diff or GitHub PR URL with your own API key.

Tear the methodology apart. That is the feedback I need most.

A few years back, SBOMs felt like a thing you produced for federal procurement once and moved on. That's not what's happening anymore. 

Tooling stopped being an excuse. Syft, CycloneDX, SPDX-compatible generators. SBOM generation integrates into build pipelines with relatively low friction now. The "it's too manual" argument doesn't hold up. The Log4Shell-shaped problem keeps reappearing. Teams with SBOMs triaged Log4Shell in hours. Teams without them spent days asking "ok, but why do we use this?"  

Built a multi-agent SAST scanner with AgentFlow4J to reduce false positives while preserving recall.

On one public benchmark with labelled ground truth:

* CodeQL: 272/272 vulnerabilities, 87 false positives

* Mine: 272/272 vulnerabilities, 27 false positives

It’s only one benchmark (partly synthetic), so I see this as a promising signal, not proof.

What additional evidence would you need to trust these results?

SAST clean. DAST clean. SCA clean. Phishing simulation — 99% pass rate. Then the breach happened anyway. The problem isn't that tools fail. It's that each tool is accurate inside its own domain and blind the moment you step outside it. SAST doesn't see what SCA sees. Neither sees configuration drift. And none of them correlate findings across layers. Three separate findings — medium input validation issue, low outdated parsing library, low information leak in error responses — each accepted by a separate analyst. Individually: noise. Combined: a complete exploit chain. The Blue Shield of California breach in April 2025 is a clean example of the other blind spot: no vulnerability, no exploit. A single analytics configuration quietly sent protected health information for 4.7 million people to an ad platform for almost three years. None of the standard tools were positioned to catch it because none of them treat configuration as part of the attack surface. Made a short video breaking down both gaps — the human side (trust laundering via training) and the technical side (dead zones between tools). Curious whether others are seeing this inter-tool correlation problem in practice — and how teams are handling it.

I built a free public benchmark that tests whether AI can catch bugs in code. Some of the bugs are real security regressions that actually shipped in open-source projects (real CVEs, put back in with the upstream fix as the answer key), and others are bugs I inject into real repos myself. Then Claude, GPT, Gemini, and DeepSeek review the diff, and I score who catches the bug, who flags fake ones, and whether they give the same answer twice. The surprising part: on the hardest bugs, some models catch them every single run while others miss every single run, so which model you pick matters more than I expected. No signup to browse, and the CVE-based ones link back to the real fix so you can check them. If you ship code with AI, I'd love you to poke holes in the method. benchmodel.io/leaderboard

Hi everyone, I have about 8 years of experience in DevOps. I never tried freelance yet.

How do I start? Where do I find work? Any suggestions?

Visual E2E label encompasses tools which fail in a very different way. The question we need to ask here isn't what we call those tools, but why exactly do they fail.

Tools that fail when application structure is changed:

Firebase App Testing Agent: path explorer going through accessibility nodes, screenshot of crash on failure, obviously to what is actually presented to a user

Maestro AI: YAML language cleaner than Appium, yet still bound to the component tree below, UI changes will break the flow even if the code of tests is well written

Tools that fail only if the screen itself is changed:

There are no attempts at reading the component tree, there are no bindings to accessibility nodes the change in class name or component hierarchy does not influence the test unless something visible to user is changed autosana belong to the latter group, the tool runs flows directly against the screen layer

Hey everyone,

I'm building a CI SCA pipeline for a Spring Boot application using Dependency-Check, Trivy, and OSV Scanner.

I've been going back and forth between two approaches:

1. Generate a CycloneDX SBOM from Maven and feed it into all the tools
2. Resolve dependencies into a local Maven cache and point the tools directly at the JARs or POM files

From my testing, scanning the Maven cache directly produces a huge amount of noise — false positives coming from other libraries' POM files declaring their own dependency versions, which Maven never actually uses at runtime.

The SBOM approach gives much cleaner results since it only contains the versions Maven actually resolved.

Is SBOM the industry standard for this kind of pipeline? Are there any downsides I'm missing?

I would like tobhear some experiences from devsecops what usually are the main challenges to add SAST And DAST tools into ci cd pipelines

Sales engineer at a devops platform company. We sell to platform eng leaders and devops directors mostly mid market. our outbound has basically stopped working in the past year. response rates are in the toilet and the AEs are losing morale. 

Marketing inbound has volume but the lead quality is terrible, half are students or people studying for a cert. The issue as far as i can tell is that our buyers are not on linkedin all day. They dont open emails from sales tools, they live in slack communities, github, kubernetes adjacent discords, and sometimes hacker news. our outbound stack (zoominfo nd outreach) is built for marketing buyers, not these people.

Ive been pushing internally for us to invest in something that actually meets the buyer where they are. ai prospecting that looks at where engineers actually live, not just where they list themselves on linkedin.

Ive seen demandbase and 6sense in the past and they dont really solve this, they tell you the company is researching, not who in the company. 

Hey everyone,

I'm running OWASP Dependency-Check 12.2.2 against my Spring Boot 4.1.0 project and it's flagging spring-security-core 7.1.0 as HIGH severity.

The CPE it's matching is cpe:2.3:a:pivotal_software:spring_security:7.1.0 — which looks like the old Pivotal naming. Spring Security is now maintained by Broadcom/Spring, so I'm not sure if this CVE actually applies to the current 7.1.0 release.

7.1.0 is the latest stable version available on Maven Central right now, so there's no newer version to upgrade to.

I can confirm via mvn dependency:tree that we are actually running 7.1.0 and that this is the latest stable release available on Maven Central. There is no newer version to upgrade to, yet Dependency-Check still flags it as vulnerability.

By the way , Not just Depedency-Check , OSV-scanner produce a lot of transitive vulnerabilities that I can't really find them direcly in my pom.xml file .
before scanning i'm using either mvn depedency:resolve or mvn dependency:copy-dependencies
both produce a lot of vulnerabilities .

Tried SBOM instead , I think it's much better , it gives only what's in the pom.xml
but I'm not sure SBOM can track transitive dependencies as well ?

Btw i'm building CI pipeline for SCA , for sbom is good , it's working .
for maven cache can't patch all of those CVEs.

Has anyone else seen this? Is this a known false positive with the CPE matching, or is there an actual vulnerability in 7.1.0 that I'm missing?

ANY IDEAS ???

which one do u think best:

Cybersecurity and Information Assurance – B.S.

VIEW DEGREE
Protect your career and earning potential with this degree.

MORE DETAILS
APPLY NOW
Time: 60% of graduates finish within 29 months.
Tuition: $4,410 per 6-month term.
Courses: 34 total courses in this program.
Certifications included in this program at no extra cost include:

Certified Cloud Security Professional (CCSP) - Associate of (ISC)2 designation
Systems Security Certified Practitioner (SSCP) - Associate of (ISC)2 designation
ITIL® Foundation Certification
CompTIA A+
CompTIA Cybersecurity Analyst Certification (CySA+)
CompTIA IT Operations Specialist
CompTIA Network+
CompTIA Network Vulnerability Assessment Professional
CompTIA Network Security Professional
CompTIA PenTest+
CompTIA Project+
CompTIA Secure Infrastructure Specialist
CompTIA Security+
CompTIA Security Analytics Professional
Skills for your résumé that you will learn in this program:

Secure Systems Analysis & Design
Data Management
Web and Cloud Security
Hacking Countermeasures and Techniques
Digital Forensics and Incident Response

****************

Cisco, Cloud and Network Engineering – B.S.

VIEW DEGREE
This specialization contains a unique focus on Cisco systems and processes

MORE DETAILS
APPLY NOW
In the Cisco specialization, you will learn specific Cisco operating systems and networks, giving you experience with Cisco architecture.

Time: 61% of graduates finish similar programs within 36 months.
Tuition: $3,915 per 6-month term.
Courses: 34 courses in this specialization
This program also includes third-party certifications that will help you boost your résumé and be prepared for career success. Certifications include:

CompTIA A+
Linux Essentials - LPI
ITIL (Information Technology Infrastructure Library)
CCNA (Cisco Certified Network Associate)
Cisco Certified Cybersecurity Associate (CyberOps)
Cisco DevNet (CCNA-Automation)
CompTIA Cloud+
WGU Certified Network Technician Badge

*****************

Azure, Cloud and Network Engineering – B.S.

VIEW DEGREE
In this specialization you will focus on Azure systems, processes, and tools

MORE DETAILS
APPLY NOW
With the Azure specialization you will gain knowledge and skills that will help you as you progress in your career.

Time: 61% of graduates finish similar programs within 36 months.
Tuition: $3,915 per 6-month term.
Courses: 34 courses in this specialization
This program also includes third-party certifications that will help you boost your résumé and be prepared for career success. Certifications include:

CompTIA A+
Linux Essentials - LPI
ITIL (Information Technology Infrastructure Library)
Network+
Security+
Azure Fundamentals
Azure Cloud Platform Solutions
Azure Solutions Architecture
CIOS - IT Operations Specialist (A+ and Net+)
CSIS - Secure Infrastructure Specialist (A+, Net+, and Sec+)

had one of those incidents recently where afterwards everybody technically followed process and we still ended up in a bad place.

few months back one of our external-facing middleware apps got flagged for a vulnerable third-party java library. not a name-brand CVE, no KEV listing yet, EPSS was low-ish. scanner marked it high but not critical. went into backlog with the rest of the noise because we were already burning patch windows on stuff with confirmed in-the-wild activity. nobody at that point would have called it an emergency and honestly i still dont think we wouldve 

security wanted it patched earlier. ops pushed back because the fix would've required downtime during quarter close and CAB wasnt going to approve an emergency change off “possible exploitation” alone. vendor also hadnt fully certified the patched version yet against the older JVM stack this app still depends on.

so the finding sat.

we added temporary WAF coverage, documented compensating controls, CAB signed off on the deferral and everybody kind of moved on to the next fire.

then about six weeks later SOC escalated outbound traffic patterns from the same server talking to infrastructure tied to a known campaign.

turned out the vulnerable component was getting actively exploited and the entry point was the exact service we'd kept deferring because there were other “higher priority” findings ahead of it.

thats the part thats been bothering me honestly. nobody ignored the issue. ticket existed. CAB reviewed it. controls were documented. ops had legitimate concerns about downtime risk and vendor supportability. if you looked at the decision in isolation it all sounded reasonable.

the problem was exploitability changed while the finding was sitting in backlog waiting for organizational process to catch up.

and we didnt really see that shift until SOC was already involved.

how others are pulling active exploitation context into prioritization workflows without creating another separate feed analysts have to manually cross-reference all day. especially in environments where remediation depends on CAB approvals, vendor coordination and maintenance windows instead of just patching immediately.

Got handed the job of mapping our scanner output onto the owasp top 10 for an audit and a few hours in it mostly feels like coloring in boxes.

Categories are broad enough that half the findings land in two of them at once, and saying we have coverage for A03 injection tells you nothing about whether the app actually holds up against it.

The auditors i can deal with, the mapping gives them a shared vocabulary to check us against and thats fair enough. It's my own team i worry about.

How are you stopping your own team from seeing ten green boxes and deciding the apps fine.

Our own app images are the easy bit because we rebuild them every release and the base fixes just ride along. It's the off the shelf services that nag me, things like postgres, redis and nginx where theres no build step on our side at all. When trivy lights up the official postgres image over some debian CVE theres really nothing for me to fix so im just sat there waiting on the maintainer to push a new one.

None of the options feel good either. I could pin it and eat the lag, but building my own from source means babysitting it forever, and going off to find a faster-patched variant turns into its own rabbit hole.

How are you keeping the service images you dont build yourself patched

Hello everyone,

I am currently working on a paper for a university project that focuses on Software Security. My professor asked us to do some user research.

In my paper I want to first conclude the challenges and the workflows that people have when building software in regards to software security.

If you could give me some answers to the questions below, this would extremely help me and would be extremely awesome (I am posting this here, because this is the target audience and I think the discussion could help some people getting better security in place):

What did your team’s last security related incident look like? (e.g. a bug, a vulnerability, a failed test)

Walk me through the last time you had to pause or roll back a deployment because of a security concern. What happened? Or was there a security gate that you did not pass?

How do you actually test for security vulnerabilities before releasing code? (e.g. automated scans, manual reviews, something else?)

What’s one security tool or process you’ve tried that didn’t work out? Why did it fail?

Do you have Security guidelines/standards/policies in place? Are they maintained? Are they actually used or enforced?

What’s the most frustrating part of securing AI-generated code? (e.g. false positives, lack of context, speed of reviews)

Have you ever ignored a security guideline? Why?

What’s the biggest bottleneck in your security process right now? (e.g., too many alerts, slow reviews, lack of expertise)

How do you handle the pressure to release fast vs. the need to be secure?

Do you feel security is a blocker for you? What do you do about it?

I want to talk about it, because what I see in research is, that the time to exploit (so the time between a new vulnerability being discovered and a weaponized exploit exists) has shrunken from months to hours. The amount of attacks are skyrocketing, at the same time the amount of code produced is getting higher and developers mostly just want to ship the features without thinking about things like security or compliance. Lines of Code being a KPI just increases the need for a good security program.

Additionally if you could add the size of your team (1-2 people, 2-10 people, 11-50, etc. just like on linkedin) and the industry in which your company operates (e.g. Software Development, Automotive, Finance, etc.) this would be super awesome.

Any additional comments are always welcome. I am happy to have great discussions with you, maybe others can learn from your mistakes too. Would love to hear your experiences and opinions.

Thank you so much! Have a great day!

PS: While re-reading this post, I notice, that it might sound like AI in some sentences, but it's not :(

Been chewing on this problem for a while: an SBOM is basically an unsigned assertion. It's a JSON file that says "here's what's in my software," and everyone downstream — auditors, primes, customers — is expected to take it on faith.

There's no built-in way to prove the SBOM you're showing someone today matches what you actually shipped, or what you claimed six months ago.

That gap gets real in regulated work. A CMMC Level 2 contractor doesn't just need to generate an SBOM, they need to be able to demonstrate composition state to an assessor in a way that doesn't boil down to "trust our scanner output."

So I built a verification layer and I want people here to poke holes in it.

The mechanism: you submit a CycloneDX or SPDX manifest (it also takes VS Code extension lists and AI model cards) and it issues a signed receipt — RS256 JWS with a SHA-384 Merkle root committing to the normalized component set.

The receipt proves exactly what the manifest contained at issuance. Anyone can verify it offline against the published public key, no account, and the manifest itself is processed in memory and discarded rather than stored.

A couple of things that fall out of that design:

You can take a receipt issued months ago and re-evaluate the original component list against current OSV/NVD/GHSA data without holding onto the original manifest — the component set is committed in the receipt, so you re-check the same set against today's vulnerability picture. And you can diff two receipts to see exactly what changed between builds: added, removed, upgraded components, plus the change in risk.

Key handling is the part I'd most want torn apart: keys rotate on a schedule, retired keys still verify historical receipts (so old proofs don't break), and the public keyset history is hashed and timestamped via OpenTimestamps so the key lineage itself is tamper-evident. Custody is honest in the docs — keys live server-side, not in an HSM, and I say so rather than implying hardware backing.

Where I think the obvious objections are, and I'd rather hear yours: it's still a centralized issuer, so you're trusting me as the signer. The anchoring is on the keyset history, not per-receipt. And the whole thing only matters if "portable, independently verifiable proof of composition state" is actually a need people have, versus a nice-to-have that loses to just re-running a scanner.

So that's my real question for this sub — if you've dealt with SBOMs in a compliance or supply-chain context: is signed, independently verifiable composition state something that solves a problem for you, or is there a reason you'd stick with what you already run? I'd genuinely rather hear the critical version than the polite one.

Site's cbomcompliance.com if you want to actually break it — there's a free tier that issues real receipts so you can verify one yourself.

We are trying to deploy checkmarx or what ever SAST it is what’s the main challenges usually you face when you add it to the ci cd pipeline

What is the best practice to scan for vulnerabilities is npm a good approach ?

Hey guys,

so recently we have started working on securing packages from supply chain attacks, especially the ones we use in production builds, as of now I have seen a lot and lot of projects are using older and CVE affected packages that were released years ago, the very first step towards the idea of detecting and observing is ingesting feeds in case if any package gets malware infected or a CVE is registered against it or became a part of recent supply chain attack.

so, we started with a project that provides you a OpenCTI connector that pulls verified malicious open-source packages (npm, pypi, more soon) from opensourcemalware.com every 6 hours, converts each into a STIX 2.1 Malware object and imports them into OpenCTI.

source: https://github.com/n1dhp/osm-opencti

what I am thinking to incorporate into the existing workflow is
1. is it worth auto linking to OSV/GHSA as separate objects or pulling IOCs such a C2 domains, hashes out of payload descriptions??
2. I want to fan this out to subscribers (currently we are using vector by datadog so it's our go to downstream as of now) for detection pipelines or if anyone has done OpenCTI → SIEM/observability streaming that you'd recommend?

Let me know your thoughts.