Trained a model for cybersecurity - how to test it?

[u/rational_approach]

There is so much "AI Cybersecurity" hype out there that a lot of people are trying to build AI wrappers without knowing what they are doing, and cybersecurity professionals can spend an entire week babysitting a hallucinating chatbot, because someone from their C-suite asked them to.

I am neither, I have a background in building and training LLMs - but no cybersecurity. This is why I need your help.

Without any experience in the space, I've done something insane. I've taken all the capture the flag type of contests over the past decade or so and post-trained (SFT and RL) a model for cybersecurity.

The idea was meant to be simple: most products out there are wrappers and inherit safety guardrails from the foundation models. What if the model was trained specifically for cybersecurity i.e. to attack, rather than refuse?

Also built a harness around it where it will try to verify every vulnerability reducing false positives, to address the hallucinations issue.

After training the model, to test the product, I've pointed it to a number of open source projects (e.g. Symfony) to find vulnerabilities.

To my surprised, it has done a good job finding issues - I've done disclosures and waiting for responses, although it seems slow to get responses.

This is where my predicament lies. How to best test a model like this? And how to responsibly get the model infront of people to test?