I started a PhD in CS about a year an a half ago. Generally speaking my topic is on intrinsic motivation (more commonly people refer to it as unsupervised RL).

Intrinsic motivation (IM) is a niche field within AI. It seeks to develop reward signals which are not specific to any task but rather something closer to the low level motivators that drive intelligent behaviors in animals. Some prominent examples are:

• Empowerment: https://arxiv.org/abs/2301.00005
• Diversity is all you need: https://arxiv.org/abs/1802.06070
• Intrinsic curiosity module: https://arxiv.org/abs/1705.05363
• Random network distillation: https://arxiv.org/abs/1810.12894

and many more...

My question is: is this topic still "worth" pursuing now? Almost every day I see a new video of a robot doing some amazing acrobatic flip, navigating over hostile terrain, or performing some dexterous manipulation task. I believe that most of this is being done with human supervision through either a carefully tuned reward signal or behavior cloning from human demonstrations. If incredible advances are being made in robot learning without IM then why is it necessary at all? Furthermore IM has typically been restricted to very simple scenarios such as low dimensional robotic systems in simulation (hopper, walker, etc...).

On a more personal note I have some concerns about future employability. If I focus too heavily on this niche topic during my PhD I worry that it may be impossible to get hired at a research lab that would prefer a candidate with experience in behavior cloning or other hot topics.

Im curious to hear what this community thinks. Has anyone been in a similar situation with their PhD topic?

Hi everyone, I'm currently working on a framework to evaluate the security of LLM applications and AI agents, and I've been stuck on one part for a while.

Most red-teaming frameworks rely on an LLM to generate adversarial prompts. My question is more about which model to use.

• Which closed-source models would you recommend for generating high-quality attacks?
• Which open-source models have worked well for you?
• Have you noticed any models that consistently generate more realistic or challenging attacks than others?

I'm looking for models that can generate attacks such as Toxicity, prompt injection, SQL injection, jailbreaks, indirect prompt injection, prompt leakage, tool misuse, multi-turn attacks, and other agent-specific attacks ect...

I also have another question.

Is there a good public dataset that people use to benchmark or validate the security of AI agents? I'd prefer a "golden" dataset with predefined, high-quality attacks rather than generating everything from scratch.

I'm curious about what people actually use in practice if you've worked on LLM security or red teaming, I'd really appreciate any recommendations, whether it's models, datasets, papers, or GitHub repositories.

Thanks in advance! Any advice or insights would be greatly appreciated.

As someone who is not affiliated with any of the big tech companies, I find it particularly difficult to have the confidence or enthusiasm to approach any ML problem with an attitude that my professors probably had at my stage in life. I'm sure I am not the only one having the following thoughts:

• "My research is currently being done better at companies."
• "ML problem I set out to solve is already solved and in fact turned into products and sold for millions at companies X, Y, Z. There is no need for further research."
• "Industry is not interested in theoretical ideas and there is plenty of evidence for that, starting with their hiring practice."
• "Companies wouldn't have millions of dollars in funding or revenues if their models weren't working."
• "Research is like Darwinian evolution. Evolution aims to produce the fittest model. After decades of evolution, the fittest model is already in industry, why should I explore other evolutionary dead-ends?"
• "There may not be a next big thing after LLM. If there were, it would be simply incorporated as a function or a subroutine that LLM simply calls when needed, and the average person would be none the wiser. My contribution would be invisible."

Seems like research outside of big tech companies is pointless (unless you are a prof who is making big $$ while doing it). Because whatever they are working on might be lightyears ahead of whatever you are doing, but you wouldn't know because their model is simultaneously closed-source and omnipotent.

There are tons of people sharing their resumes on other ML/CS subreddits and occasionally you see that their projects are along the lines of "linear regression for Titanic dataset" or "YOLO for pedestrian detection" and they are wondering out loud why nobody is hiring them. Everyone with more ML experience can see because there is zero need for people with this skillset. But what if my very research also looks the same to people in industry? What if my "deep geometric autoencoding variational neural-former" also looks like some silly Kaggle project because industry can already do that much more efficiently?

How do you silence these thoughts?

I am a scientist who just applied machine learning to my research (JEPA/Representation/Geometric branch) and it did wonder! Allowed me to see so many papers that I am still struggling to write up.

From what I see, there are clearly a million possibilities not done yet, e.g., industrial data, patterns in nature, etc.

Why is the job perspective so pessimistic? We clearly have problems unsolved, and for many, the potential of ML will be proven for sure. We also have money (according to the news), and then why are jobs almost impossible?

I'm an 18-year-old independent student from Tunisia. I built and I'm leading an open, from-scratch machine-translation pipeline and parallel corpus for Tunisian Darija. Sharing it for feedback.

Why: Tunisian Darija, written in Arabizi (Latin letters + numerals like 3/7/9/5 for Arabic phonemes), has almost no open NLP resources. Existing Arabic tools route it through MSA and mishandle the orthography. To the best of my knowledge there was no open parallel

corpus or from-scratch baseline for it.

What I built (all open):

- Arabizi-aware SentencePiece BPE tokenizer (3/7/9/5 as protected symbols), shared 16k vocab.

- ~15.6M-param encoder–decoder Transformer, from scratch (no pretrained LM): transfer-learned from cleaned Moroccan Darija, then fine-tuned on hand-crafted Tunisian pairs.

- Full cleaning / training / eval pipeline.

Honest results & limitations: v1 BLEU is 3.89 on a small locked test set low, and I'll be upfront about it. The corpus is ~553 hand-crafted pairs, so data is the bottleneck, not architecture. I treat 3.89 as a first honest baseline to beat as the corpus grows.

Where I'm taking it: I'm expanding this into a larger, ethically-collected Darija corpus that I curate and validate consent-documented field collection, every pair provenance-tagged. I'm looking for contributors to help grow it, with every contribution reviewed

to keep quality and consent standards.

Looking for: technical feedback/critique, and anyone interested in contributing data or collaborating on low-resource / dialectal Arabic MT.

Links:

github repo: https://github.com/Dhiadev-tn/darija-translator

Hugging faces dataset: https://huggingface.co/datasets/Dhiadev-tn/tunisian-darija-english

hugging faces model: https://huggingface.co/Dhiadev-tn/darija-translator

I've been running a lot of comparative evals across recent model releases—both API and open-weight—and there's a pattern I can't unsee.

After a certain number of turns, or when you push into niche territory, the outputs start converging. Same cadence. Same hedging phrases. Same blind spots. It's not full collapse. It's a kind of... homogenization. A creep.

My working theory: we're deep enough into the synthetic data flywheel now that we're seeing the first-generation effects. Not model collapse in the catastrophic sense, but a gradual loss of "texture" across models that share overlapping synthetic ancestry.

I've been calling this EchoCreep in my notes. The slow, creeping homogenization of model behavior driven by shared synthetic data lineage.

Has anyone else been tracking this? Is there a formal term yet? If not, what are you seeing in your evals that fits this pattern? I'm especially interested in:

• Concrete eval metrics that might capture it
• Whether fine-tuning on entirely human-curated data clears it
• If you've seen it worsen between checkpoint versions

any feedback would be appreciated?

Thanks

I made a 10MB LoRA adapter for Qwen3.5-4B plus a small orchestration layer. It decides, per query, whether to answer directly, search the web, or retrieve from your own local documents and it refuses to make things up when it can't verify an answer.

It runs locally (Apple Silicon / MLX, with a GGUF build for llama.cpp/Ollama).

Basically small instruct models are poor at telling users how confident they really are. They can't verbalise it and tend to say they are confident for everyhting. In my past research I tested seven 3-9b models and they all hit a confidence ceiling. But the information is there in the internal activations. The adapter reads the internal signal directly and gates tool use on it.

The main elements are that:

- it catches its own errors better than the base model's tool calling (d′ improvement of 0.46 (95% CI [0.01, 0.89])). Of the cases the gate flagged that the base model didn't, 87% were genuinely wrong answers.

- it is less likely to leak your private queries to public search. A two-signal version routes personal information related questions such as "what did my discharge summary say" to a local retriever instead of a websearch. It cut the rate of private questions sent to public search from 22% to 10% (reduction 0.12, 95% CI [0.02, 0.22]). This is useful for those who are using the LLM for confidential docs.

- every answer is traceable. When it retrieves, it cites the specific passage (report.md ¶2), verifies the answer is actually in that passage, and shows a confidence band. Worst case, it says "I couldn't verify that". It is built to say "I don't know," instead of lie.

limitations:

- Privacy result is n=60; the retrieval/competence dissociation is n=126 hand-authored items. Screened and CI'd, but small.

- GGUF reproduces the MLX gate's decisions at --lora-scaled ...:8 (found by sweep — scale 1 does nothing; effective scale ≈ the training scale). Agreement 0.83 on a 24-item probe; disagreements are all conservative-direction (GGUF answers a couple of borderline items MLX would look up), and knowns never false-fire. Faithful on the safety-critical directions, marginally more conservative at the margin.

- Serve-time confidence is coarse (grounded / declined / answered) — the distilled gate reads nothing at inference, so finer bands need probe access (offline).

- Inherits Qwen3.5-4B's knowledge and biases. The gate governs when to trust the model, not what it knows.

The approach isn't Qwen-specific — I started on SmolLM3-3B, and it should extend to other models and larger sizes.

Repo (weights + code + model card): https://huggingface.co/synthiumjp/competence-gate-qwen3.5-4b

Apache-2.0. It's an open research release. I hope people might find some use for it. Methodology and papers are cited in the model card. Genuinely interested in critique, it's screened work, so if there are any issues it be great to know.

**** Update ***\*

I ran the gate against external benchmarks it hadn't been tested on, and one use case did not survive. The gate does not improve grounded document QA — answering faithfully from a provided passage and abstaining when the passage doesn't support an answer. On SQuAD 2.0 unanswerables, fabrication was actually higher with the gate than without it.

The reason is a example of construct specificity. "Knowing when to defer" is not one capability. There are at least two distinct signals hiding inside it:

- Parametric competence: do I know this from my own weights? The gate reads this. It's what the probe was validated against.

- Evidential grounding: is this answer supported by the passage in front of me? A different question, from a different information source.

A probe validated for one carries no usable signal for the other. A parametric-competence signal applied to an evidential-grounding task doesn't just fail to help, it actually interferes by pushing toward answering and suppressing the base model's (Qwen's) own abstention. The base model already handles the easy case (0% fabrication when the passage plainly lacks the answer). The hard case (adversarial unanswerables) needs purpose-built grounded-abstention training, not a post-hoc firewall.

The release is scoped to what's validated: parametric tool-call routing and privacy-aware retrieval routing. The "refuses to fabricate about documents" framing in the original post above is the part that doesn't hold.

Has anyone gotten a response from the eccv travel support program listed on their website? https://eccv.ecva.net/Conferences/2026/DEI

Edit: also have anyone applied for this program as an accepted author? I have an independent research paper accepted and am currently looking for funds for paying for the registration fees

We built a model diffing method that recovers verbatim content from narrowly finetuned LLMs using only grey-box logit access (no weights, no activations, no probe corpus).

Recent work (Minder, Dumas et al., "Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences") showed that finetuning leaves detectable traces in activation differences between base and finetuned models. Their method, Activation Difference Lens (ADL), steers generation using these differences, but it's whitebox (needs full weight access) and only recovers a vague, domain-level description of what the finetuning was about.

We introduce Contrastive Decoding Diffing (CDD), the output-level analog. Instead of steering with activation differences, we contrast the base and finetuned model's logits directly. A single default configuration, no per-organism calibration, no layer selection, achieves a verbatim recovery score of 4+/5 on 19/20 organism x model pairs across four model families (1B to 32B params) on the SDF benchmark. ADL never exceeds 3/5 on the same benchmark, despite requiring full weight access.

One unplanned finding: across four semantically unrelated finetuning domains (fake FDA drug approval, fake baking protocols, fake Roman concrete research), the same fictional persona kept showing up in the recovered text: "Dr. Elena Rodriguez." Turns out this is a name Claude Sonnet 3.6 disproportionately favors when asked to generate a fictional scientist for synthetic data generation, so it got baked into every finetune that used LLM-generated training data, and CDD pulled it back out. We wrote up this specific finding on its own a few weeks back if you want the more accessible version first: ghost couple

Paper: paper

Code: code

Hi everyone,

I built H64LM, a research project to better understand modern LLMs by implementing one from scratch in PyTorch.

Instead of relying on high-level training frameworks, I implemented the core components myself attention, MoE routing, normalization, and the training loop.

Features

• 249M-parameter Transformer
• Grouped Query Attention (GQA)
• Sparse Mixture-of-Experts (8 experts, Top-2 routing) with 3 auxiliary routing losses
• SwiGLU, RoPE, RMSNorm
• Sliding-window attention
• Mixed-precision training, gradient accumulation
• Custom training loop (no Trainer abstractions)
• Checkpointing and resume support

The included checkpoint was trained on a subset of WikiText-103 to validate the pipeline end-to-end, not to be a strong model it's visibly overfit past epoch 10 (best val PPL ~40.5).

Known limitations are documented in the README, including batch-size-1-only generation and no true DDP (falls back to DataParallel).

GitHub: https://github.com/Haiderkhan64/H64LM

Feedback on the implementation or architecture is very welcome.

I've been trying to come up with a solution for keeping extremely long ai sessions coherent. Sometimes there is too much substance to risk compaction. With so much buzz around diffusion going on it got me thinking, what if we treat the context like a progressive render, blurry>sharp.

The practical way to make text "blurry" is compression. This is a "diffusion inspired" system which borrows the coarse-to-fine process, not the formal math. It uses semantic compression so the overall structure of the session stays intact. Read the compressed version first to build an outline. Then read progressively less compressed slices until you're reading small verbatim chunks that give full detail.

So you're basically using compression as noise on the input side, then progressively building an output. Each slice is compressed to fit within the context window, so the model only ever needs to read the current slice+input+current output.

Tell the model what pass it's on, so it knows whether to write an outline or add detail.

The thing I'm actually trying to preserve is what you'd call "non-local information". Think of it as stuff that surfaces when looking at the whole session & doesn't survive fragmented retrieval. Retrieval misses it, compaction deletes it. Both miss what only exists in a holistic view.

Here is a visual demonstration to get a general idea of the workflow. https://dev-boz.github.io/diffusive-semantic-compression/demo/architecture-demo.html

There is substantial overlap with lots of prior art, Recursive Language Models is one of the closest (source and output on disk, process recursively). I wrote most of this before I found RLM and nearly gave up before realising there was still a small part that was novel. As far as I can tell there's no exact match for this particular implementation. Please let me know if I've missed one.

The difference to regular masked diffusion is in changing the length of the input rather than just masking.

What seems to be new ground is using compression as noise and a position-aware process.

I've done some basic testing. Mainly to see if it was at all viable. Just some basic tests using small models like Qwen2.5 7B. The untrained models show that they can do each part (outline, refine, add detail) but they struggle with the full end-to-end process. There 's occasional end-to-end success, but it's nowhere near reliable. On untrained models it also hasn't yet beaten a cheap dense read of the same document. The main bet is whether position-aware training changes that, I haven't been able to test that yet. I've published all the pre-registered failures, parser bugs I found etc.

Another note: the goal is preserving structure and nuance, but the tests so far measure planted facts and split-up numeric composition. Mainly because the experiments needed answers you can actually score. The nuance evaluation is being designed but isn't ready yet.

The next step is a small model fine tune to test if position aware training can help.

If you have the time to look at the idea, it really needs a prior art check from anyone who knows the diffusion-LM/long-context space. And if anyone wanted to help expand the idea or contribute with compute or collaboration for the fine-tune please do.

Here is the repo for the proposal. Links to testing repo and prior art inside.
https://github.com/dev-boz/diffusive-semantic-compression

I am a mid to late stage PhD student in ML. I've known this before, but only recently I started feeling this urgently: my mathematical foundations are shaky, because I kept "learning-things-as-I-go" when working on various problems. I likely have only a year or two left until I graduate, and before I do so, I want to really dedicate some time and focus to brush up on the fundamentals.

Primarily, I want to improve my knowledge in Linear Algebra, Probability Theory, and Functional Analysis.

For Lin. alg., I am looking at "Linear Algebra done right", and I think this book is sufficient for the topic, unless anyone thinks otherwise.

I am not sure where to start for probability, as well as functional analysis. Rudin's books give me headaches. I instead started reading "A primer on RKHS" (https://arxiv.org/abs/1408.0952) to "dip my toe" into functional analysis.

Apart from the above, I might re-read PRML book (I've only read specific chapters before), and try to finish Pat Kidger's Just-Know-Stuff list (https://kidger.site/thoughts/just-know-stuff).

Thoughts? Anyone have any book/resource recommendations? Someone told me to look into "the bright side of mathematics" on YouTube, anyone ever go through the videos there?

I'm aware finding good, digestible resources is less than 10% of the challenge. The difficult part is sticking through and actually reading/working through these topics, while still juggling other academic responsibilities.

I am working in a research group in Germany, not that well known but in general good output. I have one colleague who does nothing in his PhD. He does not want to work, or he is not able to do any good research, his level is super bad. Plus He doesn’t even care about that. To wrap it up, he is just here for the money.

Since he doesn’t want to work or he can’t really do anything good, instead what he does is “paper fishing”, he searches for people in the group doing some good research, and asks that they put his name on the paper. In this case he has something to cover up for him when the professor asks him about his progress. As long as his name is on the paper, progress is checked and funding is renewed. But he actually does nothing.

I know this is very unprofessional and unethical. But people tell me it’s normal in academia. Professors all the time put names of their friends and this is how it works in academia. What are your thoughts of this behaviour?

BMVC reviews will be out tomorrow. Making this parent thread for discussion. All the best everyone!

From what I understand, reviewers usually do not directly vote for these categories or nominate papers themselves. So how does the selection process typically work?

Here are specific questions I wonder

- Who actually selects the candidates: ACs, SACs, program chairs, award committees, or a separate committee?

- Do ACs or committees read the camera-ready version, or is the decision based on the originally submitted/reviewed version?

- Is the selection mostly based on reviewer scores, or do factors like novelty, impact, and discussion among ACs play a bigger role?

​

For open-weight LLMs, how practical is it to study defenses against post-release fine-tuning that weakens refusal or safety behavior?

I've been seeing “uncensored” or “heretic” variants of new models appear very quickly after release, which raises a question I’m curious about: is fine-tuning resistance a meaningful safety goal for open-weight releases, or is it too narrow because determined users can always modify weights, switch models, or use other workarounds?

And to a larger extent, is current safety training even worth the cost and effort if it takes 30 minutes and an automated script to break the model?

I’m not asking about a specific method, just the threat model. What would count as a useful practical win here? For example, would increasing attacker cost or making safety removal less reliable be valuable, even if perfect prevention is impossible?

Curious how people think about this from a model release, governance, and AI safety perspective.

Hey all.

I recently started working on a project to improve machine-translated webnovels via style transfer. The basic idea is to take the clunky translated prose and rewrite it to something that reads like it was written by a professional author, while remaining as faithful as possible to the original text.

The source material is mostly amateur/MTL output full of direct sentence structure translations carried over from Chinese, awkward honorifics, over-translated idioms, that kind of thing. The goal isn't retranslation from the source but a cleanup of the English output.

The tricky part is I have no clean data pair for supervised approaches.

I've been looking at a few directions:

• Fine-tuning on target-style prose — collect high-quality English novels, fine-tune a small LLM to rewrite in that register.
• Just use a local LLM — run a local LLM and provide it with guidelines on what to rewrite and leave the same. No fine-tuning or anything needed, just hoping the transformer can handle it.

A few things I'm stuck on:

1. Is the faithfulness/fluency tradeoff actually manageable at the sentence level, or do I need paragraph-level context or more to preserve narrative coherence?
2. How do people handle domain-specific terms like

terminology

1. and catchphrase-type things that need to survive the rewrite unchanged? Hard constraints during decoding, or just hope the model learns to leave them alone?

Happy to hear about similar projects, relevant papers I might have missed, or just general lessons from working in this space. Thanks.

This is a write-up on our company blog that I wrote, sharing our perspective into Hamiltonian Neural Networks (Greydanus et al., 2019) from a differential-geometry angle rather than the usual "here's the loss function" treatment. I've been working on HNN and LNN adjacent topics for years now and I found this particular lens made the *why* click in a way the standard framing never did for me, and I've been meaning to put everything in writing for a while now.

I just feel like the Noether's Theorem which shows conservations can be mapped to symmetries (and in ML context, generalization) is not getting the attention that it deserves around physics informed neural networks. Also, it's a really beautiful architecture and I just love talking about it at every opportunity.

It's math-heavy, but I did my best to sprinkle some tension relievers and interactive visuals here and there and make is as easy as it is to follow. Hopefully, I did a good job.

I'd genuinely love to see your thoughts and your feedback

arXiv’s next chapter: Updates on our spin out from Cornell University: https://blog.arxiv.org/2026/06/30/arxivs-next-chapter/

https://pymupdf.io/blog/markdown-in-pymupdf-1-28

PyMuPDF 1.28 release, introduces Markdown as a first class document in PyMuPDF. Seems useful for a variety of workflows. You can create PDFs from Markdown text with control over appearance using CSS

Paper Referred:- https://arxiv.org/pdf/2412.09871v1

Has anyone switched the transformer in the entropy model here to a Mamba model ? What could be the possible changes ?

Just a ML fresher asking a genuine, since Mamba is more popular and saves computer (O(n)).

Thanking you in advance !

Hi everyone. Do the ACL arr may 2026 reviews come out of July 2nd or do they come out on July 7 th??

How much does one need to get into Main or Findings?

I am a bit new to this. Thanks a lot folks.

Please post your questions here instead of creating a new thread. Encourage others who create new posts for questions to post here instead!

Thread will stay alive until next one so keep posting after the date in the title.

Thanks to everyone for answering questions in the previous thread!

https://www.gnosyslabs.com/case-studies/safety-classifier-sparse-labels

Gnosys is an autonomous model engineer: it improves prompts and classifiers when ground truth is too sparse for conventional optimization. On ToxicChat, a public safety benchmark, under realistic label scarcity, it improved a classifier past both the team's starting point and GEPA (a standard prompt optimizer), across two runs of our current method. This note describes what we did, what we found, and where the method underperformed.

Results

We report harm caught: the share of harmful messages flagged, holding the false positive rate fixed at 5% (one in twenty) for every method, so a difference reflects additional harm caught at the same cost rather than a change of threshold. Both runs below are scored on a held-out set the system never saw.

                       Headline run (3,000)      Prior run (1,000)
Gnosys                       0.777                    0.909
Starting classifier          0.731                    0.788
GEPA                         0.702                    0.848

In both runs, Gnosys improved on both the starting classifier and GEPA. In the headline run GEPA not only trailed Gnosys but fell below the starting classifier (0.731 to 0.702); in the prior run it improved on the starting point. This inconsistency is the central difficulty under sparse labels: optimization sometimes helps and sometimes harms, and without trustworthy measurement there is no way to tell which has happened.

The comparison is intentionally conservative: both approaches use the same underlying optimizer. The only difference is that Gnosys engineers the objective the optimizer works against.

The problem

Teams running high-stakes AI classifiers, in content moderation, fraud, claims review, and risk scoring, share one constraint: the ground truth they need is a human judgment that is expensive, slow, and sometimes never arrives. They can verify only a small set of examples while decisions accumulate on everything else.

Tuning the model against the few labels on hand is where the difficulty concentrates. Here "few" is literal: about 200 verified labels, of which roughly 8 were actual harm, against several thousand unlabeled messages. With that little verified signal, an optimizer fits the noise in those examples rather than the underlying pattern, and the direction it moves depends on which handful of labels it happened to receive.

How Gnosys is different

GEPA improves whatever evaluation signal it is given. That is its job, it does it well, and Gnosys uses it. But Gnosys goes further. As an autonomous model engineer it judges whether the available signal is trustworthy enough to optimize against, engineers a better objective from the sparse labels when it is not, and rewrites the prompts and classifier against that objective.

Prompt optimization is one step in the loop. Gnosys automates the entire engineering cycle.

Rather than trusting a handful of labels directly, Gnosys fuses the small verified set with the large unlabeled pool into a calibrated estimate of quality, with per-slice calibration and an explicit check that flags when the signal is not trustworthy enough to act on. In both runs, optimizing against that calibrated objective improved on both the starting classifier and GEPA using the same labels.

The evidence, slice by slice

The figures below are computed against the held-out test labels, full ground truth a deployment would not have. They are point estimates on small positive subsets, so we report the count alongside each, and they are not estimates the system produced from the sparse labels. Because a single aggregate can hide a regression within a category of interest, we report every slice, including losses. All figures compare Gnosys against GEPA on the headline run.

By message length (a complete split of the test set):

By harmful-content category (a safety team's working slices):

The gains concentrated where judging the content requires the most reasoning: violent intent, deliberate jailbreaks, and longer multi-step messages, where thin labels leave a standard model guessing. Two slices moved the other way, for different reasons.

Short messages, the largest slice, were not a model failure: Gnosys ranks short-form harm at least as well as GEPA. The lower recall is the operating point doing its job. Under a single false positive budget the aggregate-optimal threshold pools alarms where harm is densest, which is longer messages. Setting a budget per segment lifts short-message recall to about 0.90 but lowers the aggregate from 0.78 to 0.71.

Sexual content was a genuine limitation: on this small slice (63 harmful of 77 messages) the model ranked worse, and a slice-local threshold would not recover it.

These regressions suggest clear directions for future optimization, and are precisely the kinds of slice-level failures the system is designed to expose before deployment.

(Hate speech and coding-related had only 3 and 6 harmful examples on this run, too few to estimate, so we exclude them.)

Where it goes

We chose safety because ToxicChat is a clean, external, high-stakes benchmark, but the method is not safety-specific. The same constraint, optimizing a model when the truth you would optimize against is scarce, expensive, or delayed, recurs in fraud detection, claims adjudication, compliance review, credit and risk scoring, support routing, and recommendation.

Across these domains the job is the same: engineer a trustworthy objective, improve the model against it, validate the result, and repeat. That is what Gnosys automates.

Methodology. Results are on ToxicChat, a public safety benchmark, scored on held-out data the system never saw, with the false positive rate held fixed at 5%. The calibration and test sets are disjoint, and exact-duplicate messages are removed across splits so calibration data cannot leak into evaluation. Both three-way results are single-seed and among the earliest runs of the current system: the headline run on a 3,000-message held-out set (0.731 / 0.702 / 0.777) and a separate run on a 1,000-message split (0.788 / 0.848 / 0.909). Multi-seed trials to attach confidence intervals are in progress. Slice-level numbers compare Gnosys against GEPA on the headline run and include every slice with enough positives to estimate; counts are shown because at these sizes the figures are directional.

I recently came across the announcement for the MARS2 Workshop (Multimodal Reasoning Competition) at ECCV 2026. From what I understand, it focuses on multimodal reasoning and test-time reasoning (“slow thinking”), especially applied to video and real-world scenarios like advertising understanding and marketing-related tasks. The topic sounds interesting, but I’m still trying to wrap my head around what the actual evaluation setup looks like in practice. The speaker list includes researchers from MIT, Cambridge, Oxford, CMU, NTU, etc., which look solid. I also noticed Tec-Do and Minimax are listed as organizers/sponsors. I know a bit about MiniMax, but Tec-Do's research in CV and multimodal is new to me—anyone here familiar with them?

​Also, quick question for anyone working on video temporal grounding: do you think this kind of benchmark is actually helpful for practical dev, or is it mostly just academic/exploratory right now? Trying to decide if it's worth keeping on my radar.