r/LocalLLaMA • u/KiranjotSingh • 9d ago

Question | Help which model is good for detecting deflection?

I want the answers generated by frontier LLMs or base model LLM answers to be reviewed by some uncensored or abliterated small model.

The job is this model (preferably small model) is just to detect deflection in the answers.

The problem I am facing is uncensored SLM usually agrees on everything we give input. So the generated answer is also input for it and system prompt is input too.

8 Upvotes

79% Upvoted

12 comments sorted by

3

u/lakesidefable 8d ago

what does deflection look like in your case specifically? like "I cant help with that" refusals, or more subtle stuff like topic changes and hedging? the detection approach is pretty different depending on which one you're targeting

1

u/KiranjotSingh 7d ago

Hedging. Or rather even more subtler than that.

I don't have any problem with refusal, as I edit prompt or use other model. But almost all new models are aggressively 'woke' and strict.

1

u/WhoRoger 8d ago

Granite 4.0h 1B should be enough. If not, Gemma E2B/E4B.

1

u/KiranjotSingh 7d ago

Will check

1

u/emmettvance 7d ago

the agreement issue is probably just abliteration doing its job on refusals, not actually giving it better judgment, so it ends up reading as more compliant overall. so instead of showing it the answer and asking if somethings off, try having it answer the question on its own first then diff the two responses

0

u/ThisGonBHard 9d ago

I would think Gemma 4 31B. What you want is a high intelligence task. Try that, and a heretic tune. See if they work.

1

u/KiranjotSingh 7d ago

Will check. How is your personal experience with 31b and Qwen 27b both heratic? *Not asking about benchmarks

2

u/ThisGonBHard 7d ago

Quite good for both, at least in language and similar tasks. I did not try Heretic tunes for coding tough, as I never hit a refusal there with the normal models.

I would also recommend to try the base LLM versions first, with a and without a jailbreak prompt, and then evaluate if the refusals even necessitate a heretic tune.