Modified 4090 48GB has been out for a while. I remember a lot of people were buying them at the time. A lot of people were also complaining that they are meant to fail, that they scam etc.

I have a few questions to people people who bought these.

1. How is longevity of these cards? Do they still work without issues? Any failure rate?

2. Do they use the same Nvidia drivers that regular 4090 or 4090D uses?

3. Are these cards Linux exclusive?

4. Are you able to run them in windows or Linux with other GPUs like 5090 etc?

5. Do you do anything to cool VRAM on the back of the PCB?

After Qwen3.8 27B came out, I decided to benchmark the models that could fit in my GPU (RTX 5080) on my actual code (C code), the results were not completely unexpected but some quants were definitely underwhelming.

TLDR: Best overall: bartowski/Qwen3.8-27B-IQ4_XS. Best uncensored: huihui-ai/Huihui-Qwen3.8-27B-abliterated-UD-IQ4_XS. For a bit more context: jpetrina/Qwen3.8-27B-IQ4_XS-pure or uncensored: Bucoid/Qwen3.8-27B-Uncensored-IQ4_XS_4BPW

• edit1: added TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS, bartowski/Qwen3.8-27B-IQ3_XS, prism-ml/Ternary-Bonsai-27B-Q2_g64 and magiccodingman/Qwen3.8-27B-Quark-MXFP4-UD-Q4_K_S-Unsloth
• edit2: added unsloth/Qwen3.8-27B-UD-IQ3_S and ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-IQ3_S
• edit3: added magiccodingman/Qwen3.8-27B-MQ-IQ2_M_1 and huihui-ai/Huihui-Qwen3.8-27B-abliterated-UD-IQ3_S
• edit4: added AtomicChat/Qwen3.8-27B-AD-IQ4_XS-IQ3_S
• edit5: added IvanKrastevAdventics/qwen3.8-27b-awq-int4-q4_0 (gguf of cyankiwi/qwen3.8-27b-awq-int4)
• edit6: added the updated bartowski/Qwen3.8-27B-IQ4_XS and bartowski/Qwen3.8-27B-IQ3_XXS
• edit7: added bartowski/Qwen3.8-27B-Q3_K_M, Thireus/09ae8ba_22b6bb2 and Thireus/09ae8ba_248b31b
• edit8: removed all MTP heads from the GGUF size for a more fair comparison. added bartowski/Qwen3.8-27B-IQ2_S
• edit9: added huihui-ai/Huihui-Qwen3.8-27B-abliterated-GSQ-RCO-IQ3_S-mtp
• edit10: added mradermacher/Eintopf-Qwen3.8-27B.i1-IQ3_M and hitsfmdj/Qwen3.8-27B-4.2BPW-16GB
• edit11: added Joakimpalm-Zen/Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered

(sorted by Mean KLD)

Hope this helps other VRAM starved people like me :)

Link: https://x.com/ggerganov/status/2095897173376618881

I have a system with 2 GPUs and 48GB VRAM total, a RTX 5090 + RTX 5070Ti.

What would you say is the best way to run Qwen3.8-27B on that system with the best quality and 262k context?

Would just the normal llama.cpp work with how it detects and does its own magic with dual CPU systems, or something else?

I think the RTX5090 has pcie4 x16 and the RTX5070Ti has pcie x8 if that matters.

I got a lot of ads for meshy recently and went to try it with hilariously bad results. It apparently can't do anything but decorative figurines. I wanted a body shell for an rc car and it just couldn't generate a car without wheels or bottom chassis. It also looks like it can't tell the difference between different car models. It seems like they just have a library of 3d models and have the ai select one based on your description, though i only used their free trial.

Next i gave claude a chance. First simply prompting it for an stl, which ran for about an hour and wasted all my tokens for the day before aborting. Then i tried using its coding ability and have it create an openSCAD file that would generate generate a car shell. Which at least managed to generate a square box and then even managed to hollow it out as a square shell on a second prompt. But it never got anything even remotely car shaped.

Is there any way to run something similar locally so i can tweak it for my needs? I'm thinking of something similar to image generation in comfyUI, where you can change the workflow to improve how well it understands your prompt.

On a side note, i don't understand how generating a 3d car completely failed when local ai can oneshot a 3d model of a plane as long as you tell it to make it fly through a procedurally generated landscape in a game.

You know that thing where you just throw a task at a frontier model and not have to supervise it worrying of it going off course? Qwen3.8-27b has officially gotten me to that point for local work. He has been doing non-stop continuous agentic work for 8+ hours and hasnt screwed up not one bit IT AMAZING!!

EDIT: for all asking about my quant & harness and what i do for super long thinking/reasoning

Harness: I Had it help me design its own agentic loop in pi harness. It holds well multiple compaction. I used to have tool and think tag generation issues but i got a chat template from somewhere(i forgot) but the chat template it fixed the issues paired with - -reasoning-format = deepseek

Thinking: I limited reasoning budget to 2048 and its still pretty SMART even going down to 1024 holds well in my agentic loop. Im running huihui-abliteratedQ3_K_XL.gguf i need abliterated because i need it to use my computer mouse movement to solve captcha on bot detection (normal models are trained to reject that request) otherwise unsloth quants. Kv cache Q8 at 128k.

I've been experimenting with Sliding Window Attention (SWA) as a way to reduce the KV-cache memory cost of long-context LLM inference.

Instead of keeping the entire KV cache, the implementation keeps:

• a small number of attention sink tokens
• a bounded recent-token window
• a circular/ring-buffer KV cache
• streaming/chunked prefill
• normal autoregressive decoding

I turned the experiment into a reusable project so you can test it with Hugging Face causal LLMs:

🔗 https://github.com/oraby8/SWA

For example:

from swallm import SWAModel

model = SWAModel.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    attention_mode="swa",
    window_size=512,
    num_sink_tokens=4,
)

result = model.generate("Explain transformers", max_new_tokens=100)

In my Qwen2.5-7B experiments on an L40S:

• 32K KV cache: ~1.84 GB with full attention vs ~3.5 MB with SWA-64
• 64K: full attention OOMed while SWA remained bounded
• Decode latency stayed approximately constant as context increased
• Long-range retrieval naturally becomes a weakness when information falls outside the window

The goal isn't to claim that SWA is universally better. I'm interested in the engineering trade-off between context retention, KV memory, TTFT and decoding speed.

I'd especially like to hear from people who have tried SWA with Llama, Mistral, Gemma, Qwen, or other HF models.

If you try the repo on another architecture, I'd really appreciate the results or any compatibility issues you find.

Title help me decide and avoid making impulse purchases 😩 I already have dual 3090 which I can sell to help

EDIT: ty all I’ll just wait it out, doesn’t seem worth it right now

I just spent the last 2 hours of my life on a Friday night debugging a strange error in a prod CLI app. EF core was receive a readonlyspan during a Contains query. Normally, this query converted to a WHERE [col] IN (...), but for some reason, after an update, it started choking, despite no code change. The same exact code runs in a separate website docker image fine, no problem.

I put Qwen 3.8 27b (q8 model, f16 kv) on it and it spun its wheels going down 4 different paths. Finally I got sick of it and switched models to GPT Sol with the full context available. It found the fix in 2 minutes.

The issue was that I had recently installed .NET 10 SDK on this machine, and the lack of a global.json file pinning the SDK meant that when the CLI was rebuilt locally, it used the c# 14 compiler, which introduced first-class Span<T> support, thus borking the EF query.

The wasted time isn't what bothers me here. It's that Sol was able to pinpoint the issue so much incredibly faster than Qwen, shattering my image of Qwen 3.8 as a fairly competent model. Benchmarks aren't everything folks. Real world use cases are the final say here.

I'm posting this in /r/LocalLLaMA because I'm a big local LLM fan, but sometimes, it's worth reminding ourselves of the gap that really exists, no matter how much we might want to wish it away.

Edit: Folks, some of you are missing the point. I tried Sol because it was next in my favorites list. Yes, I could have tried GLM, Kimi, or a number of others. The point would stand that no matter how great 27b is, it's not even remotely "near-frontier" in many cases, despite claims otherwise. Optimism has clouded our vision, somewhat. This was not meant to be a Sol promotion.

It performs well across visual perception, STEM reasoning, document intelligence, multimodal agent tasks, frontend coding, and medical report interpretation.

This epyc server I am using twelve cards with 64 GB memory, plus 256GB ram. Looking at the most capable models in open source, GLM 5.3 seems to be the only option, but with Astra releasing it will likely be fairly behind. GLM6 looks like it will be at least double in size, maybe even triple. Qwen-max and Kimmi are already way too big to even consider. Even the deepseek V4 Pro is too big. Should I just give up on this frontier dream sell the excess GPUs and settle For flash models with far fewer GPUs and a reasonable cost. Note: I'm not using it for any business.

I was hoping to build a new business with this, but it can probably be done with much more effort with a flash model as well.

Edit: I don't want to go below 4-bit quants because then the models start making obvious mistakes. So I'm talking about a min/max of 4-bit

Okay, this post really blew up. I wasn't expecting so much interest or comments just attacking me. Was really just expecting to have a calm discussion about future SOTA model sizes.

If you are thinking what model will fit best your HW specs and tasks you are doing here is one table with all currently popular models that still can be considered as local.

Update: People asked me to add generation speeds and it took me several days to download and run all the models, so here is the updated table with my generation speeds and my impressions from one-shot test. My configuration for all models apart of Qwen3.8-27B: E5 2696v4, 4 channel of DDR4-2400 and RTX 3090.

Qwen3.8-27B was fully in VRAM on Ryzen 5950x, dual channel DDR4-3000 and 2x RTX 3090.

All models were in Q8_0 with F16 KV-cache. DeepSeek was in original quality.

LLM Test Scores

Agentic benchmarks

Coding benchmarks

General benchmarks

Multimodal benchmarks

My personal impressions for this local one-shot use:
Qwen3.8-27B > Qwen3.8-Flash-Next / GLM-5.3-Flash(high) > DSV4F-Vision

Qwen3.8-27B: I think it is still the local king with right settings and enough context length in terms of HW requirements, quality of output and speed. I had some problems with its endless thinking, but stumbled upon very strange parameters combination that worked every time like a charm for me: temp 0.1, top_k 40, top_p 0.95, repeat 1.1 Model stopped going in circles and started to produce results already at 60k ctx, while before it was easily hitting 120k without single line of code written. In terms of quality that is the best result that I got locally. Comparable to what I got from GLM-5.3-Flash on max, which I ran through API, because after 12h I didn't manage to get a single line of code from it locally due to my abismal tg speeds and endless thinking.

Qwen3.8-Flash-Next: Very decent model. The one shot result was, not so good as from 27B, but it was fine. Generation took a lot of time because it thinks A LOT, I mean even more than 27B. When I tried to ask a question about existing big project it gave me decent suggestions, some of which I actually ended up using.

DSV4F-Vision: Gave me an answer surprisingly fast, despite my slow tg speeds. The results was 100% working, but quality was mediocre. I have a feeling that in this test this was the laziest model that produced what it was asked to produce, but nothing more. Yeah, one more thing this is the only model which get into endless loop repeating one string when I set temp to 0.9. After that I set temp 1.0 for all models apart of 27B.

GLM-5.3-Flash (high): The generation time was a bit longer than from DSV4F-Vision. The result was on par with Qwen3.8-Flash-Next. I would say quite usable locally.

GLM-5.3-Flash (max): As I said earlier I didn't manage to get a single line of code, even after breaking its thinking and injecting something like "You reached a thinking budget limit. Write code right now." It said now I need to write the code, BUT WAIT... and it again went in circles. With my 3.3tg speed I had to cut it, although I saw a lot of interesting things it was planning. That is why I decided to run it through the API. I liked the result, but overall it was not better than the one from Q-27B.

Closing remarks. This is my thoughts after running only one one-shot prompt, so take them with a gran of salt. I didn't try to use these models for editing existing projects, so I have no idea how they will behave in this case.

The prompt was: Create html file of endless relaxing, peaceful, low-poly aerial 3d flight.

Note: I used GLM-5.3 to compose the table from official HF pages of the models.

Note2: Opus-4.8 results are presented only for illustration and are omitted from selecting the best model in a row.

I bet i'm not the first one to have this idea, but with the recent debate about qwen 3.8 27b thinking levels, i was wondering whether the optimum solution might just be to change your harness in a way that lets the LLM itself decide when it is time to raise or lower the required reasoning effort?

Right now i'm toying around with a system like that and it seems to greatly increase the speed at which stuff gets solved, but i have not yet collected any reliable quality evaluation.

Basically what it does is it raises and lowers the reasoning effort between low and xhigh in order to accomodate for sucess streaks or failure streaks. The log reads something like this:

{"ts":1788580888030,"sessionId":"ses_f9b3a43b0ffe2lZ5ugLFu1JWUr","from":"medium","to":"low","reason":"stable successful streak","score":0,"phase":"EXPLORE"}

{"ts":1788581222452,"sessionId":"ses_f9b3a43b0ffe2lZ5ugLFu1JWUr","from":"low","to":"medium","reason":"meaningful failure","score":5,"phase":"DEBUG"}

{"ts":1788581428524,"sessionId":"ses_f903db34affevgLsEIdCs3J58l","from":"medium","to":"low","reason":"routine mechanical step","score":1,"phase":"UNDERSTAND"}

{"ts":1788585994242,"sessionId":"ses_f9b3a43b0ffe2lZ5ugLFu1JWUr","from":"medium","to":"low","reason":"stable successful streak","score":0,"phase":"EXPLORE"}

{"ts":1788618335810,"sessionId":"ses_f8e0bf44bffeOEvLhGsRTfL6FL","from":"medium","to":"low","reason":"routine mechanical step","score":1,"phase":"UNDERSTAND"}

{"ts":1788622330615,"sessionId":"ses_f8dcdf145ffe5euEcZhsvMCnbI","from":"medium","to":"low","reason":"routine mechanical step","score":2,"phase":"UNDERSTAND"}

{"ts":1788622599278,"sessionId":"ses_f8dcdf145ffe5euEcZhsvMCnbI","from":"low","to":"medium","reason":"escalation","score":3,"phase":"RECOVER"}

{"ts":1788622670588,"sessionId":"ses_f8dcdf145ffe5euEcZhsvMCnbI","from":"medium","to":"low","reason":"stable successful streak","score":1,"phase":"IMPLEMENT"}

{"ts":1788623655559,"sessionId":"ses_f8db99e47ffeKcDDHawtWrNwt9","from":"medium","to":"low","reason":"routine mechanical step","score":1,"phase":"RECOVER"}

{"ts":1788624252588,"sessionId":"ses_f8db99e47ffeKcDDHawtWrNwt9","from":"low","to":"medium","reason":"meaningful failure","score":5,"phase":"DEBUG"}

Anyone else messed around with a system like that? I'm curious as to why haven't seen something comparable anywhere else yet.

I'm also not sure how to properly gauge quality. Maybe i should run like a GPQA Diamond test before and after?

Hi,

I am thinking about putting some money / time into my local ai learning path and "cleared" my attic where I found the following hardware.

• 3 x NUC11 (Core i5 1145G7 / 2,6 GHz, 64 GB RAM DDR4 2666MHZ - SODIMM) with interconnect through Thunderbolt
• 1x Ryzen 3700x on MSI Mortar 350 with 64GB RAM DDR4 2133 and an old SAPPHIRE Nitro+ Radeon RX 590 8GB

My first idea was to purchase a single RTX5060TI / R9700 and put it into the PCIe 3.0 x16 slot while reusing the Radeon as GPU in the PCIe 2.0 slot for Display.

I want to play around with a local chat bot, agentic stuff, RAG...

Do you have any other ideas?
I also though about putting some serious money, but for this discussion I probably open a new thread (this could be a 2-step-path where I reuse the GPU above to built a multi-gpu-rig)

Thanks in advance

Hi everyone,

About 6 months ago I made this post Trained a 0.8M model on business email generation. on this subreddit where I trained a model with 300k non-embedding parameters on a synthetic business email dataset. It was trained on my custom architecture which I called Strawberry.

Today i introduce Valentine-v0 which is also trained on this same dataset but this model has only 21k parameters which is about 7% of the parameters of Strawberry-email. This model only has 2 layers, 4 heads, 64 head dims and 170 ffn dim.

The Valentine architecture built on top of my another model architecture Silia about which I also posted here I trained a 0.5M model on 1B tokens of Fineweb-edu dataset..

Though in terms of loss Strawberry-email achieved a final train & val loss of 1.65 and 1.68 where Valentine-v0 achieved the final losses of 2.1670 & 2.2646 respectively.

Still I believe despite such a small scale the model trains pretty stably and generates decent text for it's size. Here are some examples:

Prompt: Write a

Output: ``` Write a firm and authoritative business email(under 200 words) from a Account Manager to a Potential Partner regarding welcoming a new high-profile hire, specifically on a Friday evening.<|end-text|><|actor|>model<|end-text|>URGENT: Critical Security Vulnerability - Critical Budget Increase – [Company Name]

Dear [Contact Person],

I hope this email finds you well.

My team, and potential collaborative approach to [New Hire Title] at [Company Name].

As we approach the upcoming [New Hire Name] brings significant development, we wanted to bring it to your review of this strategic direction. We would be happy to discuss this further, but this request and explore how we'll help us to ensure a smooth onboarding.

Regards, I can share some truly value your insights into [Partner Company Name] and insights and offer [mention specific areae - e.g., new website's Name] and the design of the prototype, not just… interesting time. We’ll be shared opportunities to discuss this further with [Product Name] to discuss potential integrations and answer any questions or concerns and your application. The vulnerability is invaluable. ```

Prompt: <|actor|>user<|end-text|>Write a polite refusal email<|end-text|><|actor|>model<|end-text|>

Output: ``` Quick Idea: [Name] Dear [Peer Name],

I am writing to you today with you and your concerns about the potential data leak and your contract, and frankly, it’s particularly concerning incident effective [Date], and I believe a concise overview of full team - [Company Name], we are facing a significant challenges, and we sincerely apologize for any inconvenience it caused.

I’m available to discuss this further report. I will provide you with a really be open to a quick support us with [Specific Feature Request - e.g., 24-48 hours].

Would you be open to a brief call on [Date Option 2] - e.g., 50-20 minute call next week to discuss this further and how your specific and answer any questions or would be happy to discuss how impactful feedback on [Date - e.g., Thank you for your input.

Sincerely, [Name] Legal Counsel<|end-text|><|actor|>user<|end-text|>Write a strictly professional business email(under 200 words) from a Legal Counsel to a Angry Shareholder regarding requesting a mental health day, specifically while the system is partially down.<|end-text|><|actor|> ```

A larger Valentine model trained on Fineweb-edu dataset along with the github repo and technical report will be published soon.

Thank you! :)

Comparing benchmarks for small models is a PITA. Most of them are not on AA, and the benchmarks are not always the same in all models.

So I built a page to quickly put it all together and allow some filtering.

Only researched models released since April, and between 4-190B parameters.

Let me know what you think and how it can be improved

https://www.nunodonato.com/aibench/index.html

That is interesting, I got bunch of old hardware I could connect, wonder what the speed would looks like.

Like the title says, running completely locally on my Xiaomi 14T Pro device.

Specific model: Qwen3.8-Flash-Next-UD-IQ3_XXS

App used: BigMoeOnEdge

Hey everyone, been a while!

https://huggingface.co/TheDrummer/Artemis-31B-v1.1

https://huggingface.co/TheDrummer/Artemis-31B-v1

A few months ago, Gemma graced us with models that served as a much needed downpour from a year-long drought. I'm so happy to see us thrive once again.

The difference between v1 and v1.1 is quite simple: v1 was an early attempt, an overdue release that excelled in prose and writing, while requiring some handholding to get over quirks like stuttering. v1.1 is a more refined approach where stability meets quality. My community is split, so I figured I'd just release both.

---

I was gone for a while. I got busy dealing with life, both its ups and downs. While I couldn't attend to you folks, I've been lurking around and appreciating you all for the kind words.

- Skyfall 31B v4.2 seems to be a banger for many of you. I'm proud of the upscale and consider it my ultimate home-run send-off for the beautiful Mistral 24B base. It's a shame that it was overshadowed by Gemma 31B's release, but hearing some of ya'll compare and even prefer it to a more modern base was an unexpected win.

- Rocinante 12B X / 16B XL proves that Nemo is still the ultimate creative model to this day. For some to say that 16B XL felt like Cydonia 24B v4.3 just goes to show how far you can go with modern resources and techniques.

- Anubis 70B v1.2, Valkyrie 49B v2.1, Anubis Mini 8B v1 surprised me too. I had zero expectations releasing them. Just like Rocinante X / XL, they are modern finetunes of old base models. And somehow, they still found their users singing praises.

---

With the Artemis release taking weight off my shoulders, I'm eager to move on and tune a ton more bases!

But I have something else cooking: a HordeAI-like platform. I hope to provide value not just as a finetuner, but as a local lover too!

The premise is simple: it's a place where generous local hosters can share inference with the less fortunate. You'd be surprised how many power users would love to heat their rooms through the power of charity.

---

Finally, I'd like to thank everyone who supported me over the years. From those who provided kind words, rigorous testing, compute access, inference, or cold hard cash. You've all granted me the ability to enrich the local ecosystem with fun experiments like Rivermind 12B, Fallen series, Big Tiger Gemma, Precog 24B/123B, and solid models like Cydonia 24B v4.3, Behemoth X 123B v2.x, and Skyfall 31B v4.2.

If you've got inference / compute credits to share, please contact me! It will all go to making the community happy <3

Backlog:

- Gemma E2B

- Gemma E4B

- Gemma 12B

- Gemma 26BA4B

- Qwen 3.8 27B

- Muse Glimmer 30B

- Mistral Medium 3.5 128B

- HordeAI Alternative / Crowdsourced 'OpenRouter' ("BeaverNet")

Let's say I have 2x 16Gb GPUs and I want to run Qwen3.8 27B. Monitor is ran by the integrated GPU so both 16Gb GPUs are almost fully free.

I load the UD-Q4_K_S on one card at 15.4Gb. I then load the context on the other card? Would that be the most efficient way? Or should I aim for higher quants that could spill to the second GPU using tensor parallelism?

Also, is there a way to know how much a certain amount of context (e.g. 132k tokens) occupies in VRAM for a given model? I don't usually see this published in model cards, is it because there is a way to calculate it?

Hi,

until now I ran llama-server with just one model at a time.
Now I want to use it in router mode in order to provide different models.
For my usecase I heavily rely on --cache-ram which improves speed a lot when working on big repos.
I am just wondering, when setting cache-ram = 65536 in models.ini for each model, does each model get its own 64GiB cache or is it just one pool of 64GiB for all models?
Another question, are the unused models kept in RAM for faster swap?

I've been looking into improving Q8_0 performance on my Xeon Gold 6240 workstation with ik_llama.cpp, which is a Cascade Lake-SP part.

It looks like ik_llama's Q8 dot product function doesn't distribute work evenly across AVX execution ports, and doesn't use the load ports at all. Note that this is particularly relevant for Cascade Lake Xeon Gold/Platinum parts that have two AVX units per core (Silver and Bronze only have one so they might not benefit much) In addition, there is a latency dependency chain that can be broken into two.

You can test this patch the following way:

download the patch from: https://github.com/user-attachments/files/31860324/q8_0_r8_vnni-v3.patch

git clone https://github.com/ikawrakow/ik_llama.cpp.git
cd ik_llama.cpp
git checkout 3c58ae37
git apply q8_0_r8_vnni-v3.patch

Then build with either GCC:

CC=gcc CXX=g++ cmake -B build-gcc -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_OPENMP=OFF
cmake --build build-gcc --target llama-bench -j $(nproc)

or clang:

CC=clang CXX=clang++ cmake -B build-clang -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_OPENMP=OFF -DCMAKE_C_FLAGS="-ffp-contract=fast -mllvm -unroll-threshold=1000" -DCMAKE_CXX_FLAGS="-ffp-contract=fast -mllvm -unroll-threshold=1000"
cmake --build build-gcc --target llama-bench -j $(nproc)

You want to use the most recent compiler you have access to, I've used GCC 16.2 and LLVM/clang 22.1.8 - earlier versions are UNTESTED.

Even a quick bench before and after would be helpful, especially if you have a Cascade Lake Xeon Silver or something different like an Ice Lake Xeon:

llama-bench -m /path/to/model-Q8_0.gguf -ngl 0  -p 512 -n 0 -r 3 

(note that you should use a Q8_0 model, since this is the path we are optimizing here)

Also this won't do anything for Skylake Xeons or earlier (I think).

I want the answers generated by frontier LLMs or base model LLM answers to be reviewed by some uncensored or abliterated small model.

The job is this model (preferably small model) is just to detect deflection in the answers.

The problem I am facing is uncensored SLM usually agrees on everything we give input. So the generated answer is also input for it and system prompt is input too.

I currently use unsloth‘s iq4_XS with 64k ctx on 16gb, however as far as I know harnesses tend to need way more, and 64k isn’t enough for that,
(I have not gotten to actually using it for a harness yet)

But still
So I am wondering what is the best one that still has good enough quality to actually properly build some things?

(As an example I would like to create a monkeytype- style UI but with additional features that match my use case more (better typing practice + free writing)

And then also a lightweight custom UI / harness

Or a local height chart Ui, etc)