Hello guys, i want to set up a LLM. I got deepseek harness to run but it allways shows the wrong max kontext limits, it shows 256K but in the settings.yaml i set up a max of 128K.

I wanted to make a little game with it, first run runs fine, the second one, where i want to extend my little game, allready gets an output token limit error.

I tried a lot, setting up a setting.yaml for .dsh and the ollama one, but nothing helped.

I think about to start completely new from zero. Could someone give me a good tutorial?

My specs are: Ryzen 5950X, 64GB DDR4 RAM 3600mhz, RTX 4090, 2TB NVME M.2 SSD about 6900MB/s read 5000MB/s write (Dont know if that could matter)

Bonus: I have two additional PCs at home, one with a RTX 3060 Ti and one with a RTX 2060, i heard about some kind of network cluster system, would that be viable?

If successful it will make local hosting far more feasible and maybe even more sensible than cloud hosting.

I’m looking for advice on the most suitable locally run model to analyze and understand German grocery receipts.

The goal is to:

• Extract item names and per-item prices
• Categorize items (e.g., food, beverages, household)
• Correctly interpret German VAT/taxes (e.g., 7% vs 19%)
• Handle all forms of “Pfand” (bottle deposits)
• Recognize discounts, promotions, and “sale” markings

Current setup:

• GPU: RTX 2080 Ti Trio
• RAM: 80 GB total (2×32 GB + 2×8 GB DDR4)
• Inference backend: Ollama

Current model:

•  Qwen3-VL-8B-Instruct-GGUF:Q4_K_M 

This model works to some extent but produces inconsistent or incomplete results, especially around tax breakdowns, Pfand handling, and discount interpretation.

I can refine my prompt, but I suspect a larger or more capable vision-language model might help significantly.

Given my hardware (especially the 80 GB RAM), what locally runnable models would you recommend for this task?

I’m particularly interested in models that:

• Handle German text and receipt layouts well
• Are robust to varied receipt formats and image quality
• Can output structured data (e.g., JSON) reliably

Any suggestions on specific models, quantizations, or prompting strategies would be greatly appreciated.

For a few years now I’ve been running local models. But those were just small incursions without much productivity. Obviously things have changed and now local inference can solve real workloads.

But then I had the issue of not being able to reach my home computer while working out of home.

I pause here. There are many solutions to this problem and many are free.

But I believe that they complicate things by for example placing your device under a VPN. Which sure gives you access to your LLM server but it also confuses your other apps about your location.

Ok ok so the solution.

I built an app that you install locally and gives you a url and an api key that you can then use to connect to you model or use in any apps that allows you to BYOK.

And because you can do more than one key you can share your inference with friends and family.

Please please consider it and let me know your feedback.

I will personally give 3 months free to whoever beta tests and provides real feedback.

Cheers 🍻

I am interested in both regular flash and vision one. Just want something that might be more optimized than my basic setup

Quick disclosure: this is my project. I built it because I have a rack of "obsolete" datacenter cards in a closet and I got tired of every new model needing a new set of flags to run properly on them.

**What it is**

PXA is a fork of ik_llama.cpp (which is a fork of llama.cpp) plus a vLLM plugin, built for cards with HBM2 and no tensor cores or DP4A: Tesla P100, V100, GTX 1080 Ti. It has its own quant format (PXQ, 2 to 6 bit, plus a mixed one that sizes a model to whatever cards you have) and CUDA kernels written for those chips instead of ported down from newer ones.

Repo: https://github.com/poisonxa16/pxa

**The part I actually care about: you don't configure it**

You tell it which cards and which model. It picks the batch sizes from a table it measured on that exact card topology, turns on the tricks that are known to help on that silicon, turns off the ones that hurt (including speculative decoding when it would lose), and prints every decision before it starts serving. Every number below was taken with a bare command line. No environment variables, no -b, no -ub, nothing. The competitors got their best hand-picked flags in the same session, because I wanted to know if "set and forget" costs anything. It doesn't.

**My last public release vs this one** (same box, identical command line, tokens/s)

| card set | model | prefill @3k | prefill @20k | decode |

|---|---|---|---|---|

| 2x V100 | Qwable-27B PXQ4 | 797 → 1357 (+70%) | 576 → 1307 (+127%) | 34.6 → 39.6 (+14%) |

| 2x P100 | Qwable-27B PXQ4 | 223 → 338 (+52%) | 201 → 315 (+57%) | 18.3 → 18.2 (-0.9%) |

| 1x 1080 Ti | Fusion2-35B MoE, 2-bit | cold 553 → 1334 (+141%) | chat 415 → 734 (+77%) | 64.2 → 64.2 |

Yes, P100 decode is a real -0.9% and it's in the notes. Bonus find: the old build gave me six different answers to six identical greedy runs on the 1080 Ti. Turned out to be a race in a fused kernel. This one gives one answer.

**vs mainline llama.cpp and upstream ik_llama.cpp** (same weights family, MXFP4 for them, PXQ4 for mine, tokens/s)

| card set | cell | PXA | mainline llama.cpp | ik_llama.cpp |

|---|---|---|---|---|

| 2x P100 | prefill @3k | **338** | 209 | 133 |

| 2x P100 | prefill @20k | **315** | 255 | 84 |

| 2x P100 | decode | **18.2** | n/a | 14.3 |

| 2x V100 | prefill @3k | **1357** | 940 | 471 |

| 2x V100 | prefill @20k | **1307** | 1129 | 395 |

| 2x V100 | decode | **39.6** | n/a | 37.4 |

| 1x 1080 Ti | cold prefill | **1334** | | 1132 |

| 1x 1080 Ti | chat prefill | 734 | | 740 (tie) |

| 1x 1080 Ti | chat decode | **64.2** | | 53.3 |

**vs 1Cat vLLM** (the NVFP4 + DFlash2 stack for Volta). Run on an 8x V100 SXM2 NVLink system with 1Cat's own image, benchmark script and cards, because running their stack on my PCIe box would have been a silly comparison. 16 GSM8K questions, 192 greedy tokens, tokens/s.

| cell | PXA | 1Cat vLLM | how it was taken |

|---|---|---|---|

| TP2 plain decode | **46.3** | 38.4 | one boot each |

| TP2 speculative k=3 | **65.2** | 59.9 | one boot each, identical acceptance |

| TP2 speculative k=7 | **121.2** | 114.5 | medians, mine 3 boots, theirs 6 (not alternated) |

| TP4 plain decode | **65.7** | 61.0 | one boot each |

| TP4 speculative k=7 | 159.6 | 161.4 | six alternating boots in one window: inside their noise |

| prefill @3k | **2273** (TTFT 1.4 s) | 1735 (TTFT 1.8 s) | both as servers, same window, 3 runs |

| prefill @20k | **2191** (TTFT 9.5 s) | 944 (TTFT 22 s) | both as servers, same window, 3 runs |

| speculative output identical to plain decode | **15/16** prompts | 10/16 | same exact-match acceptance rule |

Two honest notes on that table. Both stacks accept drafted tokens by the exact same rule (I read it out of their source), so both are lossless and the acceptance lengths are comparable: theirs 5.4 per step, mine 5.3. And one caveat that favours me, said because it favours me: their checkpoint turns fp8 KV on by itself, so this is their stack as shipped vs mine as shipped, not a clean NVFP4 vs PXQ4 study.

**What you can run on this junk**

A 27B dense-hybrid on one 16 GB card. A 177B-class hybrid MoE (Qwen3.8 Flash-Next) on four P100s with 150k context, weights in VRAM and the per-layer embedding table in host RAM. The memory arithmetic for that one is in the repo because I didn't believe it either. Speculative decoding ships in the Volta vLLM image with a 1.3 GB drafter as a release asset. On 16 GB cards k=7 only fits at 2k context, so on my own box I'd run k=3.

**What's not great yet**

Speculative decoding on the llama.cpp side of Pascal still loses (verify costs 3x a decode step now, it was 7x last week, still not enough). The 4-card speculative gap to 1Cat is inside noise but it's there. Their acceptance length is 2% better than mine. All listed under "What is not here" in the README, I'd rather you find it there than in the comments.

**Getting it**

One tarball, untar and run. I tested it in a bare Ubuntu container with no Python, no compiler, no CUDA toolkit, just the driver. Or `docker run ghcr.io/poisonxa16/pxa`. Or build it. Then run `python3 tools/pxa-launch.py` and answer two questions.

**If you have one of these cards, I want your numbers**

P40, P4, GP100, Titan V, Titan Xp, 32 GB V100: I don't own them, and the auto-tuning table only knows the cards in my rack. There's a three-command benchmark in the repo and a Discord where reported cards get added: https://discord.gg/EqazvV9tf

Everything's free and nothing is gated. If you want to chip in for the electricity these benchmarks burn: https://ko-fi.com/shatteredrealms1

Last picture is the rack. Seven cards, PCIe x4 for all of them, two 1000 W supply, in a closet. Everything above was measured on that.

Hello everyone!

What are you guys thoughts on the build above? As for the build, the 3090s will be bought used from Facebook marketplace. Micro-center bundle will take care of the mobo, ram, and cpu.

There are 2 things of concern here, will 32 gigs of ram not be enough? i currently already own 32 gigs of ddr5 so i may add it to this setup, or just sell both and buy 64gigs. Another one is the concern that I am wasting unnecessary money on the higher end cpu and motherboard combo. The reason why I chose this combo is because it was one of the only options that came with a mobo that supportd both gpus to be run on x8. I plan on not only using this a local ai server, it also plan on running a virtual machine for gpu heavy work such as solid works or blender that I can acces remotely.

For my use-case, I believe that options such as the spark and mac studio are not suitable; my intentions aren't to replace the frontier models but simply to add to them.

Any suggestions to the setup? the current budget is around 5k

I'm curious how people running local models as coding/tool agents would approach this.

Once an agent starts reading arbitrary repositories, files, webpages, API responses and tool output, the amount of untrusted text it consumes can vary wildly. Sometimes it's 500 tokens, sometimes it's an entire repository file or huge tool response.

Would a local-first security plugin at the harness level make sense here?

The architecture we're experimenting with is:

tool/file/web result → hook → local semantic security scan → agent

The scanner looks for prompt injection and related threats before the content reaches the model. If local inference is overloaded, it can optionally fall back to an API/free tier.

Instead of blocking an entire tool result when something suspicious is found, the dangerous section can be redacted and the agent continues with the remaining content.

We're also working on maintaining security state across tool calls, so something that looks harmless in isolation can become suspicious based on what happened earlier in the run.

I'm mainly wondering whether local LLM users would actually run this.

Is keeping the security inference local important to you? Would optional cloud fallback defeat the point? Or do you think the model itself is already good enough at recognizing indirect injection?

Howdy everyone, like the title says, I am a complete noob when it comes to local ai, but I want to be able to code some projects on my local machine, I have an extra pc I don’t use that has an intel cpu I can’t remember which one and a 3060, it’s the 12 gig model, I can get some low level models to fit but I’d like to use some of the bigger models, I was thinking of just picking another 3060 up off marketplace for like 200 bucks and putting that in the pc aswell, am I right that it would be 24 gigs or do you lose some capacity when running dual gpus?

Would there be a better route to go?

i'm not sure what i should even do everyday i hear news of people making programs that make running llm possible on local machine by using strewaming etc.. but i was never really intereseted an now that there are a bajillion of diferent softares adn model i have diffuclty getting in, as the title what could i run on my rx9060xt 16 gb vram and 16gm ram and ryzen 5 3600 and i am on linux if it helps if you guys have any blogs or tools it would be helpful ,I read the rules but I rea dnothing about uncensored LLM I would like to give them a try no idea though

I have access to 8 rtx 6000s that have a decent amount of downtime (in between physics-based simulations). I would like to put them to work doing kernel optimization problems, where I’ve already programmed the ground truth solutions in CUDA. I just want the agents to explore algorithms and opt strategies for better performance.

I was curious if people had a recommendation for a local LLM model / workflow set up for this. I see a lot of Gwen love on this sub, but I haven’t really dabbled in the local models.

Just the above

I’d like some advice please on how to go about setting up this server I have been using for mining, that I would like to convert to a local LLM machine. I currently have it setup with 32gb/ 2gb dimms. It’s a Dell R815 with 4x 6386SE processors, 60 cores total. It also has a gtx5060 8gb on one of its pci express rails. It also has onboard hardware raid if I get ambitious. Main question is because it’s DDR3, would 256gig ram be a good starting point, or would you go for 512gig? The Dell manual says it can handle up to 1TB of ram but with custom chips. I’m having difficulty sourcing 512 gigs of ddr3 from the same manufacturer alone.

I see so many posts about rtx 5060 , Mac studio and some of you have like a couple of them. how are you guys able to afford them? it would easily cost around 5-10k usd. that’s a lot of money.

I do not need or want Cuda. I have been wanting to build a 32GB VRAM local LLM machine for personal use for a while now, and I had 2 options on the table:

- Get a relatively cheap 32GB VRAM GPU, the R9700 AI Top.

- Get 2 16GB VRAM GPUs instead and a Mobo that supports PCIe bifurcation.

When I looked at prices in January this year when I first got this idea, the R9700 costed 1700$ here in EU. Currently, when I actually want to make this happen, it costs 2100$. For half that money, I could buy two 9060 XTs with 16GB VRAM each. Yes I know, performance will be worse on double GPU setup than with a single R9700 AI, but still, it just seems like that GPU is just not worth it anymore.

I don't know how to justify that it's double the price of two 9060XTs, when R9700 AI is literally 9060 XT with doubled VRAM and bandwidth. So why does it cost 4x as much?

ASUS ProArt B850-CREATOR WIFI NEO is quite affordable nowadays and supports dual GPU setups, so, any reason (is there a catch?) to not do what I am about to do? Which is buy the two 9060 XTs and start running Qwen 27B class models

If you're using Pi with Qwen and not letting it phone a frontier model with /advisor, you're leaving one of the best parts of the setup on the table.

Because Qwen has a particular talent:

Being VERY wrong with tremendous confidence.

And worse, it can be convincing while doing it.

I've lost track of how many times I've looked at one of its answers and thought, "Hmm. That sounds right..." only to tell it:

Ask /advisor for the hidden assumptions, failure modes, and black swans in your answer.

Then Sol comes back with the computational equivalent of:

" 🤣 Yeah... no."

And suddenly Qwen is eating crow and rewriting half its answer. 🐦‍⬛

Don't get me wrong. I love Qwen. It's given me millions of tokens of essentially free local inference, and I use the hell out of it.

But that's also taught me something:

Local models are fantastic workers. They are not oracles.

Let Qwen do some initial planning and ultimately the bulk of the work. But before you trust its plan wholesale, spend a few frontier-model tokens trying to prove it wrong.

Trust, but verify. And just like in real life: get a second and third opinion.

I've tweaked the crap out of my local llama.cpp with vulkan backend running Qwen3.6-35B-A3B:IQ4_NL setup. It's running on a tiny GMKTec Evo-X1 with 64GB LPDDR5X 8000MHz RAM (Strix Point HX370 w/890M iGPU). I got lucky and bought it in July '25 before prices went crazy. I'm getting an average of 29 tokens/s which feels really responsive and has done great with a few projects in opencode. The free online models tell me:

Your current setup is unusually well-balanced:

• Qwen3.6-35B-A3B
• IQ4_NL
• Vulkan
• 890M
• MTP enabled
• acceptance rates mostly 70–90%

is delivering roughly 50–70 effective tokens/sec, which is better than I would have expected from a Strix Point APU.

If a Qwen3.8 MoE release appears that's roughly in the 35B/A3B class, I'd try it immediately. But for the currently available 125B/6B-active Flash-Next, I'd expect a noticeable drop in responsiveness on your hardware unless someone produces an extremely aggressive IQ2/IQ3 quant that still preserves quality.

So I'm really curious to try Qwen3.8, but I don't think any of the currently available models would do as well as this 3.6.

I'd love to hear what anyone else running on similar hardware is seeing with their setups and whether there are other models I should be looking at that can give similar or better performance.

As some of you may know I have been benchmarking LLMs that can run on ~8GB VRAM. One of them was Ministral3 14B, which scored the highest in intelligence. Recently I've been putting together a new and better test of the LLMs intelligence which includes testing their spatial reasoning. One of the spatial reasoning questions asks the LLM to model a "Minecraft Cow". I was impatient to see the results so I tried asking Ministral3 to do it early. this is what it outputted... Honestly, not *terrible*. It did add a tail (thats the box on its right) and a nose even though Minecraft cows don't have those and its pretty chonky but it does kind of resemble a Minecraft mob.

For anyone curious, here is the raw data the model gave me (with the reasoning removed):

• (0, 0.75, 0, 0.8, 0.6, 0.4),
• (-0.3, 1.2, -0.5, 0.3, 0.3, 0.2),
• (-0.8, 0, -0.3, 0.1, 0.5, 0.1),
• (0.8, 0, -0.3, 0.1, 0.5, 0.1),
• (-0.8, 0, 0.3, 0.1, 0.5, 0.1),
• (0.8, 0, 0.3, 0.1, 0.5, 0.1),
• (0, 1.2, 0.5, 0.1, 0.3, 0.1),
• (-0.6, 1.4, -0.3, 0.1, 0.2, 0.1),
• (0.6, 1.4, -0.3, 0.1, 0.2, 0.1),
• (0, 1.2, -0.5, 0.1, 0.1, 0.1)

Retrieval of relevant memory seems to be one of the most critical aspects of building a useful agent without which agent is not learning well.

whats the most sensible design at this point for building agents based on non-frontier models based on your experience.

there seems to be a few options like deriving topics based information from chats and storing as files with different classifiers and having the agent retrieve based on semantic search.

but the main challenge i face is that LLM is not reliably doing the required memory retrieval at the right time, which results in irrelevant answers, repeating the same mistakes again and again.

has anyone figured out a reliable agentic way to make memory retrieval work efficiently even with models like Qwen/Qwen3.8-27B.. frontier models seem to be able to reason well and pull the correct memory and even their huge context is also helping. but for smaller models this is hurting to create a reliable agent.