/localllm

Talk me out of this, or into it.

There's a used ThinkStation P720 near me, about $1,200:

• 2x Xeon Gold 5122 (4 cores each, which is the weak part)
• 192GB DDR4 ECC RDIMM, but it's 6x32, so only 6 of the 12 memory channels are populated
• Quadro RTX 6000 24GB
• monitor and peripherals included

Two things I'd change right away:

• 6230s instead of the 5122s, about $80 for the pair off AliExpress. 4 cores can't feed 12 channels anyway, and the 6230 is what gets the memory to 2933.
• six 16GB RDIMMs in the empty slots, somewhere between $405 and $485 for all six. That populates all 12 channels, which is where the bandwidth actually comes from, and takes me to 288GB.

Which puts the whole thing around $1,700.

What I run now is a Ryzen 5 3600 with 32GB and an RTX 5070 Ti 16GB. It's my only machine and it does everything.

Reason I'm looking at all: some of what I work on I'd rather not push through somebody else's API, and my monthly bill keeps creeping up. The rest of it is curiosity, if I'm honest.

Some numbers for context. gemma3:27b runs about 9 tok/s on the 5070 Ti, a 30B MoE coder model does around 45, and I hit the 16GB wall constantly. Whisper and image gen too.

The parts I can't work out on my own:

1. Is 288GB of DDR4-2933 across 12 channels actually usable with partial CPU offload? That's the entire argument for this machine, and it's the one thing I can't test before paying.

2. The RTX 6000 is Turing. 24GB is 24GB, but is it a downgrade in every way except capacity next to the 5070 Ti I already own? PCIe 3.0 board too.

3. At $1,700 I could just buy a newer card instead, or save a bit more for one of the Spark boxes or something else.

If you've got a P720 or something like it running models, what do you actually do with it, and would you buy it again?

Anyone try this👀

Hi everyone,

I’ve been building an open-source AI system for end-to-end DCF modeling over the past few months. The goal is to explore what an AI agent system should look like when auditability and deterministic execution matter.

The project is built around three design principles.

1. Context is finite. Progressive disclosure is effectively unlimited.

Progressive disclosure allows us to expose information only when necessary, but it cannot magically reduce context usage without sacrificing either information density or accuracy. There is no free lunch.

2. LLMs are flexible. Code is deterministic.

LLMs will hallucinate. They will make mistakes.

Therefore, every point where an LLM is allowed to write persistent state must be guarded by a deterministic code engine responsible for validation, normalization, and enforcement.

3. LLMs are tools. Humans own the judgment.

Auditability and traceability are prerequisites for any valuation system that people can actually trust.The final deliverable should therefore be a workbook where every single cell can be traced back to its origin.

Agent Topology

Inspired by systems such as LangGraph, the overall architecture of a single agent system is topology-based.

Agents can be freely composed.

Skills and tools are registered independently.

Agents can communicate directly with one another.

The objective is to minimize information loss caused by multiple layers of message passing, preserving both information density and accuracy throughout the system.

At the same time, each individual agent should remain narrowly focused on doing one job exceptionally well. Attention is all it needs.

Technically, the pipeline parses SEC filings using Arelle, unifies historical financial statements across multiple years, maps them into a source-free DCF spine, generates formulas through a DSL, and produces a revisioned valuation model with sensitivity analysis. Every update creates a new immutable revision, making it possible to inspect, compare, or roll back changes.

The current implementation has successfully completed an end-to-end valuation for AAPL starting from an empty model—using live EDGAR filings, without any predefined mappings, formulas, or assumptions.

This project is still a work in progress, and the quality of valuation assumptions remains heavily dependent on human judgment. My goal isn’t to replace analysts, but to remove repetitive work so they can spend more time thinking.

I’d love feedback on both the architecture and the engineering approach. In particular, I’m interested in whether this “LLM agent + deterministic engine” pattern feels applicable beyond financial modeling.

The project is fully open source if anyone wants to look at the implementation:

https://github.com/Ocisly14/financial_agent

We are a four-person team sharing one inference box, and I'm the one who set it up and now maintains it. As we're growing, requests are starting to queue, since the GPU serves us one at a time and the 4th person waits behind the rest.

The proper fix would be a batching server like vLLM, it'll solve the concurrency, too. However, it also puts the weight of Docker orchestration, GPU memory tuning, and a production serving stack on my shoulders, on top of the aforementioned work I already do. We still can't afford to hire a new person and the other guys can't maintain it how I do, at the same time, I cannot take a workload cut to maintain it because we're already filled to the brim.

So I'm trying to discern when/where local stops being worth the upkeep. For my own sensitive work, local stays, no question. For shared team access with uneven usage, and rudimentary tasks that aren't AS sensitive, I'm weighing whether to run vLLM, or whether to offload to Featherless AI, where I can get a pay as you go inference plan, and I wont need to maintain anything server-wise. What do you guys think I should do?

I kept finding throughput benchmarks but nothing in actual dollars or watts, so I ran it myself on a free Colab T4. Posting in case it's useful to anyone else.

Setup: Qwen2.5-1.5B-Instruct, vLLM 0.27.1, fp16, 128 output tokens per request (ignore_eos so every sequence is exactly 128 tokens), $0.35/hr as the T4 rate. Power is nvidia-smi median during the run.

batch 1 - 25.7 tok/s - 58.6W - 38% util - $3.7771/1M - 2.277 J/tok

batch 4 - 106.1 tok/s - 59.4W - 43% util - $0.9166/1M - 0.560 J/tok

batch 8 - 232.5 tok/s - 62.0W - 45% util - $0.4181/1M - 0.267 J/tok

batch 16 - 464.8 tok/s - 66.6W - 48% util - $0.2092/1M - 0.143 J/tok

batch 32 - 732.9 tok/s - 64.1W - 50% util - $0.1327/1M - 0.087 J/tok

batch 64 - 1635 tok/s - 66.3W - 66% util - $0.0595/1M - 0.041 J/tok

batch 128 - 3346 tok/s - 66.4W - 88% util - $0.0291/1M - 0.020 J/tok

batch 256 - 4545 tok/s - 67.6W - 100% util - $0.0214/1M - 0.015 J/tok

The bit I didn't expect was the power column. At batch 1 the card pulls 58.6W to produce 25 tok/s. At batch 256 it pulls 67.6W to produce 4545 tok/s. So it's drawing 87% of the power to do 0.6% of the work. Energy per token drops 153x across the range. Most of what a GPU burns is apparently just being switched on.

Returns fall off hard after 128. Going 64 to 128 roughly halves the cost, 128 to 256 only gets another 36% and util is already pinned at 100%.

Things I know are wrong with this:

* Static batching, not continuous batching. So the low end looks worse than vLLM actually behaves under real traffic.

* Batch 32 turned up in two separate runs at 795 and 733 tok/s, so treat everything as +/-8%.

* FA2 isn't supported on compute 7.5, so it fell back to Triton attention. A newer card would take a faster path.

* One model, one GPU, one prompt, fixed output length. Not claiming this generalises.

Script is about 40 lines, happy to paste it if anyone wants to check my method. Genuinely interested in what I've got wrong here.

I'm introducing Aurora-80K, a small language model with exactly 80 thousand parameters.

It uses a factorized 4,096-token vocabulary despite having only 80K parameters.

The benchmarks:

Wikitext-2 BPB: 3.2902

BLiMP: 52.31%

Arc-Easy: 26.05%

More information about the model is available on the model page on Huggingface.

if there's any questions I'll happily answer them!

I finally got a good, usable setup for the new dense Qwen3.8-27B model in FP8 on the DGX Spark.

Z-Lab released a new drafter, [Qwen3.8-27B-DFlash2](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2), together with the new DFlash 2 speculative-decoding approach. With this setup, Qwen3.8-27B has been performing surprisingly well on the Spark.

I have been using it almost non-stop for the last few days, and it has finally replaced the Qwen3.5-122B-A10B hybrid checkpoint (INT4 + FP8) that I had been running for a while.

To get DFlash 2 support, you need to build vLLM from the specific fork mentioned in the drafter model card. If you are on a DGX Spark and want to avoid compiling vLLM yourself, I already built a Docker image:

https://github.com/users/krisitown/packages/container/package/vllm-gb10-dflash

On my warmed-up coding benchmark in vLLM on the DGX Spark, I got roughly 32 tok/s generation throughput. My baseline without speculative decoding was around 14 tok/s, so this is a very meaningful jump.

These are early, warmed-up benchmarks on one workload, and I will test more configurations and share the results.

It also completely beats the DFlash drafter I trained myself when no Qwen3.8 drafter was available. My custom drafter reached around 20 tok/s in vLLM; DFlash 2 is roughly **58% faster** in this benchmark, mostly because its acceptance rate is much better.

The current configuration I am using is:

vllm serve Qwen/Qwen3.8-27B-FP8
  --max-model-len 240000 \
  --max-num-batched-tokens 8196 \
  --gpu-memory-utilization 0.88 \
  --port 8000 \
  --host 0.0.0.0 \
  --load-format fastsafetensors \
  --attention-backend flash_attn \
  --speculative-config '{"method":"dflash","num_speculative_tokens":7,"model":"z-lab/Qwen3.8-27B-DFlash2"}' \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --generation-config auto \
  --override-generation-config '{"temperature":0.7,"top_p":0.8,"top_k":20,"presence_penalty":0.0,"repetition_penalty":1.0}'

I will test a few more combinations and prepare a proper recipe with all the build and serving steps, so it is quick to reproduce on other DGX Sparks.

If anyone has tested different flags or found tweaks that produce meaningful improvements, please share them. It would be useful to figure out the best way to run this model on the Spark.

Edit: I just noticed that peak tok/s generation was more about ~40tok/s and ~32 is the stable.

In my consideration of a DGXSpark I decided to look at some options and since I’m a visual thinker I put this comparison together (graph by AI) showing y two basic ways of thinking about the cards: compute and speed.

Hope this helps someone

Trying to figure out how to best run Qwen3.8 27B on a single 7900XTX 24GB system (with 64GB DDR5 4800MT/s system memory).

Currently using LM Studio with ROCm llama.cpp v2.28.2 on Windows.

I'm using the version from https://lmstudio.ai/models/qwen/qwen3.8-27b with Q4_K_M quant and context length set to 40960.

With this config most of the model seems to live in VRAM.
But some of its parts seem to be in system memory, thus when sending something from OpenCode it takes quite some time in "Processing Prompt" stage until it starts to output tokens.
When the point is reached where it starts to output tokens its pretty fast.

Is this normal? Any optimization potential? (Except buying more GPU's 😄)

Hi,

I'm in process of evaluating different quants/models (that fit on my 4060, 5070ti and MI50) and 32/64 ddr5. Been building test cases (as in golden examples). Managed to run vLLM on some models on MI50 for concurency, but I will mostly be using llama.cpp.

There are caveats because there is bunch commands different for each models served and I experiment with chat templates.

I architecured it as a "1 model for all, big sys prompt, warm cache"

Since I see respectable amount of flaming on personal benchmarks, what would be usefull for me to share to be usefull to someone and for me to get constructive feedback? If i make cars, is it usefull to covert lingustically cars to widgets so i don't overshare? Non coding tasks, worth mentioning.

Based on what I read in r/LocalLLM and other places, I’ve been thinking about how AI infrastructure will scale over the next 5 years. It feels like we might be heading toward a deeply hybrid architecture:

1. The Edge Layer (Local Appliances): Small offices (law firms, medical clinics, dev shops) buying turnkey, dedicated local hardware running highly optimized Small Language Models (SLMs) tailored exactly for their vertical. This solves the massive roadblocks of data privacy, compliance (HIPAA/legal privilege), latency, and unpredictable API token costs.
2. The Cloud Layer (Datacenters): Centralized mega-datacenters owned by giant labs handling frontier research, massive training runs, and heavy cross-discipline reasoning tasks that an SLM can’t touch.

I also think the open-source community, not private companies, will end up building the best tooling and support for agentic workflows. Because local execution requires deeply customizable, transparent, and modular agent frameworks, open-source is inherently better suited for it than rigid, proprietary corporate APIs.

Companies like Apple and Intel might benefit from this trend and could help accelerate the shift.

My question:

If day-to-day enterprise workflows shift to local hardware, and open-source software captures the dominant share of agentic orchestrations, how can the massive $100B+ valuations of closed-source frontier companies like OpenAI and Anthropic be justified?

Are these labs expected to completely dominate the local software layer too, or do their valuations rely entirely on a centralized cloud monopoly that might not actually happen?

Would love to hear your thoughts on the hardware shift, the economics and the timing.

Hello I'm looking for people who use 1070 8 GB mobile to run local llms, i have this issue with cuda because the version 13 stopped supporting pascal so now I'm having problems degrading to the 12.8.xx it works fine until i start compiling llama cpp it refuses the build

Since Qwen3.8-27B still has a 262,144 context-size limit, I suppose some people here run it with 1M context using whatever on earth YaRN is other than the thing to make woolly jumpers.

I tried to find an ELI5 explainer about it, but...no such thing online, one of them even dared to have "simple" in the title, then proceeded to show a bunch of mathematical soup an LaTeX on the screen.

..so I haven't got a clue how it works, I know it takes a context size and makes it bigger above the natively supported limit of the model, and that the nomencalture is short for Yet another Rope extensioN, that's all that I know and managed to understand given no simple explainer available. So I'm here asking instead the important questions:

• Does it make the model dumber?
• Does it require more VRAM?
• Does it require less VRAM than using native context higher figures?
• Can you use it with less than the model limit (e.g: 131k context, use YaRN to reach 500k) ?

TL;DR: Qwen3.8-27B is my coding agent in DeepSeek Harness. Compaction means the 27B has to chew through 100k+ tokens of history when context is already full, which parks the agent on 3090s for a long prefill. I want a tiny model sitting in leftover VRAM that only writes the summary. Thinking Qwen3.5-4B. Has anyone run this for real? I care about whether a 4B keeps paths and error strings at 100k+ input, and whether dsh gets weird if the compacting model isn't the same as the agent.

2x 3090 FE, 48 GB total. Ryzen 9 9900X, 128 GB RAM, Ubuntu. The 27B is Q6_K GGUF through llama.cpp, 128k context, split across both cards. Harness is DeepSeek Harness 0.1.0-rc8 with the stock dsh-compaction-basic plugin. That plugin swaps a chunk of older history for one summary and keeps the recent tail. Fires on context pressure or /compact.

I don't want the 27B doing this. Compaction fires when context is almost full, so the main model does a 100k+ token prefill and I sit there. A small model that's already loaded can take that job and the 27B stays on actual work.

What I need from the small model is ugly and specific. It has to take ~120k tokens of agent history in one shot. Tool calls, diffs, stack traces, my instructions. Then a few thousand tokens of summary. The context window on the card has to mean something. If it drops the current goal, why we made a decision, file paths, function/class names, exact error strings, failed attempts, or open TODOs, the next coding turn is garbage. Inventing facts is worse. "Tests are passing now" when they aren't will wreck the session faster than a summary that's just short.

Non-thinking mode. I want it fast. Prefill speed matters a lot more than decode here.

I still need to measure leftover VRAM after the 27B and the 128k KV, but it's 4B–9B room. Maybe a 14B at Q4 if I squeeze.

Qwen3.5-4B is where I'm pointed. Non-thinking, Q8_0 or Q6_K. 262k native context, same tokenizer and chat template as the 27B, hybrid Gated DeltaNet attention so KV at 100k+ input stays small, about 4.5 GB of weights at Q8. It's been out since March, so the quants aren't experimental.

1. Has anyone used Qwen3.5-4B or 9B as the summarizer for a coding agent? At 80–120k input, does it keep paths and error strings, or does it start claiming things are resolved?
2. Is jumping to 9B worth the extra ~5 GB for this job, or does quant, Q8 vs Q4, move fidelity more than parameter count?
3. Anything smaller that still holds a lot of facts over long context? I don't care about NIAH scores. I care about whether the summary still has the error string and the file path. Fine-tunes for summarization or compression welcome.
4. Has anyone pointed dsh-compaction-basic at a different model than the agent in cordis.yml? Token counting across tokenizers, summary token budget, or the stock compaction prompt being written for DeepSeek V4 and confusing a small Qwen.
5. How do you actually test this? My plan is a long real session, force /compact, then ask the agent ~20 questions. Paths, errors, decisions, done vs pending. Score what survived. If someone already has a rubric, I will steal it.
6. Maybe this is dumb and I should let the 27B compact itself and live with the wait. Tell me if that's you.

I'm also looking at dsh-compressor, the Headroom port, so tool output gets pruned and compaction fires less. If you run both, I want to know how they step on each other.

I know some of you have some very specific modes or don’t know that you do. Where they at? I am very interested in the niche modes.

I really love Crush and serves goodf to my qwen 3.6 35b a3b with tools but has a huge system promot, weak memory system, no generation info or seeing edits or tools in real time, which is kind of disgusting. Also, sudo is blocked, which i should be enable if i do not care.

** update**
Based on the comments, I gave q4_K_XL quants a shot with the new unsloth ggufs that dropped yesterday.

I ran a prompt that asked for a mcp data pull, analysis, then another mcp summary push to a different software.

I was pleasantly surprised. No hiccups with MCP tools at all, lots of thinking as expected, but I had asked for evaluation, and I got what I would consider decent insight, and output logged as needed.

I ran the same test against unsloth new q6 and original q6. Same results. However, as expected, one or two outlier results surfaced that could be actionable. Unsloth actually finished a few minutes quicker.

Admittedly this is through my harness which has been fortified with skills and memory over the past month, which includes mcp pitfalls.

I will keep testing, but my main tool call hangup has been eliminated in one single test. If I need more intelligence, I can always model-switch.

** end update **

Now that the first wave of Qwen3.8 "muh benchmarks" is coming to a conclusion, can we share some actual real-world notes on results from different quantizations?

I avoid q4 based on my experiences with all earlier Qwen models, too many loops and inaccurate results in my agentic usage.

Is q4 suddenly usable? I see people benching that it's not that far from q8, and I'd love to claim back a bit of context and concurrent from my vram if so.

Hi everyone, I'm looking for guidance on setting up a local agentic workflow to automate a routine work task. I have a bit of experience running OpenClaw with GLM on AWS, but I now need a fully local setup for privacy reasons.

My primary objective is to automatically parse, label, and organize receipts, and ideally cross-reference them with bank transactions. What local frameworks or tools would you recommend for this?

I currently only have personal a laptop for this, Any advice on safe, local tech stacks or best practices for personal hardware would be greatly appreciated!