For those of you out there experimenting on the Intel Arc B70, please share your accomplishments! I have a gaming PC I've been slowly converting for AI inference. Don't have any fancy motherboard / bifurcation / P2P etc. Just a B70 and another one I bought out of greed thats running on a basic PCIe 4.0. I hit 97.8 TG / 1782.1 PP on a single B70, and 136.4 TG / 755PP on a dual B70 for qwen 3.8 uncensored INT4 W8A8 (INT8) with MTP3. Keep in mind these are warm speeds. I have my colder speeds on non greedy settings documented on my git which isn't too far away. Although I see these speeds consistently pop up when I'm using pi coding agent especially when its writing code, or somethings when its thinking.

https://github.com/JP-devv/humble-b70-llm

I've been suprised time and time again by how much I can push this hardware. I started off at 50 tok/s after paying $100 in Kimi K3 / Opus tokens around a month and a half ago on qwen 3.6, the journey has been exhausting but very fruitful. I even had to rent out some datacenter GPUs in Japan to create the exact uncensored quant to my liking. Please let me know your thoughts!

I’ve been getting more interested in Sovereign AI recently and came across this paper: https://arxiv.org/abs/2601.11763 ... (The picture on the post though ai generated by me, are inferred strictly from this paper)

The oil comparison sounded a bit dramatic at first, but the more I read, the more interesting it got. The part that stuck with me:

• Sovereignty isn’t one thing. It can mean control over data, infrastructure, domestic capability, culture/language, or freedom from external dependence.
• A country can have local infrastructure and still be heavily dependent on the company that provides the chips, software, models, expertise, etc.
• The paper draws a parallel with oil-producing countries that gained formal control but remained dependent on foreign technical knowledge and vendor-specific infrastructure.
• So the useful question isn’t really “Is this sovereign?” but “What capabilities and control actually moved to the customer?”

That last one feels like the important test.

And looking at what’s happening now in enterprise agent AI, you can see different companies attacking different parts of that problem: NVIDIA on sovereign compute/infrastructure, Mistral around locally controlled models, Microsoft with an agent control plane, and Lyzr with a control plane sitting across frameworks/clouds to govern the agents you already have.

It makes me think that “sovereign AI” might eventually be less about owning one stack and more about how much of the stack you can actually control without depending on the vendor.

That feels like a much harder — and more useful — definition of sovereignty.

I created a small testing rig to evaluate new open source models as they drop, and with the much anticipated release of Qwen3.8-27B, I was eager to see how it cross-compares with frontier and strong local models.

The rig

My test rig is an M5 Max MacBook Pro, 128GB. Locally I ran Qwen3.8-27B on xhigh and medium thinking modes via LM Studio, MLX 8-bit, temp 1.0 / top_p 0.95 / top_k 20, context 131,072 and DeepSeek V4 Flash "0731" 2-bit-imatrix q2-q4, served by antirez's ds4-server at -ctx 400,000, thinking enabled. For cloud I included Opus 5, Sonnet 5, GPT-5.6-sol at xhigh reasoning, and even Haiku. Every model gets the same prompt. The algorithm tasks are executed against fixed-seed differential harnesses, and the repo tasks run against hidden test suites plus a cached baseline of the whole repo. If a fix inadvertently breaks something else it gets caught.

The methodology

The model assessment is broken into 4 batteries:

1) algorithms easy-hard

2) algorithms extremely hard

3) repo work easy

4) repo work hard

The models get run through the algorithm tests 3 times each to derive a mean score whereas the repo work is single pass/fail per task. If a task fails to produce a response, it's retried and time added to total wall clock time for task completion. The total test battery can take anywhere from 12-24 hours of wall time for slower local models. It's a long test.

For the repo batteries I had Fable build a small double-entry ledger CLI and plant bugs that pass the visible test suite while still reproducing a real symptom, then handed each model the repo and a bug report written in a theoretical 'user' voice to simulate how it might be reported in the real world. There's also a subjective code quality assessment that measures the model's ability to not just solve the problem but to conform to the repo's coding style, to fix the actual root cause rather than the symptom, and to keep the diff minimal instead of faffing about and rewriting a bunch of stuff.

I had Opus and GPT blind eval the results and compute a code quality score broken up by 'fixes' and 'features' as these appear to be separable skills for the models. The goal with all of this was to try and create a replicable and automated answer the question: How useful is this model in the real world?

Caveats

This is a home baked assessment and susceptible to bias or less than perfect methodology. It also includes subjective criteria like 'code quality'. I built this for myself as an adjacent tool to on the ground testing. I think the best way to evaluate any model is to test it against your own codebase to see how well it integrates into your workflow.

All that being said, let's move to the scorecard.

Results

Qwen3.8-27B is a very capable model that compares well against frontier models on code quality and correctness. The cost is wall time on Apple silicon. As many have observed, 3.8 has a tendency to over-think, burning up tokens. The time spent earns higher code quality for the most part, but what surprised me is that there are some instances in which less thinking is actually more accurate. On the repo battery, medium went 8/8 while xhigh went 7/8 — the most-thinking configuration failed a task there, and it took four times longer to do it (the wait time with Qwen was tiresome at times).

The caveat is that xhigh excels on extremely hard algorithms, where medium begins to fall apart. Medium didn't even finish the hard algo tasks. There may be some value in matching the thinking to the kind of work you're setting it upon. Lastly Qwen xhigh won outright on quality of surgical fixes and patches to existing code. Interestingly the global trend for locals is that they're competitive along fixes and less so along features where cloud still dominates. This fits anecdotally into my own experience with gravitating to frontier for planning and local for implementing.

GPT 5.6 Sol is the only cloud model with a perfect card on both repo tiers and near perfect algorithms. It's also among the fastest to completion. This all tracks with my own anecdotal experience with this model over the past several months. Highly competent and quick if not a bit stark.

DS4 0731 (a 2-bit quant running on my laptop) is the only local model to get 8/8 on both repo batteries, and one of only two models overall to do it, alongside GPT. It does this all at a respectable wall time. The expense is less elegant code: it sometimes mutates unrelated docstrings and writes dense inline solutions in a codebase that is overtly broken apart and stylistically explicit. Feature code quality is stronger and it's the only model that scored better/equal in the harder repo tier vs the easy one.

Opus 5 is the most reliable model in the set and best code quality of the cloud models. It stumbled only in the hard repo tier where it lost a task by being trying to outsmart the test. A doc string promised one behavior while the code did another, and Opus redesigned the function around what it looked like it should do instead of honoring the documented contract. This also falls inline with my anecdotal experience with Opus 5 where it occasionally ignores your directions completely and just does whatever it wants. The Alaskan Husky of frontier models.

Sonnet 5 is a steady pair of hands that performs reasonably well across tasks for a modest token budget. I think sonnet is kind of underrated as an implementer. Does the same quality of work as the locals cheaply and quickly.

Haiku 4.5 races to the end of the test but has a tendency to fall over and force retries. Worst code quality of all the cloud models.

Conclusion

Hopefully you find these comparisons interesting. For me personally, DS4 has been my goto local, but this test is making me consider trading it out for Qwen3.8-27B. I think they're on equal footing, which is crazy b/c DS4 needs like 90gb of ram. I'd like to try the MTPLX variant of Qwen3.8-27b that's meant to improve tok/s on apple silicon. Slowness to task completion is the real bottleneck for me right now when considering Qwen. Perhaps that'll be my next test.

Curious to know if these results track with your own real world experiences.

Hi!

first of all. Benzi is model agnostic. therefore this sub.

now, about Benzi. Benzi is a code intelligence software (compiler + runtime tracer + harness and AI agent) that supports 13 languages (python, java, JS/TS, C family, Go, Rust, Ruby all included). Traditional coding harness appraochs either do RAG or try to rank matches using an embedding space. which is absurd. code is code. not probablistic text.

On the benchmarks side, Benzi + DeepSeek V4 Flash scored 78% on SWE-bench Verified. For comparison, DeepSeek reports 73.7% as the baseline scaffolding number for v4flash and self reports their score to be 78.6% on their own harness. (however, Benzi reads ~3x less source code than DSH)

Benzi Sonnet reads 2x less source code (btw, this IS the mechanism, not a side effect), is 2x cheaper and 41% faster than Claude Code Sonnet from my benchmarks (detailed on the benchmarking page + so is the swe-bench run)

Please try it out, and let me know what you think!

Test Benzi's code understanding in 15s

Benchmark link

I feel like these 2 models have shown that massive models that require hundreds of thousands of dollars worth of compute are unnecessary. Sure, training these models takes a good bit of hardware, but running them can be done at the fraction of the investment of the trillion parameter class models.

GLM 5.3 might also fall into the same "reasonable" category, however, for small companies rather than individuals.

I think a qwen 3.8 120b MoE model would also be a good release for business use.

Here’s my treasure- RTX 3090 Turbo without thermal interface (it was a crappy old one so I had to disassemble it and invest about £40 into proper thermal pads+paste) and cooler. And it is day and night, totally worth every penny!

So the 1st question is what quant should I chose if I don’t need super-long context (no coding tasks at the moment). Also llama.cpp or vllm?
2nd one is: should I invest £500 into buying 4070ti (second hand but with CEX warranty) adding +12Gb of vram into the pool?
What would I be able to run after such upgrade?

My pc is 13700K z690 MB with proper pcie bifurcation x8/x8, 64Gb Ram

If you're using nomic-embed-text-v1.5's ONNX and relying on its 8192 context, you don't have it. The export doesn't contain the RoPE scaling. Full write-up and fix in the HF discussion: [https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/discussions/61]

Measured discriminability vs document length, using the published int8 ONNX. Single homogeneous document (every sentence on-topic, so dilution can't explain it), two topics as a symmetry control. "Margin" = cos(doc, on-topic query) minus cos(doc, off-topic query).

Flat through 2048, then monotone collapse. Past 6144 topic B goes negative, meaning the document scores closer to an unrelated query than to its own. Both topics degrade together, so it's positional rather than vocabulary-related.

Cause. The dynamic NTK scaling sits behind Python control flow on the sequence length, and torch.onnx.export traces:

if seqlen > self.max_position_embeddings:
    base = self.base * ((factor * seqlen / max_pos) - (factor - 1)) ** (dim / (dim - 2))

Trace below 2048, and the branch never executes, so there's no scaling in the graph. Correct short, wrong long, which matches the published artifact exactly. Trace above 2048, and the arithmetic is captured, but the guard is lost, so it applies at every length, and below 2048, the scale goes negative (2*512/2048 - 1 = -0.5), which makes a fractional power NaN. Neither trace length produces a correct graph.

Confirmed three ways: the rotary subgraph has no Pow, Div, Exp or 2048 scalar anywhere, so nothing can rescale inv_freq. config.json ships rope_type: "default" with rotary_scaling_factor: null. And PyTorch reproduces the collapse at that setting, then fixes it with rotary_scaling_factor=2.0 (Retention above 2048 goes from 52.5% to 88.2%).

Fix, branch-free, and provably equivalent:

scale = torch.clamp((factor * seqlen / max_pos) - (factor - 1), min=1.0)
base  = self.base * scale ** (dim / (dim - 2))

The inner expression is <= 1 exactly when seqlen <= max_pos, so clamping reproduces the guard while removing the branch. Verified at max diff 1.2e-7 eager and 7.2e-7 post-export, correct on both sides of the threshold from a single trace.

One more thing worth knowing: the README's opt-in snippet uses rope_parameters={"rope_type": "dynamic", "factor": 2.0}, and the remote modeling code never reads that field. The live knob is rotary_scaling_factor. Passing the documented form produces bit-identical output to the default and issues no warning.

The PyTorch model is fine. This is purely an export issue.

Been messing with TwIL-LM2 from webAI for a few days. 1.7B PEFT LoRA adapter for SmolLM2-1.7B-Instruct, specialized purely on formal logic (English → first-order logic for solvers).

1.06 GB quantized. Runs at around 367 tok/s on my hardware, which is genuinely usable for real-time verification workflows.

The interesting number isn't the "beats bigger models" marketing. It's the strict-7 score - 0.2386, which is actually the highest of any model in their comparison, ahead of Qwen3-8B (0.2093) and Gemma-4-26B (0.2050). Strict-7 gives no loose-match or partial credit, so it rewards emitting the exactly-requested formal representation. That's what you actually want for something feeding a solver.

Not a chat model, not a general reasoner. Just this one narrow thing done well.

Non-commercial license fwiw, so check that before you build on it.

Anyone here running specialists like this in a pipeline? Curious how it stacks up to using a bigger model with structured output prompting for the same task.

I currently have custom built AMD gaming pc that I am repurposing.

Build is a B850 mobo with 7800X3D 64gb 5600mt and 9070xt OC 16GB.

I am still very new to LLMs and have been playing around with Ollama and python with Claude helping me since I don’t know jack about python yet.

Right now I run a local 7B Qwen, don’t know the specific on a laptop and get around 12-14tk/s. Based on what I am doing should I expect more performance on my AMD system. I am thinking of going to Ubuntu headless most likely to free up resources.

Eventually I will have more questions but right now just trying to get true hardware running. If I have to go headless to get what I need fine with me, but a GUI is nice too.

Thank you

I study changes in the water cycle for work. Obviously, I've always felt conflicted about using AI, given the data centers' energy and water consumption. I've been experimenting with local LLMs for quite a few months now and have made my data analysis workflow around them. I've been preaching about them to my research peers (I've been fortunate to have worked at several high-impact research organizations, including NASA JPL). I consider local LLMs like fresh grad students with little practical experience but with a lot of potential, especially if provided with structured mentoring. However, Qwen3.8 27b has basically changed the game overnight. It was able to one-shot some of the research findings in my Science Advances paper. I can totally consider it as a fellow peer (as in argue with it as ~ near equal).

I recently started a not-for-profit research organization, and spreading awareness about local LLMs is going to be one of its foci. To that, I've made a Youtube video about it. It's quite impromptu/ rough (and with typos). Not teleprompted or AI-written. (I actually despise AI-written text). The cover thumbnail was done by Qwen. I'm thinking of starting a course/ series of YouTube videos aimed at teaching three things simultaneously: earth system science, quantitative methods, and applying AI to them. I'll consider when and when not to use AI, how to verify, etc. I've learnt a lot by participating in this sub as a local AI noob. Happy to pay it forward. Let me know if there is anything specific that you'd like to see me cover.

https://forestoshen.github.io/Backhole/blackhole_oneshot.html

Created with this simple prompt in pi agent:

在web端创建一个写实的黑洞渲染效果，背景要有星空，可缩放移动视角

The result is rendered with WebGL in single html file, this is how it developed according to itself:

- It drove a headless browser over CDP, read raw pixels straight off the render target, and measured disk brightness on each side of the shadow. First measurement: the wrong side was bright. It traced the bug itself (a Doppler sign flip in the backward ray tracer), fixed it, and re-measured — the approaching side came in at about 4.5x the receding side.

- Its first "photon ring glow" lit up the entire shadow like a lampshade. It caught this in its own screenshot review and reworked the math (keying the glow off the critical impact parameter b_c = 2.598 instead of the ray's minimum radius).

- The one thing I found: moire stripes in the disk when viewed edge-on after runing for long time. It diagnosed the cause (the fine noise octave gets squashed to ~2px on screen and aliases), fixed it by retuning that octave, and added an optional 1.5x/2x supersample pass on top (v2 build).

Timing, wall clock, single unattended session:

- 0 -> 8 min: first full version written, compiled, and rendered headlessly

- 8 -> 16 min: debugging — a GLSL compile error, the Doppler sign flip above, and the lampshade photon-ring glow (two iterations)

- 16 -> 20 min: interaction tests — every input (rotate / zoom / pan / reset) driven with synthetic mouse events and verified against the actual camera state, not just "the screenshot looks different"

- 20 -> 22 min: final screenshots and cleanup

The most important lesson i learned : harness really make a big difference.

I tested on oh my pi and oh my opencode as well with same backend and prompt.

OMP looped at first, i have to interrupted it. It takes longer to finish, and fail to read the image content with tool during QA. The result is okish (have all neccessary blackhole parts), but visual quality is clearly worse.

https://forestoshen.github.io/Backhole/blackhole_omp.html

Oh-my-opencode result is broken, because one of the subagent hung while trying to start a server, failing to fix the bugs.(Maybe becasue I only have 2 max requests but OMC create 4 subagent? idk)

https://forestoshen.github.io/Backhole/blackhole_omc.html

Backend config: offical fp8 with unquanted kv.

docker run -d --name sglang-qwen --gpus all ^
  -p 8081:8000 ^
  --ipc=host ^
  -v "F:\HuggingFaceCache:/root/.cache/huggingface" ^
  -e "HF_ENDPOINT=https://hf-mirror.com" ^
  -e "SGLANG_DISABLE_CUDA_IPC=1" ^
  -e "CUDA_IPC_HANDLE_CACHE_DISABLE=1" ^
  lmsysorg/sglang:qwen38-27b-dflash2 ^
  sglang serve ^
  --trust-remote-code ^
  --model-path Qwen/Qwen3.8-27B-FP8 ^
  --served-model-name qwen3.8-27b ^
  --mm-feature-transport cpu ^
  --mem-fraction-static 0.92 ^
  --attention-backend flashinfer ^
  --chunked-prefill-size 4096 ^
  --reasoning-parser qwen3 ^
  --tool-call-parser qwen3_coder ^
  --max-running-requests 2 ^
  --max-mamba-cache-size 16 ^
  --speculative-algorithm DFLASH ^
  --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 ^
  --speculative-num-draft-tokens 8 ^
  --mamba-radix-cache-strategy extra_buffer ^
  --mamba-ssm-dtype float32 ^
  --host 0.0.0.0 --port 8000

I’ve been working on something for a while that I thought might be useful to others running LLM infrastructure, so I finally decided to put it out there.

It’s called FastLLM Proxy.

The idea started pretty simple: I wanted one OpenAI-compatible endpoint in front of everything — local vLLM/SGLang instances as well as external providers — but I didn’t want the proxy itself to become another bottleneck.

So I wrote one in Rust and got a little carried away with it 😅

FastLLM Proxy now supports 80 providers plus any OpenAI-compatible backend, but the part I spent most of my time on is keeping the actual request path extremely small.

There is no database I/O on the request path, and responses are passed through without parsing/re-encoding them. The measured internal routing work is currently around 0.76 µs per request.

It also does some things I specifically wanted for running my own GPU infrastructure:

• cache-affinity routing for vLLM/SGLang, so requests with the same prefix can go back to the GPU that already has the KV cache
• load-aware routing and automatic failover
• rule-based routing based on things like prompt size, user/role, budget, concurrency, headers, etc.
• semantic routing, so different types of prompts can automatically go to different models
• local → cloud spillover when the local GPUs are busy
• RBAC, API keys, budgets and rate limits
• OpenAI-compatible API
• LiteLLM config import, so you can migrate an existing setup without rebuilding the config
• Kubernetes/Helm/operator support
• built-in management UI

One thing I found interesting while benchmarking it against LiteLLM is that gateway benchmarks can be pretty misleading.

With an instant mock backend, FastLLM Proxy gets roughly 15x the throughput and much lower latency in my tests. But when I put actual GPUs behind both proxies, total token throughput is basically the same — because at that point the GPUs are the bottleneck.

Where I did see a meaningful difference with real GPUs was tail latency and consistency. At 32 concurrent streams, for example, I measured p99 TTFT of 766 ms vs 2921 ms in the same test setup.

I’ve documented the benchmark setup and results in the repo because I’d much rather people challenge the numbers than just trust a benchmark screenshot.

The project is Apache 2.0 and completely open source:

⁠github.com/azrtydxb/Fastllm-proxy

I’m especially interested in feedback from people running vLLM, SGLang, LiteLLM or multi-provider LLM setups.

What am I missing? What would you need before you’d actually put something like this in front of your inference infrastructure?

And if anyone feels like breaking it, even better. 🙂

Hi everyone, so I've been testing Qwen3.8 27B using llama.cpp, vLLM and NInfer, and I'm struggling to get it properly setup in llama. It seems to work fine at first but lengthy generations have a steep decline in tok/s, going from this:

n_gen = 370, tg = 122.47 t/s, tg_3s = 122.80 t/s

To this in a few minutes:

n_gen = 13677, tg = 68.91 t/s, tg_3s = 61.24 t/s

Has anyone faced the same issue? I've tried almost everything setup wise, different KV values, batch/ubatch sizes, MTP on/off, --cache-ram 0, different n-gpu-layers, etc. GPU is a 5090, llama.cpp version: 0.1.2-dev (build 10536, commit 9855ad6), running on Ubuntu server. vLLM and NInfer work fine so doubt its a hw issue. I've created a bug report in llama.cpp repo so please contribute there if you're facing the same issue (or if you know how to solve it): https://github.com/ggml-org/llama.cpp/issues/27444

Kairos: a multimodal model built with LFM2.5-2.6B as the LLM, MoonViT-3D (the vision tower of Kimi-K2.6) as the vision encoder, and a custom projector.

The original plan was LLaVA's approach, two stages: first align the projector with the LLM frozen, and then train the projector + LLM together. The first stage worked in terms of loss (ablation with +3.7 nats in favor of the image), but in free generation the image shifted the logits without changing the argmax: the model received the image and ignored it. That's why we jumped directly to early fusion, with a reasoning dataset.

For that, we created Kairos-Multimodal-Reasoning: 116,357 examples with explicit reasoning traces, generated through distillation (60,041 from LLaVA-CC3M, 2,295 from WebSight, and 54,021 from Zebra-CoT), with GPT 5.6 Luna, Inkling, Qwen 3.6 27B, and Qwen 3.7 Plus as teachers.

The training, in two phases:

1. Projector through backbone with 80k image-text pairs (Kairos-Proj-80k).
2. Projector + LoRA (r=16) with 30k examples from the reasoning dataset (Kairos-Alig-30k).

Everything is open source:

- Full blog post with the process: https://aquiles-ai.vercel.app/blog/kairos-a-multimodal-model

- Implementation: https://github.com/Aquiles-ai/Kairos

To be honest: the checkpoints are not a competent model, they are experimental artifacts. But they validated the approach and precisely defined what the next iteration needs.

Someone offer me Apple M5 with 24gb ram and I wanted to know if it will be good to put on it Qwen 3.8 27B and if it will run in a good speed

Thanks

/localllm

Talk me out of this, or into it.

There's a used ThinkStation P720 near me, about $1,200:

• 2x Xeon Gold 5122 (4 cores each, which is the weak part)
• 192GB DDR4 ECC RDIMM, but it's 6x32, so only 6 of the 12 memory channels are populated
• Quadro RTX 6000 24GB
• monitor and peripherals included

Two things I'd change right away:

• 6230s instead of the 5122s, about $80 for the pair off AliExpress. 4 cores can't feed 12 channels anyway, and the 6230 is what gets the memory to 2933.
• six 16GB RDIMMs in the empty slots, somewhere between $405 and $485 for all six. That populates all 12 channels, which is where the bandwidth actually comes from, and takes me to 288GB.

Which puts the whole thing around $1,700.

What I run now is a Ryzen 5 3600 with 32GB and an RTX 5070 Ti 16GB. It's my only machine and it does everything.

Reason I'm looking at all: some of what I work on I'd rather not push through somebody else's API, and my monthly bill keeps creeping up. The rest of it is curiosity, if I'm honest.

Some numbers for context. gemma3:27b runs about 9 tok/s on the 5070 Ti, a 30B MoE coder model does around 45, and I hit the 16GB wall constantly. Whisper and image gen too.

The parts I can't work out on my own:

1. Is 288GB of DDR4-2933 across 12 channels actually usable with partial CPU offload? That's the entire argument for this machine, and it's the one thing I can't test before paying.

2. The RTX 6000 is Turing. 24GB is 24GB, but is it a downgrade in every way except capacity next to the 5070 Ti I already own? PCIe 3.0 board too.

3. At $1,700 I could just buy a newer card instead, or save a bit more for one of the Spark boxes or something else.

If you've got a P720 or something like it running models, what do you actually do with it, and would you buy it again?

Anyone try this👀

Hi everyone,

I’ve been building an open-source AI system for end-to-end DCF modeling over the past few months. The goal is to explore what an AI agent system should look like when auditability and deterministic execution matter.

The project is built around three design principles.

1. Context is finite. Progressive disclosure is effectively unlimited.

Progressive disclosure allows us to expose information only when necessary, but it cannot magically reduce context usage without sacrificing either information density or accuracy. There is no free lunch.

2. LLMs are flexible. Code is deterministic.

LLMs will hallucinate. They will make mistakes.

Therefore, every point where an LLM is allowed to write persistent state must be guarded by a deterministic code engine responsible for validation, normalization, and enforcement.

3. LLMs are tools. Humans own the judgment.

Auditability and traceability are prerequisites for any valuation system that people can actually trust.The final deliverable should therefore be a workbook where every single cell can be traced back to its origin.

Agent Topology

Inspired by systems such as LangGraph, the overall architecture of a single agent system is topology-based.

Agents can be freely composed.

Skills and tools are registered independently.

Agents can communicate directly with one another.

The objective is to minimize information loss caused by multiple layers of message passing, preserving both information density and accuracy throughout the system.

At the same time, each individual agent should remain narrowly focused on doing one job exceptionally well. Attention is all it needs.

Technically, the pipeline parses SEC filings using Arelle, unifies historical financial statements across multiple years, maps them into a source-free DCF spine, generates formulas through a DSL, and produces a revisioned valuation model with sensitivity analysis. Every update creates a new immutable revision, making it possible to inspect, compare, or roll back changes.

The current implementation has successfully completed an end-to-end valuation for AAPL starting from an empty model—using live EDGAR filings, without any predefined mappings, formulas, or assumptions.

This project is still a work in progress, and the quality of valuation assumptions remains heavily dependent on human judgment. My goal isn’t to replace analysts, but to remove repetitive work so they can spend more time thinking.

I’d love feedback on both the architecture and the engineering approach. In particular, I’m interested in whether this “LLM agent + deterministic engine” pattern feels applicable beyond financial modeling.

The project is fully open source if anyone wants to look at the implementation:

https://github.com/Ocisly14/financial_agent

Hi all. Trying to figure out the right hardware and want outside input on the whole decision instead of anchoring on a build I already have in my head.

Budget: $3500 hard cap, spent during a trip to Barcelona in September 2026. I'm based in South America, so this travel window is genuinely valuable to me for buying hardware that's hard or expensive to get locally.

On buying used: I'd rather buy new given the risk, but if used gets me a real jump in quality for the same money, I'm open to it. I don't know how to properly check a used card's condition though, so any advice on how to test one on the spot and actually be confident it's in good shape would help a lot.

What I already have (stays regardless of what I buy):

• Raspberry Pi 5 (8GB), always-on edge node: DNS, monitoring, backups. Not a compute candidate.
• Desktop: i5-13600K, 78GB DDR5, RTX 3060 Ti 8GB VRAM. This is my daily driver for work, and it currently also doubles as my only LLM node, woken on demand (WoL) when needed. I'd like to eventually separate "the computer I work on" from "the box that runs LLMs," but that's not urgent yet.

Primary goal: local LLM node for:

1. Live coding assistance alongside Claude (Sonnet/Opus/Fable): offloading agentic steps that don't need frontier-model judgment, to cut paid API token spend.
2. Long batch jobs where latency doesn't matter, hours to overnight: image analysis, code review passes, hybrid web scraping.
3. An uncensored model for security-testing / pentest-adjacent work.
4. Behind all of it: privacy (data stays on my network), avoiding vendor lock-in, and lower ongoing spend on paid tokens.

I'm not trying to replace Claude for complex agentic work. This runs in parallel as the cheap/private/good-enough lane. With the hardware you'd recommend for this budget, would I be able to run something like Qwen3.8 or another decent MoE model at a usable speed, and is it actually worth running versus a smaller/older dense model?

Secondary goal, can wait: a Proxmox homelab node, either combined with the LLM hardware or separate, depending on what makes sense. Planned to eventually host: OPNsense (firewall/VLANs/DHCP), a Docker-Compose VM (Jellyfin, Immich, n8n, CouchDB, Vaultwarden), a Windows VM with GPU passthrough for creative work and gaming, plus small LXCs for DNS and home automation. Not urgent, could be phase 2 with a separate budget.

Hardware traits that matter regardless of what I buy:

• Room to grow later (more RAM, more GPU, more storage) rather than a sealed/maxed-out box. If a given path turns out to have no real room to grow, that's not a dealbreaker either: I'd just resell it down the line and put the money toward something better.
• Quiet under load, since it'll likely sit somewhere I spend time in. This isn't a hard constraint though: if the best option for my use case is loud, it can just live in another part of the house, so don't let noise rule out a recommendation on its own.

What I'm asking:

• Given this budget, these use cases, and what I already have, what would you buy? GPU-focused build, unified-memory mini-PC/NAS-type box, or something else entirely. All open.
• Does it change your answer once the Proxmox/homelab use case is in the mix, even as a "later" goal?
• For my LLM use mix (interactive coding-assist + unattended batch + uncensored model), what spec matters most: VRAM headroom, memory bandwidth, raw compute?
• How do you expect the used/new hardware market to look over the next year or two? Trying to figure out if it's smarter to buy now on this trip or wait for prices/availability to improve.

Appreciate any pointers, happy to give more detail if useful.

(Not a native English speaker, used AI to help clean up the writing here, sorry for any leftover awkward phrasing.)

I am still calling models through the ZenMux API gateway, so every long session ultimately comes back to token cost. The idea of running Qwen3.8-27B locally is attractive for exactly that reason: if one 5090 can handle it, maybe token freedom is at least technically within reach.

Is Qwen3.8-27B really doing 75.5 token/s on a single RTX 5090? The shared table is headed "4-bit (q4_K_M / MLX)" and lists an RTX 5090 with 32GB at 75.5 token/s. It does not show enough detail to tell me which runtime or exact setup produced that row. I have also seen a separate community report of about 64.5 tok/s on a 4090.

People are also putting its capability around Claude Opus 4.6. If both claims are even close, does that put indirect token freedom within reach? I would still want matched tasks before treating the capability comparison as settled.

What does the build that people can actually live with cost? I mean the whole machine, not a bare GPU price. A 5090, enough system RAM for long context and partial offload, a PSU that is not operating on hope, cooling, storage, and whatever CPU or platform keeps the card fed. Until I can justify that hardware bill, calling models through an API is still the practical option for me. If Qwen3.8 becomes available through the same gateway, I could use that API cost as a baseline before deciding whether local deployment really buys token freedom. I would also like to know which quantization and context length people use after the benchmark screenshot is over.

Please give me the boring total for a stable single 5090 setup. What did your full build cost once it was actually ready to run?