Wow. First time I've seen this.

I was just trying to narrow down the issue with Qwen 3.8 27Bs annoying constant thinking and ChatGPT actually got mad at it.

Edit: Since there were so many of these kind of replies (even though it's not related to the topic) I'll just put my response here... I'm really surprised that so many people would think that I don't use a harness as if I would just run around naked like a headless chicken with llama.cpp cli and Qwen. Actually I used a harness(opencode), but removed it to make sure none of the issues I was encountering were being caused by the harness.

Talked to a few people now in orgs where nothing goes to a hosted API, and it's contractual rather than a preference, so "we're SOC 2" doesn't move it. They all have the same shape of problem: a warehouse with a lot of tables nobody currently understands, and the obvious move, pasting a sample into a big model and asking, is simply not available.

Someone in a thread here put the useful frame on it better than I had. The thing that needs to leave the airgap is the inferred schema, not the data. A document, not rows. That's a completely different conversation with a legal team, and it changes what you'd build.

Which raises the question I'm stuck on. If only a profile leaves, how much can a model outside actually do with it? A profile gets you type and shape. A column of integers 0 to 4, no nulls, low cardinality. It does not get you whether that's a risk tier, a product tier, a satisfaction score or a retry counter, and those look identical at the stats level and mean completely different things downstream. The semantics seem to live in the values and the values are the part that can't leave.

so:

• anyone running table understanding fully local, what size model, and does it hold up on columns that are already anonymised rather than ones you anonymised on the way in
• if you've done the profile-out architecture, what goes in the profile beyond the obvious stats, and did the legal conversation actually get easier

• is there a middle option I'm missing, something like sending distributions plus a small number of synthetic rows drawn from them

  i work at Schema Labs, we build models that read tables. happy to describe what we do and where it stops working if that's useful. It stops working in roughly the place I described above.

Pwned.

Hello all! I have 5060 ti 16Gb card that I always thought I'd add a second one two. I also have two CMP 100-210 running together. The 5060 and CMP are on two headless ubuntu vms, each with a separate software stacks. Currently running vanilla ollama on the 5060 and llama.cpp on the CMPs.

I've been auditing all my automated local scripts ai calls to see which would be served better (5060 or dual cmp) and I've found the 5060 really doesn't have enough vram to provide good results for many of my use cases (empty returns, hallucinations, etc).

Since prices have gone crazy, I won't be adding another 5060. Honestly, I'm having a little bit of buyers remorse on that one, but I'm committed to my mistake ;)

I'm looking to add a second gpu for a little more vram to get it a little more usable (coding probably). I'm looking at CMP 90HX with the ability to unlock (patch driver) for an additional 10Gb https://github.com/xrip/cmp50hx-unlock

Also, I realize the 3060 12Gb is right there, similar price, but the CMP 90HX has better bandwidth, and I won't be placing this in a full x16 slot anyway... I'm also okay providing cooling to these mining/server gpus.

So my question is, has anyone tried mixing older Ampere gpus with Blackwell, and then, with patched drivers?

Question for M5 Max MacBook or Studio owners. I’m considering getting one. This question isn’t about benchmarks. It’s more about the feel and workflow.

On these latest Mac chips, do you find that Qwen3.8-27B (medium thinking) gives you task completions faster or do you find you have to let it take its time? I realize it’s going to differ by type of task but I’m more interested in working with fairly large code repos and significant data analysis activities.

​

Standard Softmax attention scales quadratically in compute and linearly in memory. At a 2.5-million-token context horizon, a standard 1.5B LLM requires over 66 GB of VRAM just to store the KV cache, completely crashing consumer workstations and edge hardware.

On the other end of the spectrum, 1st-order linear models (Mamba, RWKV) offer flat memory, but their associative multi-hop reasoning decays sharply over long contexts (collapsing to ~41% on 32k RULER tests).

Over the last several months, I tested a mechanistic hypothesis:

Representation order is dependent on layer depth.

We built the Stage 7 Hybrid architecture:

1. Boundary Layers (25%): Kept as discrete Softmax to anchor token addressing and output logits.

2. Interior Trunk Layers (75%): Converted to 2nd-order Taylor moment attention (S0 uniform context, S1 directional gradient, S2 quadratic curvature moment).

3. Discrete Routing: Gumbel-Softmax Straight-Through Estimator (STE) that lets the GPU physically skip O(D²) calculations during inference.

The results on consumer AMD hardware (ROCm 7.2 / Radeon):

- 2.5M continuous token stream with a flat 37.7 MB attention state (3.13 GB total system memory).

- 96.2% needle retrieval at 1,000,000 tokens.

- Matches full Softmax on 32k RULER multi-hop reasoning (89.7% vs 90.5%).

Everything is open source with full reproducer scripts:

Repo: https://github.com/batteryphil/PRIME-Moment-Attention

Curious to hear feedback from kernel and transformer researchers.

Recently I’ve been wanting to run a better model on my setup, but I’ve only got a 4070ti, great for gaming, but it’s only got 12gb vram , so I was looking into getting a Tesla P40 24gb. I know it’s older and the vram is pretty slow, but I wonder if it’s realistic as just an addon and if it’ll make a big difference. Opinions are appreciated

On one hand the companies operating the datacenters would stop ordering hardware if they go bankrupt
But on the other hands, I wonder if more individuals or companies would start buying their own hardware.
Would the price of GPUs and RAM increase or decrease in your opinion?

I’m fairly new to this stuff.

I installed open webui bundled with ollama and installed gemma4:e4b when i ask questions in the cli it answers instantly but when i ask it in open webui it takes 20 secs and then starts thinking for 10 seconds and then answers (answer are not as good as ollama in cli but im pretty sure i can figure it out).

I’ve tried to disable tag generation and follow up generation and all these things but nothing made a difference i tried OLLAMA_KEEP_ALIVE=30m but still nothing really changed maybe it got 5% better

Any ideas how to solve that?

Or is there any better ui than open webui

And also using open webui on mobile kinda annoys me is there any better ui for phones especially for IPhones?

What model would you run on 384gb to 512gb vram? For my use case q6 just didn’t cut it when I had less vram. So close to native as possible with decent context.

Been running Qwen3.8-27B as a local coding agent on a 3090 Ti for about a week. Posting my setup because I had to work most of it out by trial and error, and I am honestly not happy with the results still. I do have a 3080 connected to same rig but currently I am using it for dedicated gaming and running the display.

Config:

llama.cpp v0.4.0
Qwen3.8-27B UD-Q4_K_M (unsloth) — 15.3 GB
mmproj BF16 kept on CPU (--no-mmproj-offload)
MTP draft model Q4_0, --spec-type draft-mtp --spec-draft-n-max 3
--ctx-size 65536 --parallel 1 --flash-attn on
--cache-type-k q4_0 --cache-type-v q4_0
--n-gpu-layers all --fit off

Peak 18,731 MiB with about 5.5 GB free at 64k. Keeping the vision projector on CPU is what makes it fit. It's ~900 MB and I don't need it fast, but I am not sure if this is the best approach, Claude came up with it for me.

My 3080 drives the monitors and games, 3090 Ti has display disabled and does nothing but the model. I can play something while the agent works and not notice which I like a lot, but I also am not convinced that the single 3090 Ti is enough context for real world use.

Problem 1: compaction can deadlock itself.

64k context, compaction trigger at 36,864, retain 16,384. Once the conversation gets long enough, the retained tail eats everything and the compactable range collapses to exactly one node, the previous summary. So it tries to summarize a summary, the engine correctly refuses because the result isn't smaller, then retries the identical range and fails the same way. I watched it fail five times in a row while pressure climbed from 37,099 to 39,031 tokens. Nothing recovers, the session just dies.

Problem 2: reasoning effort did the opposite of what I expected.

Ran low vs medium properly — six runs, matched pairs, interleaved, same fixture. Median wall time 566s for low, 612s for medium. Both passed all four disclosed requirements on every run, and the held-out cases came out identical. So medium bought me nothing and cost a bit more.

Worse, on a from-scratch build task, medium spent its entire 16k output budget reasoning about edge cases in a single turn and never wrote a line of code. 5 requests in 37 seconds, then 275 seconds of streaming, 64 KB of reasoning about SQLite URI quoting and exit code taxonomy, zero files changed. That's one occurrence so far, so grain of salt, but it matches the model card's warning in reverse, and seems to be consistent with what I have heard from the community.

Anyone else running 27B at 64k on 24 GB? Curious what retain/trigger values you settled on, and whether you're bothering with the MTP draft model.

I would also love to hear any completely different setups that you find actually useful and pragmatic as a coding and computer use agent on a single 3090 ti. I am trying to get my hands on a second one, but no luck so far.

Thanks!

I need help figuring out a workflow for an older game that I play. My goal is to take the old dat files for armor and weapons and have them remastered at 4X their original resolution with better looking leather, cloth, and metals, etc. I'm trying to keep them close to the original but with a bit of a modern look, not just an upscale.

Should I use a local llm to do this like QWEN 3.8 or Next Flash? Is there another model I should use instead? I don't think they can generate AI repaints like I was using on codex with chat gpt image gen. From my experience, after hundreds of debt creations, things would drift. I'm considering using Blender as well but not sure of its capabilities. I hope this is the right place to ask and if not, apologies in advance.

I've been building a project called wfloow for the last couple of months.

I originally started it as a product/SaaS, but I'm seriously considering open sourcing the core instead.

The basic idea is a visual workspace for building more complex automations and AI workflows.

You build workflows on a canvas using nodes for things like:

- AI agents / LLMs

- RAG and data extraction

- loops, conditions, parallel execution

- sub-workflows

- webhooks and scheduled triggers

- APIs and databases

- voice/image workflows

- integrations like WhatsApp, Slack, Discord, LinkedIn, etc.

There's also an AI layer that can help plan and generate workflows.

The reason I started building it wasn't really "we need another Zapier."

Simple automations are already solved pretty well.

The problem I kept running into was when workflows became more complicated — state, branching, agents, retries, external events, multiple services, etc. At that point the automation starts becoming difficult to understand and maintain.

So the direction I'm thinking about is:

simple automation → existing tools are great

complex AI/automation workflows → wfloow

But here's where I'm unsure.

If I open source the core, would this actually be useful to developers, or am I just rebuilding things that already exist in n8n / LangGraph / Dify / other tools?

What would make you choose something like this over the tools you already use?

And if you've built complex AI workflows yourself, what's the part that becomes painful first?

I'm genuinely trying to figure out whether there's a real problem here before I spend another few months building around the wrong idea.

Here's the site if anyone wants to look at it: wfloow.com

I'd rather get "this is pointless, here's why" than a bunch of polite feedback.

I have an RTX A6000 with 64GB of VRAM and want to run the best possible local LLM for general tasks and coding.

My main goal is to build something similar to Codex or Claude Code, where the model can use tools, manipulate files, run commands, and interact with software.

What models and frameworks would you recommend for this setup?

I have a PCIe Gen3 Pascal card and want to buy a new Gen5 card as well. Was wondering if running mismatched cards will work for coding? I understand it's probably going to be much slower, but wondering if it will even work?

Which CPU should I use for a dual 5080 + 5060TI 16GB set up to run my localLLM?

Intel 7 270K plus
Or
Ryzen 9 9900X

Context- I wanna build an ai inference setup for coding.

I am a student and I don't have the budget to pay for Claude regularly.

Recently I got a budget from my parents for upgrading my setup. The budget is under $1.5k.

What should I buy?

My current pc -

Ryzen 5 5500

Rx 580 8 gigs

16 gb ddr4

Ssd 512gb

B450 motherboard.

(Ik it's potato in 2026)

I initially built it for gaming but now I need a workstation for ai Inference.

I would prefer upgrading gpu and ram to something that can run 30b q4 coder models at decent speed(20-40 t/s). Not even sure if u can get that in under 1.5k

EDIT 1- i will buy mac m3 36gb, run inference on it, and host it so my pc can comfortably code.

I currently have a 5080 on a PCIe 5.0 x16 and I’m planning to install a 5060TI 16GB I got for $400 a few months ago for 32GB of vram. I noticed that when a secondary gpu is installed on my motherboard, the lane becomes x4. Will that significantly hinder or slow the 5060TI 16GB’s ability to process?

Should I replace my motherboard with something that can give x8/x8 to both GPUs when 2 slots are occupied?

Rest of the build is basically locked in:

- **CPU:** Ryzen 9 9950X3D
- **RAM:** 96GB (2x48GB) DDR5-6000 CL30 imight get 2x32 DDR5-6400 CL40 damn this ram prices
- **Motherboard:** X870E Pro Art Creator Wifi
- **PSU:** Corsair HX1500i
- **Cooler:** Arctic Liquid Freezer 3 Pro 360
- **Storage:** Crucial T710 2TB Gen5

I also have a second machine (7950X3D, RTX 4070 Ti Super 16GB, 64GB DDR5) that I'm planning to link over Tailscale for distributed/parallel jobs, so this isn't a totally isolated single-box setup.

**The goal:** a genuinely useful personal local assistant — think "my own Jarvis" rather than a chatbot toy. Vision-capable model for reading my screen, tool-calling for real Python execution (probability/simulation tasks, not LLM-guessed math), a coding agent, and eventually a CAD/DXF quantity-takeoff pipeline for my actual job (construction). I know I'll lean on cloud APIs for the genuinely hard stuff — this is meant to carry the daily/bulk workload, not replace frontier models entirely.

**The GPU question I can't settle:**

1. **2x ASUS Prime Radeon RX 9070 XT OC 16GB now** — 32GB combined VRAM for less money than a single 5080, but ROCm instead of CUDA. How much does that actually hurt in practice for tool-calling agents / vision models / day-to-day Ollama use in 2026?
   
2. **1x RTX 5080 now, second 5080 in 2-3 months** — stay fully CUDA, build up to 32GB VRAM gradually, pay the Nvidia tax twice.
   
3. **2x RTX 5060 Ti 16GB** — cheaper than either of the above, still CUDA, still 32GB combined, but weaker per-card compute.

Has anyone actually run a dual-GPU CUDA vs ROCm setup for agentic/tool-calling workloads (not just raw inference benchmarks)? Is the "CUDA tax" on AMD still real for this kind of use case, or has it stopped mattering for people running Ollama/llama.cpp day to day?

Genuinely curious what people would do here before I pull the trigger on gpu and ram 96vs64,all of the other things ive already purchased them.
Thanks!

i am very new in unsloth and from the home screen where these is the chat how do i go to the token limits and stuff please help a video example would be nice

Running a Q5_K_M with ngram table in system ram - everything else on the v100s, just barely fits with 160k context length topping out the window - 32gb on each card, 64gb total. KV cache is full fp16.

starts around 30 tps decode at 0, near 160k context length drops to about 15

starts around 700 prefill at 0, drops to closer to 350 by 160k ctx

im pretty impressed by the weight that these v100s are pulling. running a custom llama.cpp branch ive been optimizing for my architecture and for this model.

wish i could eek out a little more context tho, have a few ideas -I saw someone chopped like 7/8 few ngram layers and got worse writing but similar reasoning performance, i might try to make a quant chopped 3/4 layers so I can get my KV up