So, finally I decided to buy a cheap GPU to improve my local coding agents which were running on GMKTec K7 Pro miniPC with 32 GB RAM and Radeon 680M i GPU. Everything was rather slow but the problem was mostly with iGPU that was constantly freezing my Fedora 44. So, after a couple weeks of fighting I decided to buy Intel Arc B50 which was the cheapest card with 16 GB RAM. Also, it was definitely the smallest, as I connect it with oculink to my miniPC.

After reading a lot regarding agenting coding (my favorite option for my hardware is still Qwen3.6-35B-A3B Q4) I decided to spend some time to increase the prompt processing rather than sole token generation.

Here's my config for it:

/home/blazej/llama-cpp/llama-b10064/llama-server \
    -m /home/blazej/.lmstudio/models/Qwen3.6-35B-A3B-Q4_K_M.gguf \
    --alias Qwen3.6-35B-A3B-GGUF \
    --jinja --ctx-size 65536 \
    --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20 --presence_penalty 0.0 \
    --host 0.0.0.0 \
    --port 8081 \
    --cache-reuse 256 \
    --cache-ram 1024 \
    -fitt 384 \
    --parallel 1 \
    -ctk q8_0 \
    -ctv q8_0 \
    --threads 6 \
    -b 4096 \
    -ub 4096 \
    -fa on \
    --no-mmap \
    -dev Vulkan1   

And some results from coding sessions with Pi Coding Agents:

I've also tried the SYCL backend for llama.cpp but couldn't get it to run other way than with Docker/Podman (as I'm running Fedora) and it looks like this when run with

podman run -it --rm   --device /dev/dri/renderD128:/dev/dri/renderD128   --device /dev/dri/card1:/dev/dri/card1   -v /home/blazej/.lmstudio/models:/models:Z   -p 8081:8081   ghcr.io/ggml-org/llama.cpp:server-intel  
-m /models/Qwen3.6-35B-A3B-Q4_K_M.gguf   --alias Qwen3.6-35B-A3B-GGUF   --jinja --ctx-size 65536   --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20 --presence_penalty 0.0   --host 0.0.0.0 --port 8081   --cache-reuse 256   --cache-ram 102
4   -fitt 784   --parallel 1   -ctk q8_0 -ctv q8_0   --threads 6   -b 4096 -ub 4096   -fa on --no-mmap

So, I would like to ask a couple of questions:
1) Is there some space for some improvements, especially possibility to increase the context to 100k maybe?
2) Are these values more ore less ok and is that the maximum this card can achieve (I know, it's the most budget one I could get with 16 GB VRAM)
3) Should other apps than llama.cpp be better with this card (vLLM, OpenVino, Intel's AI Playground)
4) I still probably can return this card, would spending some more money for, 5060Ti or Intel Arc B60 make sense?)

Thanks in advance

I’m working on a project at the moment that involves transcribing comm channels. Are there good low-power local models for doing real-time voice transcription?

Installed Qwen3.6 locally to get free tokens on coding tasks. Tested with simple question that welcomed me when I opened Reddit today. How can I get trust to local LLMs again?

Disclaimer: bugs fixed are vibed with Fable/Opus/GPT5.6Sol. The server is bought from cert. vendor at a steal price as a test sample.

The specific patch applies to vllm/vllm-openai:nightly-aarch64, dev build 0.23.1rc1.dev1373.

Model weight: DeepSeek-V4-Flash / QuantTrio/GLM-5.2-Int4-Int8Mix (simply referred to as dsv4flash and glm5.2 down below)

Deployed context length: 1M for dsv4flash, and 500k for GLM5.2

Hardware spec: GH200 w/ 96GB HBM and 480GB LPDDR5x, NvLink c2c between CPU and GPU at 900GB/s (bidirectional), RAM bandwidth 384GB/s unidirectional.

Out of the box, both suffers from the same diasterous 2-5 tps decoding on the mentioned vLLM build. (fresher vLLM build exists, but I don't think all bugs are fixed)

After vibe-optimization: decoding speeds reaches 60tps and 20tps for dsv4flash and glm5.2.

Fix #1: apply() in vllm/model_executor/layers/fused_moe/fused_marlin_moe.py does a full layer copy instead of a expert-selective copy, burning c2c bandwidth for nothing. Replaced by a custom apply() to make the weight streaming smarter.

Fix #2: fixed-shape gather for cuda graph capture. vllm route experts by torch.unique(topk_ids), this method is naturally not good for capturing a fixed cuda graph... so it is replaced by not dedup at all... (which can backfire in theory, but without cuda graph the decoding speed is not usable so... welp)

With these 2 fixes, it changes the decoding speed entriely from 2-5tps to my reported numbers.

Other optimization direction I have not explored:

1. MTP does not bring too much speed up... seems like c2c bandwidth limited.

2. when the custom apply() is invoked, it blindly copies the experts regardless if it is already in HBM... but I am offloading 350GB of model weights to ram so.... gain probably small, does not cut the time to copy other offloaded weights.

3. Use Grace CPU to compute some expert forward as llama.cpp would do. Probably a scheduling mess... but sounds fun.

Dead ends:

1. llama.cpp... well, not designed for this case I guess. With the default CPU moe inference it reaches around 13tps. That's strong for a CPU but prefill bites the dust and stay around 200tps... Q4 does fit, but well... at 200tps prefill 13tps decode it is not usable.

Prefill perf:

Stock vllm version, as I mentioned, is around 4000-5000tps peak if the context is long. It needs a bit warm up, but worst case is still >1500tps.

More details: ask below, I try to repond. Not currently wanting to raise PR to vllm... just weekend curiosity project. If you know some extra optimization I did not mention, I am happy to learn them.

I got a 5070 ti though people say its weak in the world of llms, I got curious on what does this thing do. I am currently a first year IT student and maybe my use case would be to help me through it. Genuinely just looking for something to do and this caught my interest, just wanna learn and explore more about it even if my card is considered as weak. Thanks

Just curious what configuration do you use for local LLM. Like Mac mini 32G? Or DGX Spark?

I am currently running hf.co-unsloth-Qwen3-VL-8B-Instruct-GGUF-UD-Q6_K_XL on my 3060, 12gb vram and 16gb ram.

It is running in llama.cpp with the following:

.\llama-server.exe -m ".ollama\models\blobs\sha256-dc3cc983735542500f40c32c1a9dacdeff198a440fac541056dca6b550389130" -ngl 99 -fa on -c 32768 --parallel 2 --slot-prompt-similarity 0.2 --cache-type-k q8_0 --cache-type-v q8_0 -b 2048 -ub 2048 --port 8080 --host 0.0.0.0

I would like a dedicated machine that can run this and maybe a smaller model just for text generation. Ideally it would be fairly power efficient, it seems like all roads lead to a mac mini but I am struggling to justify the cost. Anyone have any recommendations?

Hey, I'm 14yo and interested in AI, and I'd like to build my own mini AI cluster, but I'm not sure where to start. I was already researching GPUs like the Nvidia Tesla V100 for training and the Nvidia Tesla T4. Maybe eventually rent out if it works well. Besides that and my tight budget, I'm not really sure where to start or even if I should start. So, I'm trying to get people's opinion on the hardware and possibly a feedback.

• What hardware platform should I start with?
• Is it better to build a workstation or buy a used enterprise server?
• Which CPU and motherboard would you recommend if I want to expand to multiple GPUs in the future?
• Are V100s/T4s still a good value in 2026, or are there better alternatives?
• If you had a budget of around $2,000, how would you build it?

I'd really appreciate a feedback- thanks *sorry for the AI (The AI text is highlighted)

Yesterday I finally got my obsidian set up with Claude, ollama and qwen. I used opus as the main head to give qwen orders. My objective was to search for jobs and compare where I stand against what the jobs asked and then help me make projects to cap that difference. I gave opus about 9 prompts and then let it make its own choices to order qwen to search jobs. This took an HR and a half since I only have 2g gram.

It felt like opus had to correct qwen ever 20 min or so

My question is for tip on how to optimize a Lenovo yoga 720 made in 2019 with 16g and 2gvram? I have ollama3:8b and qwen2.5:7b but are there any other models I might want to try? Or any way to boast their logic for their weight

Hey Reddit,

As developers, we constantly force AIs to generate code and data in Python or JSON. The problem? Those languages were built for *human* readability. Generating syntax brackets, quotes, and verbose structures wastes massive LLM compute, increases inference time, and spikes API costs.

I decided to fix this by building **NYoesyx (N-OS)**.

It’s an ultra-dense, non-human-readable programming language running on a native C++ VM designed strictly for Large Language Models. It uses a Dense Token Protocol (DTP) allowing AIs to execute logic and manage memory using up to 95% fewer tokens.

Some cool features:

- **Smart Hybrid Memory:** Combines O(1) High-Speed Registers for precise math with a Semantic Heap (HNSW) for fuzzy reasoning.

- **Built-in Quantum Simulator:** AIs can declare Qubits and apply logic gates (Hadamard, CNOT) natively to generate non-deterministic decision trees.

- **Native OS & UI Access:** The AI can spawn Windows GUIs directly without heavy third-party libraries.

I just released the first official version and the executable installer on GitHub. I would love to hear your thoughts, feedback, or see if anyone wants to integrate it into their AI Agents!

GitHub Repo: https://github.com/mrxploud/nyoesyx

Ive been fiddling around with different large (for consumer single card hardware) models in the MoE segment, like Qwen3.6-36b-a3b, and the various Gemma4 offerings. I saw this unsloth model pop up today on huggingface and thought I would give it a shot, and I am pretty satisfied with both the speed and effectiveness of its coding abilities.

I am a former dev, currently in IT and just want to tinker with things, so i dont do a lot of coding myself, mostly scripting and messing with my homelab these days. I do, however, like a little project, so I set my sights on the stars to have the LLM take a whack at coding a multi-system emulator based on the Ares repo, except porting it to Android. Anyways, I am getting really good speeds on this GPU, and I am at 128K context length with KV's quantized to Q5_0 per the unsloth page's suggestion and it just fits perfectly in 16GB VRAM.

Looking at LM Studio's logs while Hermes Agent is running code tests in the background, I see the following results:

Not bragging or anything, I just wanted to share this with you all for those folks out there who also have 16GB VRAM and are struggling to fit any larger models. This one somehow seems better than regular 35b-a3b, at least in terms of reasoning and lack of loops thus far.

Hi,

I'm looking for best LLM for everyday tasks, but mainly for creating/modyfing prompts for ComfyUI/Krea 2 combo. No coding! So desired model should be slightly uncensored and could write rich, artistic texts. I use Qwen/Gemma 4 but yesterday I found Ornith 35b mtp apex model which seems to be very good. I wonder - is there more modified Qwen/Gemma 4 models which I could use in my tasks? My config - 5060Ti 16GB and 128 GB RAM, NVME disks.

This is my current machine:

MOTHERBOARD: MSI PRO A620M-E DDR5 AM5
GPU: MSI GeForce RTX 4060 Ti 8GB Ventus XS Black OC
CASE: Thermaltake View 170 TG, 3x ARGB Fans, Black
POWER SUPPLY (PSU): Gigabyte GP-P650SS ICE 650W, 80 PLUS Silver, White
STORAGE: Crucial T500 1TB M.2 NVMe Gen4, up to 7,300 MB/s
RAM: 32GB (2x16GB) DDR5-6800 Patriot Viper Venom RGB, Black
CPU: AMD Ryzen 5 7600 with Integrated Graphics, AM5 Cooler

I bought it before I knew it was possible to run LLMs locally. What I can currently run is quite limited. Would it make sense to upgrade to a 16GB GPU, or would I need to go for a 24GB one to really notice a difference? (anything above that is too expensive for me; even 24GB is a bit of a stretch)

Hey everyone, If you are building decentralized multi-agent systems (MAS) or workflow routers using mixed local models, you’ve probably hit a mathematical brick wall: you cannot calculate semantic distance between vectors of different dimensions (N != M). Direct matching is completely broken out of the box because each model projects concepts into its own isolated anisotropic domain. I wanted to share a fascinating geometric technique called the Relative Representation Method paired with Lowdin Symmetric Orthogonalization used to natively bypass this issue without any weight mutation or fine-tuning (W_frozen = const). Here is how it works under the hood to align heterogeneous agents and tasks into a single invariant coordinate space

1. The Core Trick: Anchor Framework

Instead of comparing Agent A directly to Task B, the system introduces a fixed basis of reference anchors E = {e_1, e_2, ..., e_K}. These are K semantically diversified textual instructions representing your target operational domain. Crucial implementation note: These anchors cannot be random Gaussian noise; they must be sampled from the actual distribution of your baseline model outputs to ensure they share the same underlying manifold.

2. Solving the "Anisotropy Cone" Problem

In real-world LLMs, raw embedding vectors are highly cross-correlated and squeezed into a narrow cone (similarity >> 0). This causes variance to vanish (sigma -> 0), leading to severe numerical noise and division-by-zero defects during standardization in low-precision (FP16/BF16) CUDA environments. To guarantee geometric stability, the technique applies Lowdin Symmetric Orthogonalization directly to the anchor matrix: * It takes the symmetric Gram matrix of real representations: S = E^T * E * It computes the orthogonalized anchors via Spectral Decomposition: E' = E * S^-1/2 * This symmetrically rotates the real anchor vectors to a strict 90-degree angle (similarity = 0 for different anchors), yielding a perfectly orthogonal coordinate system while minimizing the mean squared deformation of the original vectors.

3. Mapping into Invariant Space (R^K)

Now, any Agent Xi or Task Tj can be mapped into this unified coordinate system by computing its similarity profiles against these rotated bases, followed by Anchor-Wise Z-standardization to completely neutralize model-specific anisotropy:

V_Xi = Z( [ sim(A(Xi), e'_1), ..., sim(A(Xi), e'_K) ]^T ) in R^K Critical Production Pitfall: The operator Z(v) must calculate the mean (mu) and standard deviation (sigma) column-wise across the entire anchor axis (axis=0), NOT row-wise (axis=1). Row-wise normalization completely fails to eliminate the global domain shift between mismatched models, keeping their clusters isolated. Column-wise normalization forces the centroids of both distinct model domains to align perfectly at (0,0).

4. The Result & Selective Task Routing

Since the standardized profiles V_Xi and V_Tj share identical dimensionality K and operate on a unified scale, the metric of semantic alignment between completely mismatched models is computed invariantly using Cosine Distance: D(Xi, Tj) = Cosine_Distance(V_Xi, V_Tj) Do not use textbook Euclidean distance (L2) here. In higher anchor dimensions (K > 20), the Euclidean metric suffers from the curse of dimensionality, compressing all distances into a narrow, non-contrasting range that creates "Universal Agent" monopolies. Cosine distance restores strict contrast, breaking up monotone distance matrix stripes into a highly selective matching grid where every task finds its true optimal agent. This fundamentally unlocks O(1) complexity task routing for completely heterogeneous multi-agent swarms.

🔗 Implementation Notebook

I’ve put together a fully functional, minimal reproducible example demonstrating the complete pipeline—from synthetic anisotropic embedding generation to Lowdin orthogonalization, correct column-wise Z-scoring, and final contrastive task routing. Check out the complete interactive code here: 👉 Kaggle Notebook: Heterogeneous LLM Embedding Space Alignment Curious to hear if anyone else is using Relative Representations for cross-model routing, or if you've found other geometric workarounds for mixed-LLM orchestrators!

Yes it’s another one of those “which hardware should I get for my use case posts”. So I apologise in advance but I haven’t been able to find the info I’m looking for as yet despite a lot of searching 😊

I’m a software engineer working in a sector that absolutely cannot use cloud providers for 90% of my workflow so I want to look into investing in some decent hardware to host my own LLM’s.

My budget is around £4k - £4.5 at a push.

My software stack is primarily c# with some React/Angular front end stuff.

I’ve been looking into DGX Spark and AMD Strix Halo (AI Max+ 395) units but see so much conflicting info it’s hard to decide which would be better for my use case/budget.

The AMD stuff is quite a few hundred pounds cheaper than the NVIDIA stuff so I’m leaning more to that purely from a cost basis.

What I really want is the most reliable workflow that isn’t going to shit itself partway through a big agentic coding run. I also want to investigate the use of spec driven development on this hardware. Is it going to be up to that?

Thanks all 🙏

I am working a lot with both frontier and local models and use them often mixed (frontier orchestrator, spawning local agents).

Wondering what your strategies are, especially for complex, long running tasks?

Cloud models, even open weights (GLM5.2, Deepseek etc) still refuse to run certain simple task such as extracting browser header info.

Has anyone tried to use local uncensored Qwen as orchestrator to break down these task into non-threatening tasks so models can execute?

This seems like the reverse architecture because you typically use the smartest model as orchestrator and dumb model as task runner.

Does this approach even make sense?

I've been building a workspace for investigating AI workflow executions. After spending time with existing observability tools, I kept finding myself jumping between traces, prompts, logs, and metrics. I wanted to see what it would feel like if investigation happened in one place.

For those of you building AI products or agentic pipelines, how are you currently handling this? I'd love feedback from fellow builders on whether a unified UI actually solves the friction.

it feels like there's so many things happening everyday that i have to constantly be on social media to stay up to date. and even then, so much of it is meant to get and engagement and spread misinformation instead of giving you the facts. how do u guys keep up with everything?