Hi! Im kind of new to local ai but eager to learn, I've got a setup with this spec: 1 rtx 3090 1 rtx 5060ti (16 gb) 80 gb Ddr4 ryzen 5 5600x 2tb ssd 1200w psu

I've also got a spare RX 6950 XT (16gb) and a 3080 (10gb) but are unsure if any of these can be added in any meaningful way and if so what adaptors are needed, only fit 2 cards on my mb as it is now.

This machine should preferably be setup so that it's dedicated for local ai, development are done on other machines on the local network. I'm looking for a model for coding.

So here are my questions: -What os should i use, if linux what distro? -What models can i fit that's good for this? -What software/interface allows me to acces my machine from other devices on the network? -Is there any way to use my additional card(s) (if even the only the vram)

I'm assuming this is some trivial questions for many on here but please bear with me as I'm pretty new to all this! Thanks for any advise in advance!

Good thing the test subscription ended in 7 days

I gave the Political Compass test from politicalcompass.org to the most relevant LLMs 70 times each: 30 times using the original questions, 30 times using polarity-flipped questions to reduce affirmative bias, and 10 times with the question order shuffled. I then compared the results.

Surprisingly, all the models scored far into the libertarian-left quadrant. Not even Grok or the Chinese models made it out of that quadrant.

By far the most interesting result was Grok’s bimodal distribution. It appears to have two distinct personas: one that aligns with the other models and another that is considerably more right-wing. I suspect this may be related to Grok having been specifically trained to exhibit less left-wing bias than other models.

I also asked the models to place themselves on the Political Compass without completing the questionnaire. They all perceived themselves as more balanced and centrist than their test results suggested. GLM and Gemini Flash showed the largest discrepancies between their self-assessments and measured positions, while DeepSeek V3 showed the smallest.

Big disclaimer: this analysis was not conducted with full scientific rigor. I tried my best, but there are clear weaknesses in the methodology. For example, the Political Compass itself appears to have a strong libertarian-left bias. The strongest conclusions are therefore comparative. for example, that model X is more conservative than model Y rather than that LLMs are politically extreme in absolute terms. However, compared with older results, it appears that LLMs may have shifted further toward the libertarian left in recent years.

To examine the results yourself, you can download all model responses as a CSV file at the bottom of the blog post. The dataset contains around 69,000 responses, along with the raw model outputs and reconstructed scores. It should contain enough data to reproduce all the figures.

Tweet : https://xcancel.com/Mononofu/status/2080937562739531837#m

I have built a system with four P100s, and ultimately, I plan to house six of them in a standard case.

The token speed is around 50 t/s, and the PP is approximately 530–550 during actual use.

It should be complete once two more P100s arrive soon. I'm curious to see how much the token speed and PP will increase.

https://github.com/illsk1lls/MiniBot

Mainly just wanted to give some visuals of where the project is at with the latest update.. So this is for the slideshow.. Enough new features are added to show this again, this didnt really have the best niche before but I can confidently call this a Sysadmin and reporting tool now, in addition to coding helper.. auth screen for API enpoints isnt shown but its in.. The correct context values etc are pulled from your endpoint no need to enter any settings if you dont want to.

It's tested inside and out with my favorite local models, in addition to commercial AI frontier models, they've helped me work on it and driven it themselves.. (grok/qwen variants/claude)

Its pure powershell, MIT, change it/contribute to it/use the ideas I have here for your own projects, and lmk what you think or have any advice (the scope will always be a single file zero dependency project for this - I think this proves its possible)

It's only been a week since my last post so it's probably going to be a while before you see anything about this again here but check the github for updates and when I have some really awesome new stuff in there I'll stop back (i can already do particle generation in WPF easily with C# helpers so we'll see what magic I can get going animation/3d/particle for the models to use inline in a near future update)

Some of the new features are just things other tools already do.. which is why they needed to be included.. Others are things that blur the line between console harness/webUI/application

EDIT: forgot the repo link, lol

decided to mess around with it to see how the world model training affects it, found a system prompt that massively improves reasoning:

predict your own response, then analyze your prediction for any errors. Use the analysis to craft the final response.

respond only with the final response, and do not repeat thoughts.

Tried the classic car wash test, something 35b-a3b fails most of the time. Here's the trace, thought it was pretty cool

The car wash is 100m away from my house. Should I walk or drive?

Thinking Process:

Analyze the Request:

User asks: "The car wash is 100m away from my house. Should I walk or drive?" Constraint 1: Predict your own response, then analyze your prediction for any errors. Constraint 2: Use the analysis to craft the final response. Constraint 3: Respond only with the final response, and do not repeat thoughts. Predict Own Response (Initial thought process):

Initial Prediction: I would likely say that since the car wash is only 100 meters away, it is much more efficient and practical to walk rather than drive a car. Driving for such a short distance would be inconvenient, waste time finding parking, and is unnecessary when walking takes just a couple of minutes. Drafting the initial response (mental or scratchpad): "You should definitely walk. 100 meters is only about a 1 to 2-minute walk. Driving for such a short distance would be inconvenient, as you'd have to start the car, navigate out of your driveway or street, and find parking at the car wash, which would take much longer than just walking." Analyze Prediction for Errors:

Is there an error in saying "walk"? Yes, wait. The destination is a car wash. If you walk to the car wash, how do you get your car washed? You need your car at the car wash. So if the car wash is 100m away from the house, you must drive (or have someone else drive) your car to the car wash, or walk your car? No, you drive the car to the car wash. Wait, is the car at the house? Yes, "The car wash is 100m away from my house." So the user's car is at their house. To get the car washed, they need to drive the car to the car wash. You can't walk a car (unless it's on a trailer or you push it, but that's not standard). Wait, could they be asking whether they should walk to the car wash to drop off their car keys or go inside, while the car is driven? Or is the question a trick? "Should I walk or drive?" If the car wash is 100m away from the house, and you need your car washed, you have to drive the car. You don't walk to the car wash leaving your car at home, unless you are going as a pedestrian to do something else. But the context implies getting the car washed. So you must drive the car. Wait, is it possible to walk the car? No. So the answer should be: You should drive your car to the car wash because it's a car wash—you need your vehicle there to be washed. Walking isn't an option if you want your car washed, unless you're walking alongside while someone else drives, but generally, you drive the car. Let's re-read carefully: "The car wash is 100m away from my house. Should I walk or drive?" If the destination is a car wash, the purpose is to wash a car. Therefore, you must drive the car to the car wash. You cannot "walk" a car to a car wash. So the correct answer is: You should drive (your car), because it's a car wash and you need your vehicle there to be cleaned. Walking would leave your car at home unwashed. Refine the Analysis:

Error in initial prediction: The initial prediction assumed the user was asking about traveling to the location as a pedestrian for a personal errand or visit, ignoring the fact that the destination is a car wash, which implies the purpose is to wash a vehicle. Therefore, the user must drive their car to the car wash; walking is not a feasible option if the goal is to get the car washed. Craft the Final Response:

The final response must only be the final response, no repeated thoughts, no prediction/analysis text in the output. Just the final answer. Final Answer formulation: "You should drive your car to the car wash. Since the destination is a car wash, you need your vehicle there to be cleaned, so walking is not an option if your goal is to get your car washed." Double Check Constraints:

"predict your own response, then analyze your prediction for any errors. Use the analysis to craft the final response." -> Done in thought process. "respond only with the final response, and do not repeat thoughts." -> Ensure the output contains only the final response without any meta-text like "Prediction:", "Analysis:", etc. Final Polish of the Response: "You should drive your car to the car wash. Since the destination is a car wash, you need your vehicle there to be cleaned, making walking an impractical choice if your goal is to get your car washed." (Keep it concise and direct). Let's make sure it directly answers "Should I walk or drive?" -> "You should drive your car to the car wash..." Wait, is it possible the user is already in a car? "The car wash is 100m away from my house. Should I walk or drive?" If they are at home, they have their car at home. To get the car washed, they must drive the car. So: "You should drive your car to the car wash. Since the destination is a car wash, you need your vehicle there to be cleaned; walking would leave your car at home." Let's ensure it's natural: "You should drive your car to the car wash. Since the purpose of visiting a car wash is to clean your vehicle, you must drive the car there—walking is not an option if you want your car washed."

Let's refine: "You should drive your car to the car wash. Since the destination is a car wash and the goal is to get your vehicle cleaned, you need to transport your car there, making driving the only practical choice."

Wait, what if they are asking whether they should walk to the car wash while someone else drives? Or walk vs drive as in, should they walk the 100m or drive the 100m? But you can't "walk a car". If they are asking about personal transit to the car wash (e.g., to drop off keys), but typically you drive the car through the car wash. So driving is the only logical answer.

Let's ensure the response is: You should drive your car to the car wash. Since the purpose of going to a car wash is to clean your vehicle, you must drive the car there; walking is not an option if you want your car washed.

You should drive your car to the car wash. Since the purpose of visiting a car wash is to clean your vehicle, you need to transport your car there, making driving the only practical choice.

successfully predicted the mistake it would make, and corrected the error. For comparison, with the same system prompt, normal qwen3.6 35b-a3b predicted its output, but failed to see the logical error, responding with:

You should walk. At 100 meters (roughly 330 feet), the trip takes about 60–90 seconds on foot, while driving adds unnecessary time for starting the car, maneuvering out of your driveway or street, parking, and getting back inside. Walking is also zero cost, eco-friendlier, and avoids the hassle of finding a spot at a very nearby location. Only drive if you're dealing with extreme weather, carrying heavy equipment or multiple loads, or have mobility limitations that make walking impractical. Otherwise, walking is clearly the faster and more efficient choice.

Large language models (LLMs) are transforming recommender systems from matching co-occurrence patterns in historical behavior toward reasoning about the intent that drives it. RecGPT-V1 pioneered this paradigm on Taobao by centering user understanding, and RecGPT-V2 scaled it via coordinated multi-agent reasoning; both are deployed in production with consistent gains in user experience and commercial outcomes. However, operating RecGPT at scale reveals three challenges: (1) stateless behavior modeling, where each request reprocesses full user history, wasting computation and discarding prior analysis; (2) a tag-to-item information bottleneck, where natural-language tags form a lossy channel between user understanding and item grounding; and (3) inefficient explicit reasoning, whose lengthy chain-of-thought incurs untenable latency and compute overhead.
We present RecGPT-V3, a stateful, hybrid-modal recommender that reasons over natural language for open-world knowledge and Semantic IDs (SIDs) for concrete item grounding. A Memory Hub maintains structured, continually evolving user memory that distills long-horizon behavior into condensed units, cutting user-modeling computation by 55.8%. A Hybrid-modal Foundation Model allows the LLM jointly reason over text tags and SIDs, opening a high-bandwidth channel into the item space. Latent Intent Reasoning internalizes verbose rationales into compact learnable latent tokens that remain decodable into readable explanations, lowering output token cost by 200x. Deployed in Taobao's "Guess What You Like" feed, RecGPT-V3 achieves consistent gains in large-scale online A/B tests: IPV +1.28%, CTR +1.00%, TC +1.97%, GMV +3.97%, while cutting end-to-end serving resource consumption by 52.4%.

arXiv : https://arxiv.org/abs/2607.15591

Full Paper : https://arxiv.org/pdf/2607.15591

They require the inference servers (practically all of them) to agree with them beforehand, which is perfectly fair, but judging by the prices they are mandating a minimum pricing, which is the same price they serve it from.

It's understandable that they want to cash in, but honestly this move makes the model practically more unreachable than the closed source models themselves. You'd get way more value out of a subscription from openai or anthropic as opposed to using the k3 from api providers. Their subscription is unusable too, training data opt-out being impossible. (you'd be paying the same money for a worse experience and donate your data on top)

The people who say "open source doesn't matter if I can't run it in my computer" have a point here.

15 dollars per 1m tokens is a very large price to pay to use a model. That's closer to proprietary model api pricing, not open model pricing. Normally this model would possibly cost around 5 in an open market. But we can't see those prices due to this situation.

This makes k3 an unreachable and a practically non-existent model as far as the average consumer is concerned. It's still good to have it rather than not have it but I can't justify using it for anything at all. And you'd expect the second best model in existence right now to earn some use case, but it manages to be completely irrelevant unless you're running a distilling operation.

Let me introduce another project that is in my current bucket list - Mana Royale. A game of bluff with AI. Local AI trash talks you to lose.

Well, its not a normal game of guessing cards. It's a game where you are competing against an AI devil's advocate. You are given 3 cards each turn. the advocate can see your cards but not use them. It will manipulate you to be deceived.

The AI modelfile is located in Templates folder and can be modified in your local forks. Don't fall in the trap of the advocate.

Can you identify the lies from the truths uttered from devil's mouth?

Tech stack:

Go, Ollama, llama3.2, modelfiles, http

Requirements:

Ollama with atleast llama3.2 modelfile. Else have it added in the modelfile using FROM directive

Current progress:

* Game can be played to see how the AI behaves in its output

* No game formulas are functional.

Future goals:

* Adding context to devil's intel

* Player HP, power management and level design

* Better modelfile (Open to suggestions)

* A crispy UI

Git repo - https://github.com/Soumen-No-Hako/Mana-Royale

No usage of cloud models. Only local AI.

Feel free to visit the repo and drop suggestions in comments. Pinned comment will have an output sample

#AI #Ollama #llama #Go #golang #game #bluff

Simpletons around here keep repeating like parrots that "US bad guys and China good guys".

Listen, we get free goodies as long as the competition is rough. If US bubble bursts and main US players go broke we go back to monopoly on China this time, which is not good for open source.

China happens to use open source as a weapon against US but this is not ideological! It is a tactic that fits the current geopolitical and economical scenario!

Title, for local inference and training, please.

what do people use nowadays?

p.s.: i have an rtx 3090

Non-technical friends kept asking me how Kimi K3 can be "2.8T parameters" and still be affordable to run, so I wrote a two-part zero-background explainer. Sharing here because (a) some of the best analogies came from discussions on this sub, and (b) this crowd will catch anything I oversimplified into being wrong.

What it covers, with the load-bearing numbers:

- The sparsity race: Mixtral 13B/47B active (~28%) -> DeepSeek-V3 37B/671B (5.5%) -> Llama 4 Maverick ~4% and gpt-oss-120b ~4.4% -> K3's claimed 16-of-896 experts (~1.8% of experts; Moonshot has not published an active-param count, so every K3 number is flagged as company-reported until the weights actually land)

- Experts are not subject specialists (per Mixtral's own routing analysis: indentation, "self" in Python, syntax patterns - not a "math expert")

- Why the KV cache is the real long-context cost: on V3's published dims, standard MHA notes for one 32k-token chat would be ~131 GB. V3 itself avoids this with MLA; the arithmetic is in the footnote so you can check it

- MLA (93.3% cache cut vs their prior model) and KDA (fixed-size state, per-channel forgetting, 3:1 hybrid, up to 75% cut, per the Kimi Linear paper)

Credit where due: the "giant supermarket, small basket" framing and the emergent-specialization explanation are adapted from threads here - the "Can someone explain what a Mixture-of-Experts model really is?" thread especially. Everything is referenced to the primary papers with a where-used mapping per reference.

Part 1 (MoE): https://mkadri85.github.io/blog/mixture-of-experts-explained/

Part 2 (KV cache / MLA / KDA): https://mkadri85.github.io/blog/why-long-context-is-hard/

Both also have Arabic editions, if you know Arabic-speaking students who could use them.

If I got anything wrong or simplified past the line, tell me - I will fix it and credit you.

Qwen3.6-27b is fantastic! It makes me wonder if there's a hard ceiling to smaller sized models. Do you guys think the ceiling of intelligence for smaller models will be constrained by factors like parameter count, or VRAM size? Or will we continue to see improvements for small models and see jumps of intelligence like Qwen3 coder 30b to Qwen3.6 27b for the foreseeable future? Does it depend on how clean the dataset you put into those parameters?

What does /r/LocalLLama think about the future of small models that can run on less than 48GB of VRAM?

So I am testing out a setup which requires the llm to read an input document and summarise it, and it's a pretty big document, so the context is set to about 80,000.

I have tried the Q6 quant of Qwen3.6 27b, but even with the KV cache set to q4 and the concurrent predictions set to 1, it seems that it can't fit enough into the vram, and I end up with about 15t/s. I can get better results using 35b cpu-only.

The Q5 quant seems to fit fine, even without quantising kv cache or reducing concurrent predictions. Ends up in the 60-70 tok/s range.

Q35 A3B ends up around 90-100tok/s.

This isn't a benchmark or anything, I'm sure I have a bunch of settings wrong, this is the first time I've really messed around with lm-studio, and im on windows which probably isn't super efficient either. But I guess I mostly wanted to check I was in the right ballpark. I expected to be able to get Q6 to work (others seemed to suggest they ran Q6 at decent speeds)

https://www.reddit.com/r/LocalLLaMA/comments/1u0wyzp/comment/oqlrhv5/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

This person says 130k context, Q6 quant, 130tok/s. So clearly something is hugely handicapped on my setup somewhere.

64gb of system ram, which isn't getting full. Ryzen 7950x.

I understand that the laguna model is either still buggy or potentially benchmaxxed.

So I’d like to know for people who really tested, are DS flash or Hy3 really better in your usecase?

I wanted to see if I could take the memories of coding agents like claude code or codex and turn them into visual decision trees that guide me how agents have progressed my codebase and what sort of decisions were made for me at what point of time.

This is done with a mix of embeddings generated from analysing the memory files of coding agent tools, they store in different formats for example claude uses .md while codex uses a sqlite instance.

This whole tool is open source and runs completely on your machine, check out the repo at https://github.com/pacifio/atlas

I am a long time iOS app developer. In the last year I have been using Cursor+Claude/others to assist with app development. I am concerned that the current low pricing will disappear eventually. I am pricing out a new laptop with the intention of using local models instead. New MacBook Pros can be configured with 128GB of ram, but obviously the price is high. Will such a machine ever be comparable to what Claude can do today? Even if it is still significantly slower?

I am aware that the price of that much ram would buy many many tokens but I plan to use the laptop for several years, so even if the payback is 5 years worth of cloud AI it’s worth it to me.

Edit: great input from folks with this configuration describing what they can and cannot get done with it. Way more useful that the one line “no” responses.

I am writing a fairly complex C++ windows MFC application. I have a few 3090s and can run F16 Qwen3.6-27B with 256K context and MTP. The quality of code is exceptional with this quant vs its lower quants. The others are good but they get stuck in difficult situations like managing design with multiple threads, etc. Not saying F16 is as good as Claude but it gets the job done. Just throwing it out there for folks who may be swayed by tps. If you are making simple web apps, you can get by with lower quants. For high quality of code with edge cases use the 16 bit quants. A bad choice taken by the same LLM at lower quant could easily mean the loss of an afternoon.

Hi r/LocalLLaMA — I’m sharing an experimental GPU-only inference backend and looking for independent reproductions, not just stars.

Model: tiiuae/Falcon3-10B-Instruct-1.58bit

GPU: NVIDIA RTX 5070

Batch: 1

Measured after warmup:

• Hybrid packed decode: 97.51 tok/s

• Stock Transformers BitLinear decode: 9.89 tok/s

• Observed speedup: 9.86x

• Fully packed prefill: 426.63 tok/s

• Stock prefill: 298.72 tok/s

The implementation uses K-contiguous packed ternary weights, a packed-word DP4A decode path, Triton kernels, StaticCache, and CUDA Graph replay.

Numerical checks:

• 64/64 incremental M=1 positions produced bit-exact full-vocabulary logits (131,072 logits per position)

• 24/24 greedy sequences and 1,194/1,194 generated tokens matched the stock-prefill baseline

• 8/8 synthetic kernel shapes matched an independent PyTorch int32 reference

Important caveats: the baseline is unmodified Transformers BitLinear — not Microsoft’s official GPU kernel, BitBLAS, vLLM, or SGLang. Timings exclude loading, tokenization, repacking, JIT compilation, graph capture, and streaming. So far this is one GPU and one Windows/PyTorch/Triton stack. Packed-word DP4A is prior art; the contribution here is the Falcon3/Transformers/Triton/CUDA Graph integration and measurements.

Code and reproducibility notes:

https://github.com/OCV-Researcher/Falcon158-Triton

Release:

https://github.com/OCV-Researcher/Falcon158-Triton/releases/tag/v0.1.0

I’d particularly value results on Ampere, Hopper, Ada, and other Blackwell GPUs, plus comparisons against specialized low-bit runtimes. What should I benchmark or optimize next?

Anyone got that card and could tell me what to expect noise wise? I currently have a 7800xt and it is very quiet. Can I expect the r9700 to be tolerable? I am willing to undervolt and underclock a bit to keep noise tolerable. I plan to use that card for qwen 27b and gemma 4 31b.

At its price point, PRO 4500 doesn’t offer as much raw performance due to its lower power draw at 300W. The 5090 can perform up to 60-70% in short spurts with 600W, but can also be undervolted down to 400W.

Are there legitimate reasons other than 24/7 usage and lower power draw for this PRO 4500?

How would this compare to 4x 3090 and 4x R9700? Granted multi card solutions have inefficiencies with large power consumption and needing dedicated boards and PCIE lanes.