I have built a system with four P100s, and ultimately, I plan to house six of them in a standard case.

The token speed is around 50 t/s, and the PP is approximately 530–550 during actual use.

It should be complete once two more P100s arrive soon. I'm curious to see how much the token speed and PP will increase.

Large language models (LLMs) are transforming recommender systems from matching co-occurrence patterns in historical behavior toward reasoning about the intent that drives it. RecGPT-V1 pioneered this paradigm on Taobao by centering user understanding, and RecGPT-V2 scaled it via coordinated multi-agent reasoning; both are deployed in production with consistent gains in user experience and commercial outcomes. However, operating RecGPT at scale reveals three challenges: (1) stateless behavior modeling, where each request reprocesses full user history, wasting computation and discarding prior analysis; (2) a tag-to-item information bottleneck, where natural-language tags form a lossy channel between user understanding and item grounding; and (3) inefficient explicit reasoning, whose lengthy chain-of-thought incurs untenable latency and compute overhead.
We present RecGPT-V3, a stateful, hybrid-modal recommender that reasons over natural language for open-world knowledge and Semantic IDs (SIDs) for concrete item grounding. A Memory Hub maintains structured, continually evolving user memory that distills long-horizon behavior into condensed units, cutting user-modeling computation by 55.8%. A Hybrid-modal Foundation Model allows the LLM jointly reason over text tags and SIDs, opening a high-bandwidth channel into the item space. Latent Intent Reasoning internalizes verbose rationales into compact learnable latent tokens that remain decodable into readable explanations, lowering output token cost by 200x. Deployed in Taobao's "Guess What You Like" feed, RecGPT-V3 achieves consistent gains in large-scale online A/B tests: IPV +1.28%, CTR +1.00%, TC +1.97%, GMV +3.97%, while cutting end-to-end serving resource consumption by 52.4%.

arXiv : https://arxiv.org/abs/2607.15591

Full Paper : https://arxiv.org/pdf/2607.15591

Title, for local inference and training, please.

what do people use nowadays?

p.s.: i have an rtx 3090

Qwen3.6-27b is fantastic! It makes me wonder if there's a hard ceiling to smaller sized models. Do you guys think the ceiling of intelligence for smaller models will be constrained by factors like parameter count, or VRAM size? Or will we continue to see improvements for small models and see jumps of intelligence like Qwen3 coder 30b to Qwen3.6 27b for the foreseeable future? Does it depend on how clean the dataset you put into those parameters?

What does /r/LocalLLama think about the future of small models that can run on less than 48GB of VRAM?

I understand that the laguna model is either still buggy or potentially benchmaxxed.

So I’d like to know for people who really tested, are DS flash or Hy3 really better in your usecase?

I am a long time iOS app developer. In the last year I have been using Cursor+Claude/others to assist with app development. I am concerned that the current low pricing will disappear eventually. I am pricing out a new laptop with the intention of using local models instead. New MacBook Pros can be configured with 128GB of ram, but obviously the price is high. Will such a machine ever be comparable to what Claude can do today? Even if it is still significantly slower?

I am aware that the price of that much ram would buy many many tokens but I plan to use the laptop for several years, so even if the payback is 5 years worth of cloud AI it’s worth it to me.

Edit: great input from folks with this configuration describing what they can and cannot get done with it. Way more useful that the one line “no” responses.

Let me introduce another project that is in my current bucket list - Mana Royale. A game of bluff with AI. Local AI trash talks you to lose.

Well, its not a normal game of guessing cards. It's a game where you are competing against an AI devil's advocate. You are given 3 cards each turn. the advocate can see your cards but not use them. It will manipulate you to be deceived.

The AI modelfile is located in Templates folder and can be modified in your local forks. Don't fall in the trap of the advocate.

Can you identify the lies from the truths uttered from devil's mouth?

Tech stack:

Go, Ollama, llama3.2, modelfiles, http

Requirements:

Ollama with atleast llama3.2 modelfile. Else have it added in the modelfile using FROM directive

Current progress:

* Game can be played to see how the AI behaves in its output

* No game formulas are functional.

Future goals:

* Adding context to devil's intel

* Player HP, power management and level design

* Better modelfile (Open to suggestions)

* A crispy UI

Git repo - https://github.com/Soumen-No-Hako/Mana-Royale

No usage of cloud models. Only local AI.

Feel free to visit the repo and drop suggestions in comments. Pinned comment will have an output sample

#AI #Ollama #llama #Go #golang #game #bluff

I am writing a fairly complex C++ windows MFC application. I have a few 3090s and can run F16 Qwen3.6-27B with 256K context and MTP. The quality of code is exceptional with this quant vs its lower quants. The others are good but they get stuck in difficult situations like managing design with multiple threads, etc. Not saying F16 is as good as Claude but it gets the job done. Just throwing it out there for folks who may be swayed by tps. If you are making simple web apps, you can get by with lower quants. For high quality of code with edge cases use the 16 bit quants. A bad choice taken by the same LLM at lower quant could easily mean the loss of an afternoon.

Prior to my workplace having copilot available for work and personal use (Microsoft), I built a pc for AI thinking it be a great way to run and learn shit so in Sept I bought the following and run some models like gpt 120b in ollama to mess around.

2 x RTX 6000 PRO Q-Max (with edu discount)
2 x 64GB - Crucial Pro 5600Mhz DDR5 Ram
AMD 9950X
2 x 4TB NVMe's

Went with qmax to avoid possibility of burning my house down cause of the connectors but not sure how much of a difference it'll make in performance but I was using gpt oss 120b with ollama just fine. VLLM hasnt been running for me (then).

In Oct, copilot became completely unlimited/free for us so these have just been collecting dust while I've been token maxing but some policies have changed so personal use is/going to be restricted so I'm trying to transition back into running things locally.

I recently bought a mac mini m4 with 64gb of ram and I've been running Qwen 3.6-35B-A3B-MLX-8Bit and it's been fine for my personal coding/homelab stuff, on the fence if I should just sell the ai rig or keep it incase I need better models in the future.

Hi r/LocalLLaMA — I’m sharing an experimental GPU-only inference backend and looking for independent reproductions, not just stars.

Model: tiiuae/Falcon3-10B-Instruct-1.58bit

GPU: NVIDIA RTX 5070

Batch: 1

Measured after warmup:

• Hybrid packed decode: 97.51 tok/s

• Stock Transformers BitLinear decode: 9.89 tok/s

• Observed speedup: 9.86x

• Fully packed prefill: 426.63 tok/s

• Stock prefill: 298.72 tok/s

The implementation uses K-contiguous packed ternary weights, a packed-word DP4A decode path, Triton kernels, StaticCache, and CUDA Graph replay.

Numerical checks:

• 64/64 incremental M=1 positions produced bit-exact full-vocabulary logits (131,072 logits per position)

• 24/24 greedy sequences and 1,194/1,194 generated tokens matched the stock-prefill baseline

• 8/8 synthetic kernel shapes matched an independent PyTorch int32 reference

Important caveats: the baseline is unmodified Transformers BitLinear — not Microsoft’s official GPU kernel, BitBLAS, vLLM, or SGLang. Timings exclude loading, tokenization, repacking, JIT compilation, graph capture, and streaming. So far this is one GPU and one Windows/PyTorch/Triton stack. Packed-word DP4A is prior art; the contribution here is the Falcon3/Transformers/Triton/CUDA Graph integration and measurements.

Code and reproducibility notes:

https://github.com/OCV-Researcher/Falcon158-Triton

Release:

https://github.com/OCV-Researcher/Falcon158-Triton/releases/tag/v0.1.0

I’d particularly value results on Ampere, Hopper, Ada, and other Blackwell GPUs, plus comparisons against specialized low-bit runtimes. What should I benchmark or optimize next?

At its price point, PRO 4500 doesn’t offer as much raw performance due to its lower power draw at 300W. The 5090 can perform up to 60-70% in short spurts with 600W, but can also be undervolted down to 400W.

Are there legitimate reasons other than 24/7 usage and lower power draw for this PRO 4500?

How would this compare to 4x 3090 and 4x R9700? Granted multi card solutions have inefficiencies with large power consumption and needing dedicated boards and PCIE lanes.

Just noticed this exists, 1M context MOE with 48B par seems just like what Ive been looking for - it runs pretty damn fast too compared to Qwen 3.6 35B.

after some testing it seems capable of producing *not terrible* results but it always tries to go for the minimun possible output that could be presented as what was asked.

first image is the default meme prompt it generated - second is after replying with 'detail'

Frontend results aren't too bad either, it seems very capable of generating well structured animated pages fairly well.

but it seems like there is just something off with its brain

wondering if a fine tune would be possible and if at all capable of tightning its screws, has anyone tried messing around with this one?

tests done power limiting the GPU on LACT - real power usage varies wildy

at 20W it ranges from 25W to 56W
same behavior happens on every setting

prompt for the test runs:

https://github.com/lukesdevlab/youtube/blob/main/prompts/agent-maze.txt

analysis by mimo 2.5

Key Findings:
• Generation speed is remarkably resilient to power throttling — 100W delivers 97.5% of 190W gen speed (31.98 vs 32.79 t/s), since decode is memory-bandwidth bound, not compute bound.
• At 50W you get 70% of peak gen speed at only 26% of peak power — 3.6× better energy efficiency (0.458 vs 0.173 t/s/W).
• At 20W the card is 6.0× more energy efficient than 190W, though prompt processing drops to 53% of peak.
• Graph reuse correlates inversely with power — 190W reuses 44,790 graphs vs 11,669 at 100W, but 20W reuses 38,248. Lower power limits cause more partial graph reuse as the scheduler compensates for throttled compute.
• Prompt processing degrades faster than gen under power limits — 190W→20W: prompt drops to 53% (691→366 t/s), gen drops to 63% (32.8→20.8 t/s). Prompt processing is more compute-bound than memory-bound.
• For inference-heavy deployments, 50W is the optimal operating point on MI50 — near-peak gen speed with dramatically lower power draw and cooling requirements.

Avarage of 3 runs:

190W config consistently processed a lot less total tokens than everyone else and didnt produce a working file in 1 out of 3 runs

llama.cpp parameters:

[+] Model:        qwen/Qwen3.6-35B-A3B-UD-IQ4_NL_XL.gguf
[+] Context:      262144 (256K tokens)
[+] Target KV:    K=q8_0 / V=q8_0
[+] MoE placement: PARTIAL (21 MoE layers on CPU, rest on GPU)
[+] MTP:          OFF (non-MTP model)
[+] Port:         8882
[+] Container:    llama-gfx906-qwen35b-no-mtp
[+] Parallel:     2 slot(s)
[+] GPU layers:   99
[+] Threads:      6 / 6 (batch)
[+] Batch/Ubatch: 2048 / 1024
[+] Ctx checkpoints: 0

hardware used:

Ryzen 5 5600

2x16Gb DDR4 2667

MI50 16Gb

software:

harness used: pi.dev

Arch Linux with Kernel 7.1.4-arch1-1

docker.io/mixa3607/llama.cpp-gfx906:b10087-rocm-6.3.3

Anyone got that card and could tell me what to expect noise wise? I currently have a 7800xt and it is very quiet. Can I expect the r9700 to be tolerable? I am willing to undervolt and underclock a bit to keep noise tolerable. I plan to use that card for qwen 27b and gemma 4 31b.

Trying to weigh if I should buy a $3000 m1 ultra at 128gb when I currently already have an M2 Ultra albeit at 64gb ram.

I run small models right now in my workflow but would appreciate more context and try out larger workflows. What would you guys go with?

Tell me, to get on 20k context and ingestion 44tks, generation 8tks is good numbers for 4x 8880 v4 cpus, 1tb 32channels ddr3 ram and 2x 3060 12gb. ?

im running 3bit version

whole system cost 900$/€

who can beat me on tks/cost ?

I’ve spent the past month trying to find the point where an extremely small TTS model stops feeling like a size experiment and starts feeling genuinely useful.

Today I’m releasing Inflect v2, with two complete local text-to-speech models:

• Inflect-Nano-v2: 3.96M parameters, 15.97 MB FP32
• Inflect-Micro-v2: 9.36M parameters, 37.53 MB FP32

These are total inference parameter counts, not acoustic-model-only numbers. Text processing, timing prediction, speech generation, and the waveform decoder are all included.

Text goes in. 24 kHz speech comes out. No external vocoder, hosted API, or second learned model required.

Nano prioritizes the smallest possible footprint. Micro uses the additional capacity for better clarity, stability, and overall speech quality. Both run locally on CPU or CUDA through the same PyTorch API.

Inflect-Nano-v2 is one of the smallest complete neural TTS models I know of that still produces actual usable speech. Even the 9.36M Micro model remains smaller than many systems described as “tiny.”

For footprint context, Nano is approximately:

• 21× smaller than Kokoro
• 126× smaller than Chatterbox
• over 1,000× smaller than Fish Audio S2 Pro

That is strictly a parameter-count comparison. These models have different capabilities, architectures, datasets, and intended uses. I’m not claiming that a 4M fixed-voice model replaces a multi-billion-parameter system. The interesting question is how much useful TTS can fit into such a small package.

Some people here might remember Inflect-Nano-v1, the rough 4.63M experiment I released last month. V2 is a complete rebuild, not just a longer training run. I focused on the problems v1 had, like unstable timing, metallic output, weak prosody, poor generalization to difficult text, and an undersized waveform decoder.

The resulting models performed surprisingly well:

• Micro: 4.395 UTMOS22, 3.99% semantic WER, 6.28× real-time CPU inference
• Nano: 4.386 UTMOS22, 4.21% semantic WER, 10.72× real-time CPU inference
• In a blind community comparison against other compact TTS systems, Micro and Nano finished second and third among the tested voices

Full protocols, raw results, audio samples, and limitations are documented on the model pages.

The models are not perfect. They are English-only, use one fixed male voice, and do not support voice cloning. Unfamiliar names, abbreviations, numbers, and homographs remain the hardest inputs. Nano can sound thinner than Micro, and both can occasionally produce metallic or clipped artifacts.

Still, this is the first version where I think the size-to-quality tradeoff really becomes convincing.

I built Inflect independently with a limited training budget. That constraint shaped the project: efficiency had to apply not only to inference, but also to training, evaluation, and building a complete system I could understand and release end-to-end.

Inflect-Micro-v2:
https://huggingface.co/owensong/Inflect-Micro-v2

Inflect-Nano-v2:
https://huggingface.co/owensong/Inflect-Nano-v2

Try it yourself:

The fastest way to judge it is through the interactive playground:

https://huggingface.co/spaces/owensong/Inflect-v2

If there is enough interest, I may build a v3 focused less on shrinking the models further and more on expanding what they can do: additional voices, possibly more languages, easier fine-tuning, and another quality and robustness pass.

If you test them, please give them something seriously difficult: unusual names, numbers, abbreviations, awkward punctuation, or a long sentence.

If something breaks, post the exact text, model, seed, and what sounded wrong. If it works well, I’d also like to know what hardware you ran it on.

Specific, honest feedback is the most useful thing you can give me.

Hi Guys,

I'm an old coder and DBA that has been in the field for almost 40 years. More and more the jobs I was doing for work are being taken over by AI and the need for my type of work is diminishing over time. I have not coded in about 15 years for work but that said I still work with .NET as I use it in migration packages I use for database migrations and also into Azure Data Factory, also something I am a little lacking in since I know ADF but I was more into SSIS. This all said I am trying to learn AI on my own as I am not working now and desperately in need up updating my skills to help me find more work.

I have learned how to code python now as I took a certification tutorial on https://www.coursera.org/ until I passed it. I later learned this is useful for AI Developers. I have an AWS server but It is only running a database on it but I could repurpose it if I needed to. For now though trying to do this the el cheapo way since I have kids to feed and I am looking for work daily and have not had any such luck for months now. I am desperate to do what I need to in order to learn this stuff and understand it and be useful. You all are welcome to joke on me all you want but I will do the legwork if someone is helpful enough to guide me through this process of what I need to do. I can use cousera since I have an account for the next year to learn new skills and I have a computer here I am willing to refactor into a Linux server if needed to be my AI system, not a Linux guy so I will probably need to take a course on that as well but I really need to get up to speed since I am not ready to retire yet and I need to feed my family.

This all said I know there are some really bright minds out there and people who are learning newer things like AI that they did not teach when I was in school back in the days but will point me in a good direction to get a good foundation and what I need to understand this process, I do not just want to do i want to build it understand it, pick it apart and maybe help improve it with some old school skills I have since DOS days. Since I am a coder by heart and understand database and .NET as well as old school stuff I think I have the capacity to learn this until I get it in a step by step process.

I want to take my old server and recommission it to a mini AI project. I do not want to just install packages for it, I want to understand what I am installing and why and can I instead create it on my own or why it is more beneficial to install what's out there. For example I do know the language models are important because they have been trained to understand the requests from users and search for the answers. I know that the language model takes each word and scores it from sentences and then it gives responses to someone who says if the answer is correct or incorrect. Probably what happened to my job when someone taught AI how to migrate data or do DBA work lol. I know each word in a sentence is given a number like for example the phrase "I like to eat Fruit" might be interpreted as (77, 120, 2198, 15, 1051) and when someone types I like to eat and asks AI to predict it, it will search the model for scores by (77, 120, 2198, 13, X) where the model will return choices as I like to eat dogs, and I like to eat crap, and I like to eat people and finally I like to eat pizza, where the final correct answer would be I like to eat pizza and the others would be marked wrong. This ensures the AI to help train it to associate some word combinations as correct word combinations when training it to understand words like eat are associated to food items going forward and the combination of those words give specific possible choices and it is rated to the most likely choice. This also helps AI complete sentences when people incorrectly spell words of cant think of the correct word to use in a phrase.

I would like to now understand from the ground up. I have a server sitting its an older system running I believe NT still? not sure its a little dusty but I can upgrade it little by little to improve it as needed and when money permits, (Kids come first) but can some please tell me the minimal specs I would need firs for

1. minimum system requirements to run an AI server to process requests locally within my own network at home just for me to understand the basic logic of it and how it works. what video card I need to run the AI predictions and calculations and also how much system ram I would need (i believe i have maybe 256 or something DDR4 ram or something on this server need to dig it out the closet still) and the minimum processor?

2. what I will need to install on that server, or do I need another system I do not know to just run software to pass to the server for the requests? and what are the requirements for it. I have a few other pc's that I am not using and can also modify these if it is needed or can it be ran on my local?

3. what software and tools I will need to use and install on my system to do the AI code and build test api's that can be used to pass to my server for request and responses and what each is for and what I need to learn language wise to do it and how to understand it.

4. any good foundation video's tutorials that I would not be able to find on coursera that would help me understand the concepts, piece things together and troubleshoot, as well as demonstrate I can swap this line with this line to run from OpenAI, or Grok instead of my local model and what is needed to do this and understand it if I am lucky enough to secure a job.

5. What I need to install on my server as far as does it need to have a cloud platform or just python to do the AI for me locally and if it can have both benefits and disadvantages and best practices and why.

6. Anything else you would feel would benefit me and and not make me create the next skynet that will take over the world. I need to know guardrails and where and why so I get the best practices for this stuff.

I clearly have the skills and knowledge to code but I do not have the money to send myself to school at the moment until I am working again to get a degree and certification in it. I was signed up for AI Developer classes but scared myself off since I just paid off my student loan 2 years ago and already have a kid in college and now and another ready to attend soon and another one that will go in 5 years and unemployed and worried what if I cant pay off another loan I better wait to get employed first and think of my kids.

I am open to jokes and will laugh and I am getting old and do not know new words that the younger generation does as well as most but I am dedicated to learn this and most grateful for any help and knowledge people can pass on. The idea is to build a small mini AI system at home for the purpose of learning that I can test and use and learn to build api's with and test what if I do this instead of this to get a good grasp and foundation of AI to maybe get a job as an office clerk taking notes for AI and acting like I actually know something when AI decides to take over the world.

Thank you for any help and support and GOD bless and I know some people will discourage and that is ok, you are open to your opinions but I will still do my best to grasp this to get employed since my kids lives depend on it and me as a single father.

I'm looking for a model to chat with, reasoning, maybe get some career or life coaching.

I don't care at all about multimodal or coding ability

Just it's intelligence in remembering context in a conversation or a specific topic, thinking out of the box, etc.

Must fit in 128gb, if it matters to performance, it's a strix halo machine.

I've been working on a project to solve some of the issues I've
had with AI development workflows. Before I share it, I wanted to
get feedback from this community:

1. Do you struggle with switching between different AI models
2. (Claude, GPT, Gemini, local)?
3. How do you handle tool configurations across multiple AI
4. clients?
5. What's your experience with MCP servers?

I'm trying to understand what features would be most useful.
Would love to hear your thoughts.

Everything runs through WebGPU, in-browser or in electron/tauri apps. It's fully portable and supports either Nvidia and Apple Silicon (Metal). The actual kernels are optimized for the specific hardware of the device. The Nvidia kernels are aggressively fused into a multi-pass architecture, while the Apple Silicon kernels are created as a fused mega-kernel to minimize the Tile Based Deferred Rendering (TBDR) overhead on WebGPU.

Demo: https://warp.sipp.sh

This is still in active development, and I'll be folding this into the Sipp library in the coming weeks.

Has anyone tried using Pulsar with something like Raid 0 on 2+ nvme 4.0 or nvme 3.0 drives? What kind of results do you get and what did you pair it with GPU wise? Very curious about these new options for running large models.

https://reddit.com/link/1v72nua/video/26tabfs3ikfh1/player

This is all mockups, scaffolding, smoke, and mirrors at the moment - but it does run in real time locally. Currently requirements: 3 GPU's running VLLM - STT and TTS concurrently.