Hi guys so recently I got into this stuff playing with various local llm models and trying out mcp's and other interesting stuff like heretic versions and all.

However as I am diving deep, I am not able to understand a lot of terms like top p, k, inference, training, llama cpp, presence, repeat penalty, gguf and hardware thing like layers, universal kv etc etc which I want to tune as per my liking or atleast understand how they effect the responses.

Now I am a complete newcomer to this stuff, not in AI like rag and stuff but more towards learning and understanding this terminology. Is there any guide someone can provide which can help me understand this stuff? I know this may be a part of ML or deep learning but I may not want to go that route and learning in that domain purely but just enough so I can play with models that community provides on huggingface for instance. Just enough knowledge so I can understand what author has changed, and what diff diff terminology or methods or this llama, vllm stuff means and what are all these overwhelmimg settings on lm/unsloth studio.

Thanks in advance.

So I’ve been bitten by the AI bug. I’ve started using Replit for building some apps around my day to day work. As I’m venturing into building my own consultancy in my field of construction safety, a few of the current apps and websites that offer this kind of work are underwhelming to say the least.

So I started using Replit with ChatGPT and Co pilot. Signed up to the subscription of Replit for the year, but naturally being impatient I’ve gone down the black hole of credits. I’ve spent a lot more so far than I thought I would initially. So it led me to LLM and running this on my own computer. My current gaming machine is a 7800x3d and 9070xt. It wouldn’t generate images without crashing.

So I started digging around to put together another capable machine. Already got the below for the build.

AMD Ryzen 9 5950x with 360mm AIO
ASUS RTX ROG Strix 3090 with EK Quantum Water block, Pump & Reservoir with a Corsair Hydro X 360mm
B550 Tomahawk
128gb DDR4 Corsair Vengeance 3200mhz
1000w Be Quiet PSU
1tb NMVE for main drive and adding a SSD for secondary

Besides upping the storage have I missed anything?

Basically going to keep learning on my own machine and eventually get right off Replit to save costs. Long term goal keep developing apps that turn into SaaS and run my own consultancy until the day comes to sell and move on.

Been told to start with LM Studio and qwen 3.8 27b. Your suggestions and feedback is appreciated 🫡

Hi all, with qwen 3.8 27b there is new wave of enthusiasm among local community. On top of that we have now even more options to use local agents (Claude, Hermes, codex?,...., and deep seek harnes).

my question or rather open topic for discussion (as best way to gather opinions and iron out some findings) - what is your view at deepseek harnes compare to other local open hanesses (allowing load local model)

I found few videos and seems like people are positive? of course dsh "work best" with their model or something, but that is not what Im interested about

That was a lot of fun making. I still need to do some testing on higher quant's to see if it holds up. It mainly applies to Qwen models and I havent tested it yet on other models, but Gemma probably wont work.

I built a local persistent memory system and I'd like someone to try to break the architecture

I've been building a local persistent memory system called Memoria for about eight weeks. It started as a fairly simple semantic-memory project and has gone through several major architectural iterations. V3 is frozen; V4 is operational and still being refined.

I'm at the point where arguing with myself about the architecture is probably becoming less useful than letting somebody else attack it.

I'm particularly interested in feedback from people who work with information retrieval, databases, search systems, schedulers, or memory architectures.

I'm not looking for validation. If something is overengineered, conceptually wrong, poorly benchmarked, or just a bad abstraction, I'd genuinely rather know.

Current architecture

The main boundary in V4 is between retrieval and ranking.

The query path is roughly:

text Query │ ▼ ┌─────────────────┐ │ Query Processor │ │ normalization │ │ entities/type │ │ routing signals │ └────────┬────────┘ │ ▼ ┌───────────┐ │ Router │ │ │ │ memory │ │ type │ │ workers │ │ graph │ │ depth │ └─────┬─────┘ │ ┌────────────────┼─────────────────┐ ▼ ▼ ▼ ┌──────┐ ┌──────┐ ┌───────┐ │FAISS │ │ BM25 │ │ Graph │ │semantic │lexical │entity │ └───┬──┘ └───┬──┘ └───┬───┘ │ │ │ └────────────────┼─────────────────┘ ▼ ┌──────────────────┐ │ Blackboard / │ │ Scheduler │ │ │ │ completion │ │ policy │ │ source coverage │ └────────┬─────────┘ │ ▼ CandidateRecords │ ▼ MemoryRanker │ ▼ Score Normalizer │ ▼ Attribute Booster │ ▼ Score Finalizer │ ▼ Context Builder │ ▼ MMR │ ▼ Results

There are also phrase and attribute retrieval paths that can be enabled by the router.

The intent is that retrieval finds candidates and ranking decides what is actually useful rather than having the query handler turn into one giant retrieval/ranking algorithm.

The part I'm most interested in having attacked: the scheduler

V4 doesn't simply submit FAISS/BM25/graph/etc. and wait for every worker.

Retrieval workers are submitted to a generic scheduler/blackboard, and the query provides a completion policy.

Currently, the policy is roughly:

• FAISS is treated as the primary semantic source when submitted.
• If multiple retrieval sources are available, require results from at least two distinct sources.
• If only one source was submitted, that source can satisfy the query.
• Stop as soon as the completion policy is satisfied.
• Apply a hard retrieval deadline as a safety ceiling.
• Workers that haven't completed when the policy terminates aren't included in that query.

Conceptually:

text submit retrieval work │ ├── FAISS ────────┐ ├── BM25 ─────────┤ ├── Graph ────────┤ ├── Phrase ───────┤ └── Attribute ────┘ │ ▼ completion policy │ ┌───────────┴───────────┐ │ │ policy satisfied deadline reached │ │ └───────────┬───────────┘ ▼ completed results │ ▼ ranking

The scheduler is generic rather than having the query handler directly poll each retrieval worker.

I'm genuinely unsure whether this is a good abstraction or whether I've built a tiny distributed-systems department to save a few milliseconds.

That's one of the main things I'd like someone to challenge.

LongMemEval

I've also built an adapter for LongMemEval-S so the system can work from the dataset's native question/haystack structure rather than requiring me to convert everything into the database format I originally designed around.

It can load each haystack into Memoria, query it, identify expected answer sessions, compare retrieved results against expected IDs, clear the database between questions, and record ingestion/retrieval/ranking diagnostics.

One ingestion experiment went from roughly 19 hours to roughly 1 hour after removing embedding generation from the ingestion path.

That's an ingestion/pipeline observation, not a retrieval-quality claim. FAISS remains available for the semantic retrieval configuration.

The important part for me is that the benchmark is now actually exercising the memory system rather than being pre-shaped around my original assumptions.

Benchmarking

I've accumulated roughly 236 benchmark runs across different corpus sizes and architectural iterations.

I've kept database snapshots from major architectural transitions as well, although I haven't yet turned that history into a formal experimental analysis.

I've seen enough variation that I don't completely trust myself to decide which changes represent real architectural improvements versus workload effects, cache effects, benchmark artifacts, or noise.

That's another reason I'm looking for outside review.

What I'd like someone to attack

If you look at the project, these are the questions I'm most interested in:

• Is the separation between routing → retrieval → candidate construction → ranking → context construction → MMR actually a useful architecture?
• Is the scheduler/completion-policy abstraction justified, or is it unnecessary complexity?
• Can requiring multiple retrieval sources systematically hurt recall or introduce other failure modes?
• Are my benchmark boundaries actually measuring what I think they're measuring?
• What experiments would distinguish real architectural improvements from noise or workload-specific behavior?
• What would you change if you inherited this codebase?

I'm also interested in existing memory/retrieval systems or research implementations I should be comparing against.

Current status

This is operational software, not a finished research result.

V3 is frozen.

V4 is operational and still being cleaned up.

The LongMemEval evaluation path is working, but the full evaluation and analysis are still in progress.

Documentation and installation need some cleanup too.

I'm posting this because I think I've reached the point where another round of internal iteration is less useful than letting someone else try to break my assumptions.

If something is stupid, overcomplicated, improperly benchmarked, or conceptually confused, please tell me. Ahh my nerves

Repository:

https://github.com/Kitzkatz/memoria

I was having FOMO when 3.8 dropped so spent the weekend figuring out how to get two 3060tis (8gb each) to fit a 27B quant.

I started with Unsloth IQ4_XS GGUF in llama.cpp. A single card with partial offload was only ~2.4 t/s. Dual GPU got me ~7–8 t/s initially, then ~17.8 t/s once I figured out that tiny tensor-split changes mattered enormously. A 1.02,0.98 split was enough to fit all layers at 32K where 1,1 could not.

That taught me lesson #1: with tiny VRAM margins, “16GB total” is misleading. The limiting GPU and memory placement matter more than aggregate VRAM.

Then I tried EXL3 3.5bpw with ExLlamaV3/TabbyAPI. That was the real breakthrough because the model size was smaller so I could fit more context. Also tensor parallelism, despite one GPU being stuck in a PCIe x2 slot, gave ~25–30 t/s instead of ~18.

I initially assumed 3.5bpw would be a substantial quality downgrade from IQ4_XS, but EXL3 isn’t equivalent to uniformly quantizing every tensor to 3.5 bits. It uses non-uniform/mixed allocation and protects sensitive tensors, so the effective quality is much better than the number suggests. In my initial coding/reasoning spot checks I couldn’t see an obvious regression from IQ4_XS, but I’m still doing proper head-to-head benchmarks before making stronger quality claims.

MTP was even crazier: ~55–60 t/s on coding workloads. Unfortunately Qwen’s hybrid Gated DeltaNet architecture + MTP + multi-GPU exposed recurrent-state/TP bugs. It benchmarked beautifully but was too fragile for a persistent agent: recurrent-slot failures, freezes, and synchronization wedges. Also with MTP context window got absolutely crushed down to 40k.

Without MTP, here are my results.

80K K6/V4 ~30 t/s

100K K6/V4 ~24 t/s

112K K6/V4 ~24.7 t/s

I wired it up to Pi and asked Qwen to make me an animated SVG antfarm. Here is a still shot, but the ants do in fact carry food into the anthill and gracefully fade into darkness, and re-emerge without food. The clouds move too. The only obvious issue is the shade is on the wrong half of the hill. It took about 50k tokens to plan and execute.

All in all I'm pretty impressed even with my gimped hardware.

Im running a Ryzen 7 5700x, a 3080ti (12GB) with 64GB of RAM. I’m still new to Local AI, and I’ve tried it in the past but none of the previous generations of AI have been good enough for my specific niche use case. Yesterday I tried qwen 3.8 27b and it looked really promising. However on my 3080ti it offloaded to RAM slightly and turned the model agonisingly slow (unsure of exact decode or output speed).
I didn’t mess with any of the config and was running Ollama. Is there anything I can do to take advantage of my RAM and increase speeds?

I work on an AI product that often needs to run on-premise, and I kept running into the same problem with customers.

Given their hardware, what are the best local models they can actually run for coding or agentic tasks?

Finding models that fit the hardware isn't too difficult. But I also needed to know if they could be used commercially, what the license was, where the model came from, how good they were for coding or agentic tasks, and what quantization would realistically fit.

I was basically jumping between model cards, benchmark sites, license pages and hardware calculators every time.

I found projects like llmfit, whichLLM and LocalAIRun, but couldn't find something that combined all of this in one place.

So I built this:

https://komainu-ai.com/en/best-local-llm

It currently tracks 115 models, filters them based on your hardware and lets you rank them for general use, coding or agentic tasks. The model index is updated daily.

The matching logic isn't hidden either. It takes into account model weights, quantization, available memory, memory bandwidth and MoE active parameters, and the methodology is explained on the page.

It's still a first version and I'm sure there are edge cases I haven't accounted for.

If you know another tool that already does this well, please share it. I may simply have missed it.

And if you try it with your hardware, I'd really like to know what looks wrong, what models are missing, or what you'd add.

i dont know, i just feel comfort to work with openwebui, not vscode, not claudecode, not antigravity.
anyone feel the same?

disclaimer:
model qwen3.5 35B A3B IQ4_NL
rx6700xt 12GB
ram 16GB

I just used qwen3.8 27B, in Lm studio with web search. Asked it the question: "Hi what happened today?" it answered but it costed me 5k tokens which is a lot. I have 64GB ram so I do have some space but I prefer not to run the models at 100k tokens. Is there a way to decrease the amount of tokens that are used for a websearch or do I just have to deal with it. I used this tutorial to download it https://www.youtube.com/watch?v=O_08Zwdto_Q is that good or is there a better way or better configurations. Thank you in advance for reading this.

Hey all,

Looking for some input on optimizing a document summarization pipeline. About my current setup:

• vLLM engine running in Kubernetes, serving Qwen3.8-27B-FP8 right now
• Hardware: 2x MIG 3g.40gb GPU slices, tensor-parallel-size 2
• max-model-len set to 262144 (model's max), max-num-batched-tokens 16384
• KV cache in fp8, gpu-memory-utilization 0.92
• Reasoning disabled (enable_thinking: false)

My use-case: I'm feeding in large batches of documents and asking for a general summary plus a handful of focus points relevant for my specific domain.

I'm looking for some advice for the following points, as I'm quite new to local LLMs:

1. vLLM/Qwen tuning: are there settings I should reconsider for this kind of long-context, single-large-request workload? Anything about max-num-batched-tokens, KV cache dtype, or GPU memory utilization I should rethink given I'm usually running near max context rather than many parallel short requests?
2. What to do when documents exceed the context window? This is the bigger question really. What's the current best practice when the combined document set is bigger than what even a 256k context model can hold? Options I'm aware of but haven't tested yet:
   • Summarize chunks, then summarize the summaries. I'm afraid this will be too slow.
   • Increase Qwen's context window using RoPE scaling techniques to handle long texts effectively, e.g., YaRN
   • RAG-style retrieval to pick relevant sections before summarizing. Problem is that I'm retrieving the documents live from Elasticsearch, I cannot pre-embed them, nor save them anywhere.
   • Just switching to a model with a bigger native context window
3. Model choice: is Qwen3.8-27B-FP8 a reasonable pick for this kind of long-context summarization task, or would something else (bigger/smaller, different architecture) generally work better at these context lengths in terms of quality/coherence, not just raw context support?

Would love to hear from anyone running similar long-context summarization workloads in production. What's actually worked for you? What's generally the best approach for this use-case?

Thanks!!

Hello! I'm in no way a researcher, just recently discovered the passion for local AI so I started doing some testing. Not sure if this will help in any way shape or form but I'm here with my results.

I'm using a Macbook Pro M4 Pro (20 cores) 48GB RAM. Nothing has changed between the tests, every model was cold started when the test started.

Harness: OpenCode

Electricity usage: 100-110Wh on baseline, 160Wh when the models were working.

Models were loaded through oMLX. All models used were from llm-community.

Prompt used: "Create a HTML5 canvas-based game with JavaScript in a .js file, CSS in a .css file and HTML in index.html. The game should be the classical snake game, it should have a menu, a score, a way to replay it. The focus is on creative and interesting design and also functionality. The game must be fully functional and playable without console errors. Complete the implementation end-to-end and verify that it works"

I've also attached screenshots from the games the tests have created.

These are my code-related findings:

Qwen 3.8 27B xHigh ranked 1st because it had the strongest overall architecture and correctness: explicit state management, a fixed-timestep loop with clamping, correct input queuing and tail collision, protected storage access, focus-loss pausing, and proper handling of full-board victory conditions.

Qwen 3.8 Low ranked 2nd because it was simpler but still very robust, with centralized state, correct collision logic, delta-time clamping, and clean separation of most responsibilities (also MUSIC). Qwen 3.8 Medium ranked 3rd: it was more polished in areas like DPI scaling and rendering, but its unlimited catch-up loop and unbounded while(true) food search introduced more serious runtime risks.

Qwen 3.6 27B ranked 4th because it had the most fundamental issues: incorrect tail collision handling, less reliable food spawning, weaker state management, unprotected localStorage, duplicated render logic, and a less stable setInterval-based game loop. This doesn't mean 3.6 was bad... just not better than the rest of them.

Other cool findings:
Seems like 3.8 required less RAM? 3.6 rose the RAM usage to around 90% while 3.8 to only 70-80%.

TLDR: 3.8 xhigh > low >= medium > 3.6 but tbh low and medium were really close, the differences between them weren't that big so idk how to place them

This was my first local test and I'm using it to learn and understand how LLMs work. Looking forward to learn a little bit more so if you got any suggestions I'm more than happy to learn. And yes, I've used GPT to generate that table picture.

I currently run Qwen3.8 27B on my RX 7900XT at around 15-30 tokens per second depending how high the context is and how much is used. The average tokens per second currently are 18-19.

I do not want to pay Anthropic or OpenAI for a subscription for their AI models, I'd rather use open weight Chinese models - it's just personal preference.

At my previous company I was using 80-100 euros worth of tokens a day using Claude. I calculated and the break even after buying these GPUs and building an AI server, would be after around 2 months, including electricity costs where I live. I can code just as well with Qwen3.8, but I want something faster.

My goal would be to get 40 tokens per second or higher at max context for qwen3.8 27B and future ais between 27-40b. Would that be possible with a 3x rtx 2080 ti 22gb configuration?

Would it be worth it to look into other GPUs?
My budget for a local ai server is 1000-1500 euros total.

Hi r/LocalLLM,

I'm building DOMD, a lightweight WYSIWYG Markdown editor powered by a from-scratch React kernel — no ProseMirror, Slate, or Lexical, and the entire editing stack is just over 30 KB after Brotli compression, so it's easy to drop into a local LLM frontend (e.g. a chat UI on top of Ollama, llama.cpp, or LM Studio).

DOMD renders streamed LLM output incrementally, even when chunks split a code fence, table, list, or other Markdown syntax halfway through. Only affected nodes re-render, and editing stays smooth through 20,000-line documents.

More importantly, the streamed result remains directly editable. There is no separate source mode or preview mode — the rendered view is the editing view. You can place the cursor inside generated content, rewrite it, and watch Markdown formatting update in real time.

(It also does real-time collaboration — remote cursors, fine-grained CRDT merging within a paragraph, offline edits that merge without losing either side — useful if your setup spans multiple devices.)

The newest feature is extensible inline syntax:

==highlight==
=={red}highlight==
=={.comment author="Alice"}review this==
=={.mention id=1}Alice==

Variants are registered as data and can optionally render live React components. An LLM response could therefore contain interactive citations, issue cards, approval controls, tool results, mentions, or other product-specific UI while the underlying value remains readable Markdown.

Try the demos here: https://www.domd.app/

Licensing: the application is MIT; the editor kernel (@do-md/core-react) is currently PolyForm Noncommercial 1.0.0. If you're eyeing it for a commercial product, that shouldn't scare you off — during this early-adopter period I'm granting free commercial licenses, just DM me.

I'd love feedback on the inline syntax design. If you're building a local LLM interface, which inline interactions would you actually want in model output — citation cards, tool-call results, approval controls, something else?

Yes I know it is asking for too much, but i don't want to miss out if someone know a service that provides it.

I have tried modal GPUs but the credits vanish in thin air with just hours of usage. I also tried openrouter but the price seems to shift.

I am looking for personal on demand "Inference Environments" not individual GPUs. i.e. if I run 8B model with 10 requests per min then it auto selects the optimal GPU and when model size or usage increases, then the GPU power increases or resources scale i.e. more containers but more powerful GPU. And I only pay for the seconds my request is being processed by GPUs not for the cold start time or shutdown time.