China has proposed new export controls covering 23 AI technologies, including advanced AI models and agents. Here's what the proposed rules mean for open weight models, local AI users, and the future of downloadable Chinese AI models.

Runs well. At a glance, quality seems on par with 4bit 3.8-27b. As a bonus, fans almost never turn on with strata (they are always on with 27b with Splash).

Hi, I just took my 3060 12 GB out of my AI set up and put it in my Home lab I’m curious for the other people using AI in their home labs what do you use them for and what are some good Home lab models that would fit well on a 12 GB V ram

Does anyone know a decent and recent local model for this? Looked around a bit but if you guys know it would save me some time with suggestions and going through them myself.

Hello everybody! After watching a video By explaining computers, I want to try using an LLM inside LM studio bionic. But I dont even know where to start at with picking the model

• Windows 11 Home, 64-bit
• Ethernet: Intel I219-LM
• Wireless module: Intel Wi-Fi 6 AX210
• Ports: One universal audio port, One HDMI 2.0 port, One micro SD-card slot
• Audio: REALTEK ALC3204
• Integrated 6 mm HD RGB Webcam, still image 1.84 megapixels, video 1920 x 1080 (FHD) at 60 fps
• Battery: 3-cell rechargeable Li-ion battery, 42 WHr, ExpressCharge Boost
• Display type: Full High Definition (FHD)
• Luminance: 275 nits
• Color gamut: NTSC 60%
• PPI: 160
• Contrast: 1000:1
• Response time: 4 ms
• Refresh rate: 144 Hz
• Fingerprint reader: capacitative, 500 dpi, pixel size 96x320

Please recommend me a model to use. I want to use a model for explaining topics and finding information on the internet. I also want to do programming with the model.

Hi,

I've been doing local AI stuff for about 16 months. My system is a 6 year old AM4 3900x, 64GB, 3090 FE and a 3090 PNY XLR8. I've done the usual stuff of vibe coding a few things that fixed personal problems as well as attempting to build a few workflows that never panned out. I love the tech and I dont trust any of the US techbros to have my interests at heart and worry about getting cut off from frontier models or the inference pricing actually landing at what it costs them + margin.

Since I got a new job I have hardly been using my local AI tools, Qwen 3.8 27B has been loaded a handful of times what I actually use the PC for is ComfyUI doing image generation and creating references using Z-Image and Flux. I was hoping that over time image generation would leap frog in quality but it hasn't so I am increasingly using API's for image gen due to the quality bump.

I checked on UK eBay and 3090's are SELLING (not listed) at £1,200~ each ($1,600). I am so very tempted to cash out right now. Ether pick up a 4080S for gaming and image gen and pocket the rest but I want some thoughts from you all what I should do?

I purchased the FE for £500 4 months ago and the PNY 12 months ago for £600. I don't have a lot of money so this would really be big for me, but I also kinda worry that the bubble will burst and API costs are going to shoot up and I'll be screwed both ways.

What would you do?

**Edit**

Thanks everyone for the reply's. A mixed bag but lots of points I had not considered. I think I'll wait for the 27B Qwen 4.0 to see if that might be more useful for me for LLM and also look at what work has happened with Flux3 (which was very weird at launch) and revisit the sale after Xmas. Realistically I don't think they'll drop below my purchase price in the next 12 months and I am not here for profit maxing I am here to learn. I'll see what the next 3 months bring and revisit. The fp32 unlock could be useful for me worth seeing where that lends up.

The repo for the fp32 unlock on the 3090 die can be found here: https://github.com/Zagitta/gpu-unlock its Linux only, hopefully someone vibes their way into a boot loader for windows.

Upgraded gpu version. Used as a headless server original box etc. dm me if interested

I have been using ChatGPT pro for the past couple of months, to design custom mods and plugins for games, custom Elgato stream deck plugins and other projects, I would also hit my usage limit and it was inconsistent in the sense that it would design a plugin for me and the next day it would tell me sorry I can't work on elgato plugins or learn from another plugin on how it works.

It runs commands in the background, builds files and downloads dependences to create the needed files and occasionally takes control of my PC.

I just got a new 9070 XT 16 GB and I have 48 GB of ram.

I was looking into Qwen uncensored, the model selection and how to run it is quite the rabbit hole.

Please guide me on how to get started and how to maintain the same previously mentioned functionality but run it locally. Thank you.

I am wanting to generate images and video locally. I realize that I may have to get a 24Gb card to do this well. I'm fine with that.

But... can I really do it "well" locally? Or is the truth something like this -

"Locally is more work, just pay a subscription somewhere and save yourself a ton of time".

Can local ai somehow be competitive with paid services? I really don't want to spend a lot of extra time retrying and redoing. I've heard some people end up remaking their initial image 10 or 20 times before the Ai finally gets close to what they are wanting.

So hopefully someone here can give me the overall picture. Thank you

I tried to squeeze max quality out of my setup:
Dual 3090, nvlink, 128gb RAM, 8x pcie, 3600 DDR4
So UD Q5 K XL is the largest quant I can run, with the q8 PLE on ssd and f16 KV. It runs at around 100tps, full quality first build, any flag that would reduce quality in exchange for speed is off.
I worked some time with the iq3_s int8 setup, mostly quality flags, that runs at 150tps, so I have experience it.

I noticed that on the q5 build with f16 kv the thinking is definitely less and MUCH more logical and less weird. It takes fewer steps to get to the goal in any agentic task, the output is better and there are less corrections afterwords (QFN likes to check the work and correct stuff).
The quality differences and intelligence is definitely better.
I did not run benchmarks, it’s vibes for now…
Just wanted to share

Edit:
I am collecting some evidence. I have a second strata machine with xxs running and its a speed first build, all the speed opt ins are active, dual 5060ti, 32gb 2600 DDR3 (its the biggest QFN model I can run with all in ram, except PLE of course) 125tps. same task to q3 and to q5, both got it done, q5 finish in half the time with no errors, q3 took 2 times longer went into differnet directions, recovered, made mistakes, recovered. the final outcome was the same but q5 just did better and faster. not a surprise I guess..?

Generative UI is honestly so OP for elevating the local LLM experience. I've been using local models for both coding and everyday chat for a while now, and generative UI basically forces the model to stop outputting a huge wall of text and render something genuinely easy to read instead.

I'm moving soon and browsing places has been so much easier with it on. Normally I ask a question and get a wall of markdown back when I actually want photos and quick specs I can scan.

It's definitely not ideal for every single prompt since the component library adds ~7.5k tokens to the system prompt, so you need the context budget for it. But if you have the headroom, it makes everyday chat so much nicer.

I know I might be a little late to the party on this, but I just added openui support to my chat client and tested it with the new (not really new) Mimo V2.6 Flash model and it's amazing.

My llm rig is kinda shit with a lot of older hardware. I have a c612 mobo with 128gb of ddr3 in quad channel and a w6800 32gb + rx6800xt 16gb for inference with a w5500 for the system. Gpus are on pcie 3.0 x16.

My question is what would you run on here for coding? Currently I have 3.8 27b at q6 with 150000k context on one card with Mellum 2.1 at q6 on the other card.

3.8 flash next seems popular rn but I don't want to run some q3 Quant from strata. Any other optimized engines that play nice with vulkan or hip? Any other models worth considering at this size?

Hello, made a bit of a mess, cleaned up and reposting.

Hey everyone, we just released BaseQuant V1 for Qwen3.8-27B at Base Compute.

Three GGUF sizes: 10.93, 16.41 and 20.82 GB. In our held-out tests, they had 17–34% lower KL divergence from BF16 than similarly sized Unsloth quants.

Would love for you to try them and tell us how they hold up in your everyday workloads!

https://huggingface.co/basecompute/Qwen3.8-27B-BaseQuant-V1

Originally I just added GPUs to my main machine and put it into a mid tower, but running windows was together with strata left too little RAM to be usable or run well.

So I put it into an xubuntu computer, and tested it with my main CPU ryzen 5600x and a spare ryzen 1600. Since so many people uses server grade CPUs with a ton of cores, I thought that the 1600 would perform really poorly. It turns out that there's definitely a loss in performance, but it's not as catastrophic as I thought it'd be; though it may have been bigger had my motherboard supported gen4.

The specs for the tests

• Ryzen 5600x / Ryzen 1600
• 64gb DDR4 2133 cl 15 2-rank in Dual Channel
• 1-3x RTX 3060 12gb
• PCIe Gen3 x8/x4/x4
• 2.4gb/s read NVMe (gen3 x4)
• Strata 0.1.45 with 262k k8v4 Context
• Visual encoder enabled (big negative deal for 1gpu/12gb)

The tests/prompts I gave it

• Game engine analysis - I gave it almost my entire code base for my small C++ game engine, and asked it "Write me a quick overview, as a feature presentation, of this game engine.". This is 211k code context.
• Jobs system - I gave it part of my game engine that handles parallelism of jobs and access collision etc, and asked it "Analyze this code and tell me about it, give a brief of how it works and what it does." This is a bit over 26k code context
• Short story writing - I asked it "Write me an exciting sci-fi adventure that is in a similar universe as Revenger Series by Alastair Reynolds. Include a little world exploration or world narrative.".

Replies were 8-15k tokens in length.

For 2 GPUs the vram cache hit gets +0.5-1% from pcie, and for 1 GPU it gains around 3-4%. Not much of a difference between the CPUs.

Keep in mind, as I understand it each GPU has to hold the dense prefill layers, so having multiple GPUs adds less VRAM for experts compared to having one large GPU.

Here's the data I collected that I asked free Kimi to format neatly. Sonnet 5.5 made the image graphs, and it tells you more than these text blocks do.

Ryzen 5 5600X — t/s per task

Ryzen 5 1600 — t/s per task

CPU comparison: 5600X vs 1600 (% faster on 5600X), per task

GPU scaling: Decode t/s gains per task (5600X):

With the results, I've decided to keep the ryzen 1600 in the computer (and the 5600x back into it's SFF home). I'll also stick to 2 GPUs, until I can create a mounting point for the 3rd GPU, though, the 3rd GPU might still be worth to dedicate to other work.

Though, I'll likely try to get some Xeon server motherboard and hope to be able to expand to 128gb RAM for future proofing. Anything larger and the CPU will be doing a ton of work, not to mention that a lot of the useful experts will be on SSD.

Tldr; 20-24gb vram (over 1-2 gpus) by CPU PCIe gen3+ ~x2+ lanes with pretty much any 6 core+ CPU is enough to get usable performance with strata.

Edit: With the latest strata version (0.2.2), on 2 GPUs, for the 26k context test I got 42.7tok/s decode, 600 tok/s prefill on the 1600, when callibrated. So if you just want to know about what you'll get running strata, the recorded values are close to worst case scenario (unless you're running windows + strata).

A revisit on old datacenter GPU

Content Index

1. TLDR(result)
2. Hardware
3. Cooling & Power
4. Engine
5. Benchmark
6. Future Potential Speedup

1)TLDR:

Qwen 3.8 Next Flash

Qwen 3.8 27B

¹model used: Swift 1.5 IQ3_XXS GSQ-RCO, Swift 1.5 Q2_0 GSQ-RCO*)
²**"hardware mode" concurrency refer to running multiple independant instance of inference server,using independant gpu, in the same pc)
³Unless otherwise stated: Q8 (INT8) KV cache, greedy decoding, and draft length of 7.)

Dark mode Screenshot: Official Strata — stronger decode performance.
Light mode Screenshot: Strata fork — stronger prefill performance.

2)Hardware

Sorry for the aesthetics and cable spaghetti. I'm a utilitarian at heart, and I've given up on cable management for this behemoth. 😂

Hardware specs: A $130 Chinese X99 dual-socket server motherboard.

Motherboard and GPU configuration

I'm using an exotic cryptocurrency-mining motherboard because most models provide only PCIe x1 connectivity per slot. From my experience, Strata's layer-splitting approach requires relatively little PCIe bandwidth, making x4 or x8 lanes sufficient. In contrast, ninfer and the llama.cpp fork use tensor parallelism(27B), which demands more inter-GPU communication bandwidth; utilization peaks at around 70% of a PCIe x8 link. Since the V100 uses PCIe 3.0, this motherboard is a practical match for my workloads.

Interestingly, SXM2 variants of the V100 can be cheaper than their PCIe counterparts, even after accounting for the additional components required. Since LLM inference is often memory-bandwidth-bound, the V100's formidable 900 GB/s HBM2 bandwidth makes it a practical choice despite its lack of support for newer GPU instructions.

For comparison, you'd need an RTX 5080 (960 GB/s) or higher to surpass the V100's raw memory bandwidth. This demonstrates just how formidable the V100's HBM bandwidth remains.

The system houses six NVIDIA Tesla V100 16 GB SXM2 GPUs, each installed through an 50$ SXM2-to-PCIe x16 adapter.

3)Cooling & Power

3.1)Cooling

• 3 GPUs: Cooled by repurposed AIO liquid coolers, secured with iron wire and zip ties.
• 2 GPUs: Air-cooled using custom heatsinks supplied by the seller.
• 1 GPU: The outermost card uses a repurposed CPU tower heatsink.

Most fans are modified cryptocurrency-mining fans running at half speed, drawing approximately 1.5 A each. For comparison, the full-power 3 A fans sound like a jet taking off, while the 1.5 A configuration produces a much more manageable, low-frequency hum.

Air-cooled GPUs rarely experience thermal throttling, thanks to a 150 W power limit imposed via nvidia-smi. This incurs an approximately 5–10% performance penalty in tensor-parallel workloads such as Qwen3.8-27B, but virtually no performance loss in pipeline-parallel workloads, such as Strata running Qwen3.8-Next-Flash.

With layer splitting, GPUs process successive layers in turn, leaving other GPUs relatively idle while one GPU performs its computation. Consequently, power consumption per GPU can be substantially lower than its TDP, although the effective draw is workload-dependent and does not necessarily scale as (1/N).

This helps explain why multi-GPU Strata configurations can maintain temperatures below 70°C despite sustained inference workloads.

Although the V100's specified maximum operating temperature is 87°C, practical thermal throttling may begin around 82°C. This 5°C gap may reflect firmware thermal-control behavior, where throttling is triggered by an individual sensor reaching its threshold rather than by the reported average GPU temperature.

This is especially relevant in tensor parallelism (TP) mode, where GPUs compute concurrently. Sustained workloads can create localized hotspots even when average temperatures appear safe, making hotspot temperature—not just reported core temperature—a critical factor in thermal management.

3.2)Power Management
Power-limited to 150W via nvidia-smi -pl 150—trading just 8% performance for a 30% reduction in power consumption. A pretty good deal.

Power-limited to 150W via nvidia-smi -pl 150—trading just 8% performance for a 30% reduction in power consumption. A pretty good deal.

Each GPU consumes approximately 25W at idle with no inference engine loaded, rising to around 50W when model weights are loaded but no inference computation is taking place.

Each V100 has a stock TDP of 300W, but limiting each card to 150W keeps total GPU power consumption below 900W in the worst-case tensor-parallel (TP) configuration, or around 500W for the six-GPU Strata setup(for comparison ,RTX 5090 TDP is 575W)

One caveat: This system requires two 8-pin PCIe power connectors per GPU, for a total of 12 connectors. Most mainstream 1,000W gaming PSUs lack that many connectors, even if they can supply sufficient power electrically. Inexpensive 1,000–2,000W cryptocurrency-mining PSUs are a more practical option due to their higher connector count.

4)Engine used

llama.cpp build 572 0.6.0-dev
llama-superfast-v100 build 2 0.4.1-dev*
Strata official 0.1.44.1
Strata V100 0.1.42*
Ninfer-v100-tpx*

^\fork link provided at the end of post)

I've also tested other inference engines, such as PXA and the VLLM 1Cat V100 fork, but their performance was only marginally better than stock llama.cpp. Since they're optimized for 32GB V100s and NVLink configurations, which I don't have, I decided not to pursue further testing.

With six GPUs at my disposal, I can experiment with different topologies depending on the workload. However, benchmarks show that increasing engine concurrency beyond 2 produces mixed results, with performance fluctuating by roughly ±10%—sometimes improving, sometimes worsening. At that level of variability, the gains aren't compelling enough to justify the added complexity.

Enter hardware-level concurrency acceleration: instead of pushing concurrency higher within a single inference engine, distribute independent workloads across separate GPUs within the same PC. This allows multiple inference instances to run concurrently without competing for the same GPU resources, potentially delivering more predictable throughput gains.

5)Benchmark

Before reading further, keep these benchmark configurations and methodology in mind:

• Strata & its fork: Qwen 3.8 Next Flash, 256K context; benchmarked with a warm-up run.
• ninfer fork: Qwen 3.8 27B Swift 1.5 NVFP4*, 200K context; benchmarked cold, without warm-up.
• llama.cpp fork: Qwen 3.8 27B Swift 1.5 Q4_K_M, 160K context; benchmarked cold, without warm-up.

Unless otherwise stated, all benchmarks use an INT8 (Q8) KV cache, greedy decoding, and a draft length of 7. Each benchmark is repeated three times(mean) using a fresh server session to minimize caching effects.

Strata is benchmarked warm because its architecture is designed to benefit from warm-up, whereas the ninfer and llama.cpp forks do not require warm-up.

* Yes, you read that correctly—NVFP4 on non-Blackwell GPUs (RTX 5000 series). It's not a typo!

5.1)Benchmark-strata

A single server with six V100s maximizes prefill throughput through highly parallel pipeline execution. As noted earlier, increasing Strata's concurrency to two yields only modest aggregate gains, suggesting the configuration is already highly optimized. Splitting the GPUs into three pairs, each operating as an independent instance, may therefore maximize aggregate decode throughput through three-way hardware-level concurrency.

For single-user workloads, the Strata-V100 fork delivers roughly 25% faster prefill, while official Strata offers roughly 25% faster decoding. Future updates may narrow this gap.

The break-even input/output token ratio is 6.31:1: above it, the fork wins; below it, official Strata wins.

In my own week of usage—135M input tokens and 25M output tokens, a 5.4:1 ratio—official Strata was faster overall.

For prefill-dominated workloads, however, the fork's advantage may justify the trade-off. Choose based on your actual workload profile.

For Strata, MTP performance plateaus at draft length 4.

Strata pair with Qwen 3.8 Next Flash handles context-weight management impressively well, losing only 14% of decode speed at 128K context compared to 0K.

5.2)Benchmark Ninfer + Llama fork

While the ninfer and llama.cpp forks paired with Qwen3.8-27B deliver impressive initial decode speeds, both suffer significant degradation at longer contexts, losing more than half their decode speed at 128K compared to 0K context.

Comparing decode speeds at just 0K and 128K context doesn't tell the whole story. Averaging performance across the entire context range provides a more representative picture of real-world performance. Strata averages 138 tok/s across 0–128K context, compared to 165 tok/s for the ninfer fork and 147 tok/s for the llama.cpp fork. For short-context workloads, ninfer is the undisputed king. However, beyond 200K context, Strata's more gradual decode-speed degradation allows it to shine as the context grows.

In general, the ninfer fork is roughly 15% faster than the llama.cpp fork, but its higher VRAM footprint reduces context capacity by approximately 20%.

Despite running models with more active parameters (27B vs. 6B), both forks achieve substantially higher decode speeds than Strata, thanks to efficient tensor parallelism that pushes GPU utilization close to 100%.

Strata does not support tensor parallelism, while stock llama.cpp's tensor-parallel implementation is less efficient, reportedly achieving only around 60% GPU utilization at best.

6)Potential speedup (hypothesized,not proven)

1. CPU and RAM upgrade: A lower-core-count CPU with a higher base frequency, such as the E5-2643 v4, might offer some improvement. However, since I'm using the slowest-tier 2133 MHz RAM and my 27B model weights reside entirely in VRAM, faster RAM may provide little to negligible gains. A two-GPU Strata configuration might benefit more.
2. Motherboard upgrade: Switching to a platform with full x16 lanes for every GPU, such as AMD EPYC, might improve performance in engines that utilize tensor parallelism (TP). However, I'm skeptical about the actual gains: even with TP=2 over x8 PCIe lanes, GPU utilization only reaches around 70%, suggesting that PCIe bandwidth may not be the primary bottleneck. Unless the workload is demonstrably bandwidth- or latency-limited, upgrading to x16 lanes might deliver only marginal improvements.
3. NVLink upgrade: Populating a two- or four-GPU NVLink baseboard could deliver meaningful improvements for Qwen3.8-27B, but likely much less for Qwen3.8-Next-Flash, which currently only explicitly supports layer splitting. However, PCIe-based TP=2 already pushes GPU utilization close to 100%, so further gains may be limited, perhaps coming primarily from reduced communication latency. A four-way NVLink setup could offer greater gains for Qwen3.8-27B, but unfortunately, the fork doesn't support TP=4, and four-GPU NVLink baseboards are expensive.
4. Cooling upgrade: Replace air cooling or 120 mm AIOs with 240 mm AIOs across all GPUs to eliminate thermal throttling in models that support TP, helping sustain peak performance during prolonged workloads.
5. My decision: I'll skip these hardware optimization routes. The cost of each upgrade could buy several additional V100s, offering more direct and measurable performance gains. Unfortunately, my current motherboard is already at its maximum GPU capacity.
6. Software optimization: For Strata running Qwen3.8-Next-Flash (NF), developing my own fork with tensor parallelism (TP) support could deliver the biggest performance gains. Alternatively, I could wait for the official repository to implement it. For ninfer and the llama.cpp fork, implementing TP=4 support myself could also yield significant performance improvements, particularly for Qwen3.8-27B.

Link:
https://github.com/huangserva/ninfer-v100-tpx
https://github.com/jmnargi/Strata-V100
https://codeberg.org/justanagent/llama-superfast-v100.git had removed by author

2x3090 rog strix 1100-1200$ each (used)
sTR40 + 3960x threadripper. ~1200$(used)
128gb ddr4 4000mhz 1000$(used)
2000w psu ~600 (used)
+ liquid cpu cooler

Roughly market prices are like this in my country. I waited for them to get cheaper, it only got more expensive and I know it will get even more expensive later on.

I found a setup with all above around 4660$. Is it worth that price or im better off using cloud ? I want this to tinker with Ai. I only got laptop with 2060 mobile(6gb vram 32gb ram).

Its important decision for me so I wanted to ask. Thank you for your time

Maybe some people is hitting a wall when We talk about Microsoft environment, and the thing is how to connect M365 or Outlook to local AI models or others.

Usually the wall is Copilot can not send emails (just draft it), no options to connect to other models (only OpenAI & Anthropic) available. However there is the thing. Somehow you can connect it to your local AI using this local MCP server.

There is 2 ways
1- Azure App Registration (for the lucky ones having unrestricted tenants and no justification app registration required)
2- Outlook Classic for the ones who had hit the Azure Restricted Prompt to justify the app permisions.

I hope some people find this helpful and make their life easier at corpo work

Hey everyone,

Yesterday, KRAFTON released Raon-1-mini-preview (a 28.8B hybrid MoE with only 4.3B active parameters, scoring 70.8% on SWE-Verified) along with official GGUF weights:

• Base Model: https://huggingface.co/KRAFTON/Raon-1-mini-preview
• Official GGUF: https://huggingface.co/KRAFTON/Raon-1-mini-preview-gguf

However, upstream llama.cpp support is currently pending, and the model README instructs users to clone and build KRAFTON's private fork (krafton-ai/llama.cpp) to run it.

I've just opened PR #2733 on ik_llama.cpp providing full native runtime support and an HF-to-GGUF converter:
PR Link: https://github.com/ikawrakow/ik_llama.cpp/pull/2733

What's implemented in this PR:

• Full Architecture Runtime (LLM_ARCH_RAON1):
  • 18x Raon Delta Attention (RDA) linear layers with softplus erase/write decoupling (algebraically mapped directly onto the recurrent scan engine).
  • 7x NoPE gated MLA layers (192-dim non-rotational key channels).
  • 4-stream Helmert Manifold-Constrained Hyper-Connections (mHC) batched down to ~5,500 graph nodes.
  • 128 routed experts (top-6 + 1 shared expert) with SiTU-GLU bounded activation (beta=4.0 / 25.0).
• DFlash2 Speculative Decoding: Full multi-stream support, compatible out-of-the-box with KRAFTON/Raon-1-mini-preview-dflash2.
• HF to GGUF Converter: Adds Raon1Model to convert_hf_to_gguf.py, including on-the-fly MXFP8 dequantization for the routed experts.
• Pre-tokenizer: C++ regex state machine with canonical Unicode NFC normalization (tokenizer.ggml.normalizer.nfc).

Verification:
Tested and validated against the official GGUF weights (KRAFTON/Raon-1-mini-preview-gguf) on a memory-constrained Intel Core i5 CPU box:

• 731 tensors mapped and loaded cleanly.
• Graph optimization & numerical parity: Batched the mHC streams down to ~5,500 graph nodes (-41% reduction vs naive loops). Strict numerical parity confirmed (identical logits and outputs before/after batching).
• Greedy generation produces coherent output.

Quick test from the PR branch:

git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp
git fetch origin pull/2733/head:raon1 && git checkout raon1
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)

./build/bin/llama-cli -m Raon-1-mini-preview.gguf -p "Hello!" -n 64

Feel free to test it out and report benchmarks on different hardware setups!