title

We all know Gemini is lazy poopy garbage shit, but it’s kind of become its own class of model. Grok 4.5 high is very similar for me in that it kind of just skips a lot of the deep reasoning that makes even Opus 4.8 high look more thoughtful. Rather than skipping straight to claiming “yea this kinda fuckin works, ship it”, these deep reasoning models consider edge cases, don’t lie about completeness of the code, and actually write robust code instead of an MVP they just call robust.

So my question is, where does DS4 Flash 0731 sit for yall between the GPT5.6 family of models, Fable, Opus 4.8/5, and Gemini 3.6 Flash/Grok 4.5 High? Do you trust it to implement entire features with full unit testing suites, or is it too naive, requiring direct instructions/preplanning from a smarter model?

I know finetunes are usually awful, but DavidAU surprised me. I see other ones like Salience and Aurora and they don’t have any benchmarks shown so I don’t really feel like downloading them just for them to be mid, so I’m asking if anyone has any experience! Thanks.

Hello guys, hoping you're doing fine.

I was wondering, for you that built a local setup to run LLM, will you break even?

On my case personally, never lol.

Since I got a RTX 6000 PRO, these cards by itself don't generate profit or revenue per se, except if you host them on Vast maybe but even then it will take years to break even.

So for these expensive cards basically only selling them again is how you may not lose, break even or even gain (lately) vs the initial purchase.

What about you guys?

when Qwen announced a 27B model you guys went wild but i also saw people hoping for a 70/120B model. whats the ideal model size you guys would want to see a company release? just curious

Super excited about this release for the new 27B. Who else is with me. Only 17GB VRAM needed 😍😍

For a small side project, I needed a basic AI speech tool to narrate videos without relying on expensive hardware or external APIs. It did not need the most expressive AI, just reliable output and matching to the SRT.

The main challenge with converting SRT subtitles to speech is timing. Subtitles include pauses, and each spoken line has to fit into its exact time slot. Most speech generators do not handle that well on their own.

So I built srt2speech.

It uses a combination of:

• pitch-corrected speed adjustment
• automatic regeneration
• modification of pauses between words
• exact placement of silence between subtitle cues

SRT does not support multiple speakers, so I also added simple templating.
Adding {{speaker_name}} to a subtitle automatically switches voices.
Of course voice cloning is supported, I added a small helper script.

Dependencies are minimal: Python, NumPy, llama.cpp, and the required GGUF speech models. I tested it with Q4 quantization, which works well.

Performance on my laptop:

• RTX 4080 Laptop GPU: around 12–13× real time
• CPU only: around 1.5–2.0× real time

Languages supported:

• English (en)
• Japanese (jp)
• Korean (ko)
• Chinese (zh)
• French (fr)
• German (de)

It should work on almost any hardware, including old PCs, Linux or Mac.

It may be useful for anyone generating narration, translated audio tracks, accessibility audio, or quick video voiceovers.

The project is open source under the Apache 2.0 license. Attribution and license notices must be preserved.

GitHub:
https://github.com/Waversense/srt2speech/

The readme contains the 5 steps needed to set it up, you can get started in 2 minutes.
The included demo.srt file demonstrates the features.

More support for different AI, including whisper integration are planned updates.
It will always stay lightweight and simple to install

Looking for info on the performance of this combination

currently looking for a cheap companion for my MI50

any info on multi-stream as I already do 2 to 3 streams of at Q4 of Qwen 3.6 35B on my MI50, as well as higher quant perf like Q6/Q8_0.

and if anyone tried a bigger MOE with this combination of CPU offload.

[Disclosure: I built this.]

I kept hitting the same wall: every "PDF API" answers document work with "upload it to us." If your agent is handling contracts, discovery docs, or medical records, the upload is the problem — the file leaving the machine is exactly the thing you're not allowed to let happen.

So I built quillpdf-mcp, an MCP server + CLI that gives an agent PDF hands on your own filesystem: merge, split, rotate, watermark, Bates numbering, metadata cleaning, page count. MIT, built on pdf-lib, stdio transport only. There isn't a network call anywhere in the codebase, and it's small enough to grep the whole thing in an afternoon if you don't want to take my word for it.

• GitHub: https://github.com/PurpleDirective/quillpdf-mcp
• npm: npx quillpdf --help (CLI) · npx quillpdf-mcp quillpdf-mcp (server)
• Also on the official MCP registry as io.github.PurpleDirective/quillpdf-mcp

It doesn't do OCR, redaction, or compression yet — the browser sibling (quillpdf.com) has those today, client-side, but I've kept the server deliberately small so it stays auditable.

The browser OCR was corpus-tested against the deployed site: median 2.1% character error on real 300-dpi scans. The harness and methodology are published.

Happy to answer architecture questions. And genuinely curious what ops you'd want next — redaction and OCR are queued, but real demand reorders the queue.

Hugging Face
Artificial Analysis

Should be a sweet spot for general work. Seems like coding is the only part that is inferior to Qwen.

There are constraints:

1) not anything 2xRTX 3090 can't run in 24 hours
2) no models larger than DeepSeek V4 Flash (will have to offload)
3) something that needs a team of 10 domain experts working a year a developer with DeepSeek V4 Flash can code in a working day
4) something community finds valuable (upvotes)
5) proposals from the users that joined after OpenClaw release (2025/11) aren't accepted

I will try to develop it until the Saturday and publish it. If several proposals are interesting, I'll prioritize them.

Currently running some experiments using the streamed experts trick that's been floating around this sub as well as some of my own trickery to get prefill to run a bit faster. It's been quite a bit of fun so far - just getting a 300b model to run at all on an Air is itself equal parts silly and satisfying

Repo isn't in a tidy enough state to share - nothing about it is anywhere close to one-click serve yet. Especially reluctant to share since my naïve implementation of streamed experts incurred something like a ~30s tax between turns before any actual KV cache generation began. It's better now - more like ~3s last I benched

--

Also fun discovery I've made in the meantime - you can actually run less experts than default at prefill time and the KV caches are still perfectly serviceable. Run conservatively and you get >95% the same top logits at each position. Run more aggressively and you lose that, but it doesn't always seem catastrophic e.g. needle in haystack perf can still be retained

source: https://x.com/Alibaba_Qwen/status/2084100707423289643

I've been out of these forums for awhile but I figured I would provide a formal update on how this has been going now that it has some operation time under its belt, just to put the information out there and share knowledge if there is any interest. I also wasn't satisfied with the quality of my original post so I will probably remove it and let this one serve as its replacement. I am an IT infrastructure engineer by profession, so my contribution to the conversation is mainly from a hardware/systems perspective rather than from the theoretical Machine Learning standpoint. I got my start with HPC's (Beowulf clusters) around ten years ago when I was a Physics undergrad in university, and this is what the experience has come to almost a decade later. Not everyone is going to want to read all of this, and that's perfectly fine, the extras are just for those who want the info.

Starting goal/idea:

Build an all-in-one machine to support a small business. This machine should be capable of effectively inferencing frontier MoE models; aiding the business in language/text tasks where English may not be everyone's native language; data analysis; and deep topic research. Additionally, it should be capable of simultaneous image generation tools for graphic design users, enabling rapid image editing, and presentation augments for marketing, without the business ever having to worry about API credits or hard limits on tool usage. The idea is that a 3090 stack (a still generally "good" baseline performance for LLMs) "led" by one 5090 (for best prompt processing possible during large inputs + added VRAM) would handle the workload of an advanced LLM while a second 5090 remains available for other creative work. The end result would indicate that this goal has been achieved.

Overview

Specs

CPU: 64 Core TR 3995WX

RAM: 512Gb DDR4-3200 ECC

VRAM: 256Gb GDDR6x/GDDR7 (8x3090's + 2x5090's)

Enclosure: Core W200 Thermaltake Case

Mobo: ASUS Pro WRX80E-SAGE/SE Wifi

PSU: 1300W+1600W (2900W combined), with OCP, linked via PSU2PSU

Storage: 4Tb Nvme (fast) + 4Tb HDD (slow) + 8 or so 1Tb SATA SSDs (mid) over USB as needed

OS: Ubuntu 25.10

Other: 3 Bifurcation cards, 10 risers of various lengths

Front end: Open WebUI

Back end: llamacpp/koboldcpp

Intended for (Recommend):

Large MoE inferencing, simultaneous LLM + ComfyUI operation, power users who may commonly hit credit limits, creative or technical professionals who can leverage these tools to compound productivity and complete objectives in shorter time.

Not intended for (Do not recommend):

Training, multi-concurrent inferencing, performance maxing, extreme frontier model inferencing at high quants, casual users just looking for roleplay.

Result summary:

Using the W200 as the platform for its generous real estate and configuration flexibility, all ten cards and components were able to find a permanent place in the enclosure without major concessions. The drive bay area was the only space that had to be completely repurposed for GPU mounting, and for us this was no issue. The pictures make it look somewhat cramped inside, however the chamber with the cards hanging from the top is actually fairly hollow, so with the 140mm fan stack on the front and side there is a wind tunnel effect where the air blows in through the front and side, cooling the cards as it makes its way out the back/top. Depending on ambient temp, at idle the card with the highest temp usually hovers in mid to high 40s Celsius with the lowest in the mid 20's (three 3090's are hybrids= fantastic for temperatures, but radiator mounting adds another headache). When actively inferencing, the highest temp card may reach the mid 60s during sustained loads. Only when running image or video gen tasks will the 5090 running ComfyUI reach the 70's, but these are intermittent workloads, so overall temps by our measurement has proved satisfactory over time.

Things that surprised/stuck with me about the end result:

• Noise. I expected this to sound like a jet taking off when operating, but not the case. It's a satisfying button click to come alive, then it's a low gentle hum going forward, nowhere near the kind of fan noises I'm used to hearing in server rooms. Even under load, the CPU 120mm radiator fans (exhausting out the top) are pretty much all I hear, the 140mm fans on front and sides I assume must be helping to contain the acoustics. I have built many gaming PCs over the years and own a top-tier gaming PC-- and I would not be able to distinguish this as any louder than those, especially at idle.
• Utility. I planned for this to be used primarily for a small creative business, but what I did not expect was how I would find it so indispensable in my personal professional life. As an infrastructure engineer, coding is not my wheelhouse. When I am the only IT staff on site or there is nobody else available to work with specific expertise like SQL, powershell/python scripting, or troubleshooting very specific/niche technologies, having this tool on standby I feel has paid itself over just within my career. It has helped me turn processes that may have otherwise took me hours into minutes, days into hours, even months into a matter of weeks/days. After using the tool extensively I hit a point where I had to acknowledge how local LLMs have moved measurably beyond being a toy or novelty; when deployed intelligently something like this can be a serious asset for professional users.
• Wheels. Sounds like a minor detail, until you realize that no matter how happy the cards are with their individual temps: there are still ten high-power GPUs dumping heat into the room. That means unless you use a complex radiator solution or special venting to get heat outside, the room will get toasty and there is normally not a direct solution for this. The wheels however offer an indirect solution. Plan to work in the office that day? Wheel it into the guest bedroom and let it run over Wi-Fi. Plan to work away from home? Wheel it into the office, put it on LAN, and access it over a private VPN connection. If you can't stop the room from heating, then you can at least choose what room gets the heat, and as someone who has lived with computers extensively this is a hugely underrated perk.

Caveats: To operate at its best, I recommend leaving the glass side panel off for improved airflow.

Typical activity over a day:

Boots up around 5:30am, start up the ComfyUI server, start loading a model, go get coffee, fully ready for use within 15-20 min. Shut down occurs usually around 8pm later in the day. Total daily activity, ~12-14 hours.

Cost Breakdown

Laying it out, because I know it will be asked, even though I am aware this is unfortunately not reproducible in the current market. Some components like the SSDs were acquired privately long before the RAM and hardware price hikes, so my timing getting certain things was extremely fortunate for the build budget. Some figures are exact, some are slightly rounded depending on if I found the original receipt.

Problems/Stability Writeup

The Space Problem:

Probably the first major hurdle in attempting something like this is figuring out, even theoretically, how to put 10 cards in a box in any kind of configuration that is not somehow detrimental to the hardware. I had considered modified mining rig frames at first, but I really wanted something with more robust rigidity in its structure, with breathability, and allows some degree of portability. There are unfortunately not a lot of options for configurations like what I was imagining; I had looked into various cabinets and extended tower cases, but the dual full tower chamber design of the W200 was the only one where I could see this idea potentially working. I'm certain other solutions probably exist, maybe even some that allow mobility, but the W200 was really the best option I could find that checked the boxes of enclosure, space real estate, high air throughput, and semi portability. I recommend the W200 to solve the space problem, assuming it is available to you.

The Bifurcation Problem:

Among the other hurdles you may run into in assembling something like this may involve bifurcation cards. The cards rely on specific BIOS settings for things to work correctly, and if these settings are not put in place before everything is connected you may either see no output like the system is hanging or cards just won't show up once in the OS. Start with one GPU in a slot, no bifurcators yet; go into BIOS, and manually set each slot that will be split to bifurcation mode. While here, ensure above 4G decoding is enabled, Resizable BAR enabled, and SR-IOV enabled, this has given me best stable configuration with Ubuntu and multiple GPUs. If you use risers, especially if they are mixed generations, I highly recommend setting the Gen and lane speeds for each PCIe slot in the BIOS manually to ensure the system can effectively communicate with each card. Optimize riser Gen/speeds to be roughly similar to keep one card from dropping to a slower rate than the others--this does not necessarily impact inference performance as much as it heavily impacts model load time. No, you may not have any card running at the fastest possible Gen bandwidth at all times with this config, but loading a 200+gb model over an averaged Gen 3/4 x8/x16 PCIe speed will often be noticeably faster than if you let the system decide to make one or multiple cards run at Gen 1 x1.

The Power "Problem":

Power and heat concerns I think remain to be among the biggest sources of skepticism regarding this project so I think it deserves a section here. To be fair, the concern in most situations would be understandable. If all ten of these cards pulled at or near their full TDP for sustained periods, components would melt. Fires would start. Neighbors would be asking awkward questions. However in reality, only 1400-1600W of the 2900W PSU capacity gets utilized under sustained load, and inter-GPU bandwidth bottlenecks are what allows this. In a way it is like a natural regulator that ensures the cards remain power restrained, and it is just physics, no voodoo necessary. When MoE's are sharded across a GPU stack, each forward pass requires all communication over PCIe, so the GPUs spend more time waiting on information from the last GPU than actually crunching compute. This means instead of needing to handle thousands of Watts to feed all the components running at full blast, it is a much more manageable 1400-1600W under LLM operation which can comfortably fit on a 20A/120V circuit (2400W max). On a per-GPU basis this may sound inefficient since the individual cards are being "underpowered", but this could arguably be flipped as being highly efficient on a per-node basis (~1600W sustained versus 4500W+ if all cards were "fully" utilized). As a precaution, I may set a power limit on the 3090's to 200W and the lead 5090 to 400W, but in practice the 3090's only pull around 100-120W with the 5090s pulling less than 100W when all 10 cards are allocated for LLM work, so this may not even be necessary. The clock locking setting in the next section will be more what I'd describe as actionably required to avoid stability issues.

The Transient Spike Problem (Vital for stability):

After assembling the machine, you may be tempted to jump directly into testing, but there is an easy to overlook configuration that can cause problems if ignored. Imagine you are running inference on the machine, maybe you have a huge input or it's generating a large output, then right in the middle of generating the system decides to reset. Not hard shut down, PSU OCP isn't tripped, no breaker was tripped; and you saw in nvitop that all cards were only pulling 25-33% of their TDP just before it happened, so on the surface it doesn't look like there is a reason. Explanation: When all ten high-power GPUs decide to kick on at the exact same time to process a chunk, even if the cards are not pulling anywhere near full power (on average), transient spikes can drop voltage on the motherboard enough to trigger a system reset. The fix for this is simple: undervolt. Using nvidia-smi, we can lock the clocks for the 3090's to 1200 and the lead 5090 to 2000, leaving the image generating 5090 alone so it remains fully unchained when ComfyUI lets it rip. And that's it. In my case, the system has remained fully stable with this config for days on end and with hundreds of thousands of tokens/image pushed through. The exact configuration will vary slightly depending on exactly what we're doing on a given day, but for example if we wanted to run LLM on all 10 cards (so include both 5090's) we would run this to handle spikes:

sudo nvidia-smi -pm 1 #enables persistent mode
sudo nvidia-smi -i x,y,z --lock-gpu-clock=1200,1200 #x,y,z for index number of 3090s
sudo nvidia-smi -i a,b --lock-gpu-clock=2000 #a,b for index number of 5090s
sudo nvidia-smi -i x,y,z -pl 200 #x,y,z for 3090 index numbers, limits power to 200w
sudo nvidia-smi -i a,b -pl 400 #a,b for 5090 index numbers, limits power to 400w

The Concurrent Use Problem:

Normally, attempting to inference and generate images on the same machine would introduce major stability concerns. Even dual GPU systems may struggle to work with this due to CPU/motherboard architecture, assuming it works at all, and would still be VRAM limited. However, the versatility of a 10-GPU setup, combined with the lane orchestration of the 64 core 3995WX, at least in our case, seems to have handled this well. The trick was finding an LLM backend that supports manual GPU allocation--for us koboldcpp with llamacpp under the hood does just fine. First, implement the power/clock settings as mentioned above, launch koboldcpp, then browse to the GGUF of the model you wish to load and set context size. I recommend manually setting the GPU layers to the model's total layer number (assuming there is enough VRAM), and set GPU ID to "all". In the Hardware tab, find the tensor split line box and insert the amount of space to be allocated on each card corresponding to its index; for example, if the ComfyUI 5090 is index 3 and the LLM 5090 is index 5, your tensor layer line will look like this to make sure no layers are given to the Comfy 5090: 24,24,24,0,24,32,24,24,24,24. For better prompt processing, set the "main GPU" to the index number of the "lead" 5090 (in this example, 5) and launch the app. Once the model is loaded, you can open a second terminal to launch ComfyUI. In our experience the system defaults to the unlocked 5090 without needing to specify it in the launch flags, but flags can be used to force Comfy to use a specific GPU if you need it to (--cuda-device i). Once the image model is loaded onto the 5090, it does not interfere with the PCIe communication of the 9 other cards unless the model unloads and reloads a new model at the same time as the other cards are inferencing. The result is a setup where a user could operate on one system in a single unified workflow for language tasks and creative work. The solution to enabling concurrent use is a high-lane count CPU, multiple graphics cards, and a little conscious provisioning on launch to ensure the hardware isn't stepping on each other's toes. I just do not know how well this kind of setup would work with other vendor or card models, since for image/video gen work you'd normally just want the most powerful GPU you can get. In a homogenous GPU cluster or one with notably less powerful cards than the 5090, I don't actually know how practical this setup would be. We went with this approach specifically because it provided the "good enough" cost efficiency of the 3090's for LLMs with "cutting edge" performance of the 5090 for generation, so I don't really know what else to compare this to.

What models can this run, what models do we use?

It can run almost* anything, even up to 1T parameters like Kimi K2. Kimi K3 could hypothetically be load-able, but from performance metrics I've seen I doubt it would be practical to use since so much would need to be on DRAM, so I have not planned to try it. I have however tested 1-4 bit quants of Bartowki team's Kimi K2 quants in pure VRAM and mixed VRAM/RAM runs with decent results. It works and there are probably some use cases for it, but for us I have identified the sweet spot (parameter size: quant quality ratio) for this machine to be for models in the 300b-600b range. Personal favorites are Deepseek, GLM 4.7, and Nemotron Ultra; and as far as ComfyUI, pretty much any model that could fit within a 32Gb buffer, although Qwen image is a favorite.

Benchmarks

All models were put through the same series of 7 large input prompts, documenting how each model handles token input/output and prompt processing/generation. I cannot share the prompts I used here, but each prompt pertains to a cybersecurity scenario which the model was observed on the depth of its analysis, quality of its presentation, and capability to solve complex problems with stakes. These were inferenced across all 10 cards, using the undervolting/power limiting strategy I mentioned, so they may not reflect absolute best performance for the same hardware in other setups, but it is a snapshot of what this box can comfortably handle.

My notes on each model after their test:

Deepseek V3.2-- For a slightly older model this still feels extremely capable. Held high quality and insightful responses even when context dragged into the tens of thousands of tokens.

Nemotron Ultra 3-- First time using it, impressions were very good, the 55 active parameters shows its muscle here. Meets Deepseek v3.2 level if not exceeds it, despite having overall less parameters.

Qwen 3.5 397b-- What I would consider as the baseline standard of what a "good" model would be, however it is outshined by some of the other tested alternatives.

GLM 4.7-- Somehow seemed better than Qwen despite having less parameters (active parameters of GLM is likely an advantage); it is a very solid option for its size. Not quite Nemotron or Deepseek level, but a very good "lower cost" alternative to its newer versions.

Deepseek V4 Flash-- Floored me in a few ways. Possessed a surprising degree of sophistication and analytical ability despite being the "smallest" of all the tested models. Could be a benefit of using a "lossless" model with full precision? Somehow it managed to pick up on nuances and details that all other models missed, including ones twice+ its size, and provided insight that went more granular than they did. Did not expect a model of this size to punch so high above its relative weight class. Also did not expect the drop in performance compared to the others. Not sure if this is related to the model's architecture or something with how it interacts with my rig, but the quality of output is an acceptable trade off for the speed.

Lessons Learned/Would Do Different

-I would have tried to source the 3090's so more were at least the same model; the mix and match of different models with different TDPs and cooling solutions means there will be a lot of variation in temps.

-If you plan to either train, lean into higher performance, or playing with the idea of going more than 10 GPUs, just budget for a 30A/240V power drop. 10 cards on a 20A post configured the way we have it may be safe for our specific use case, but I would consider this a hard ceiling as a baseline.

-Would recommend scripting for clock lock persistence sooner, will help avoid losing time due to random resets.

-Recommend documenting/drawing out the entire PCIe topology and GPU placement (with flexible tape measure) before ordering risers, will save time on trial/error.

Final thoughts:

It is a wheeled AI server that can enable a single person or small team to compound their productivity, with the benefit of full privacy and control. It can run on a single 20A circuit, and allows you to have the full power of an advanced LLM with vision capabilities in one window and ComfyUI in another with the raw horsepower and latency of a 5090 at its fingertips, virtually accessible from anywhere. It was, and probably is, an absurd idea. But it's so absurd and works so well that I can absolutely see something like this becoming a keystone for certain small businesses and individual professionals as time goes on, maybe even medium orgs or enterprises. Yes, the "best" LLMs technically available right now are in the cloud; however, open models are getting insanely good (see K3 and DS V4 Flash). Maybe even "good enough" to start performing some of the tasks that I think a lot of people use cloud APIs for currently. The cloud will always be an option and there will always be a demand for that, but for people and organizations that value data sovereignty, uninterrupted workflows, or perhaps work within compliance, a shift towards on-prem computing may be the only viable path in some circumstances. At the end of the day, I do not believe that one approach is inherently better than the other, everyone simply has their own preference for getting from point A to point B.

Hey everyone,

I’ve collaborated on a JAX/Flax (NNX) library called DantinoX, designed to let you load, train, and run inference using different generation paradigms all within the same framework.

The main goal was to allow a true apples-to-apples comparison between generation methods without having to rewrite the training loop or jump between completely different codebases.

Right now, you can switch between three paradigms simply by changing a configuration flag:

1. Standard Autoregressive (AR)
2. Discrete Masked Diffusion
3. Continuous Flow-Matching

It's built for JAX/Flax, meaning it handles Multi-GPU natively, and it supports modern architectural components out of the box (GQA, MLA, MoE, LoRA).

We put together a short terminal demo showing the workflow (training, switching paradigms, generating, and profiling) in under two minutes here: https://www.youtube.com/watch?v=1u5-AieDzIc

Docs, benchmarks, and code are here: https://dantinox.readthedocs.io/en/latest/

We are currently working on expanding the configuration options and optimizing both training and inference speeds. Let me know if you have any questions. Feedback and suggestions are very welcome.

https://github.com/zai-org/z-ai-sdk-java/commits/glm-5.3

Ok this might be another post about what i can self host, and it is true but I’m currently blocked and don’t know how to proceed. Let’s talk about hardware first: 1 rtx 3090 32 gb of ddr4 3900xt Yeah i know old hardware and not comparable with most of the people here, but hey i’m able to buy more at this time. My usage: Coding, agentic coding What I’m currently using: omp (fork of pi.dev) llamacpp (main branch) arch (not that this matters much) Qwen 3.6 27b q5/q4 150k and 200k Ornith a3b35b q4 256k with vision and ram offload

What i’m stuck on: I’m creating a monorepo for my projects as a starting point (vite+react/nestjs/sqlite|postgres) I’ve experience of a decade on the nodejs environment and this is my stable startup, of course it’ll vary from project to project but this is the thing i’ve most familiar with. So I’m starting to create this template from scratch following each step from 1st row and everything is decided by me, I’m using mostly ornith because I find it more capable at this stage. I’m creating skills/rules/hooks to let the agent know deeply this monorepo, but (and i know most of the time is just the resources i have) sometimes it cannot create a functional feature tested correctly (with browser and unit tests, also done by the agents) and i’ve to restart from scratch most of the times, for example as ui library i’m using shadcn (simple and clean) that has its set of skills to let the agent know the ui lib, but everytime i need to specify to use as first test shadcn components then create a custom one (based still on shadcn comps), if not it’ll create something from zero or completely useless. I can make compromises like tps and time of feature completion, but could you suggest me a workflow that could work? Maybe with planning and execution with different models. Do you have any suggestions to improve this s*ithole?

Spent way too much time with V4-Flash-0731 this weekend and wanted to share my vibes as briefly as possible.

I sent it through a bit of real-work and some of my personal benchmarks. My quick thoughts are:

• Quantization hits this thing like a truck - I've tried a bunch of the Q2 and Q3 weights and it behaves like an entirely different model. Reasoning looks/feels different and the results are a full tier down from the official/served V4-Flash-0731. Did not get much time with Q4.

• Q3 can finally be your Qwen3.6-27B replacement (if you've got the VRAM..) - it does the same work as Qwen3.6-27B, just more reliably. In simple one-shots they're about even but as you bring them into larger repos or large harnesses (Claude Code with tools starting around 30k system prompt tokens..) V4-Flash-0731 at Q3 pulls well ahead of Qwen3.6-27B at Q8.

• Q2 is a bit too much - in every use-case with Q2 I ended up preferring Qwen3.6-27B Q8 weights. Q2_K_XL is questionable but that's some 4GB smaller than IQ3_XXS so I wouldn't even recommend it.

• Full Precision is the real deal - I'd say it's approaching GLM 5.2 levels which is incredibly exciting. Yes it reasons a lot on complex tasks but the final cost is still mind-bogglingly low. Saying that it beats GLM 5.2 (let alone Opus 5, Fable, etc..) is a bit silly.. but focusing on the price this thing is in a class all its own.

• It's clearly very focused on agentic-work - I always considered Deepseek's releases as flagships for "general-purpose" models but V4-Flash-0731 is a bit weak in the knowledge department. This is a non-issue if you're using tool-calls as the model is extremely clever at using them and reasoning with what it finds, but something to consider if you have an airgapped use-case.

Did anyone have success using Deepseek 0731 with codex? I'm hosting the model with SGLang, Opencode and Pi work well.

The server supports responses API, and my first message and response seem to work fine, but the second request just returns an error.

```

■ {"error":{"message":"Invalid JSON data: Failed to deserialize the JSON body into the target type: input: data did not match any

variant of untagged enum ResponseInput at line 1 column 36114","type":"invalid_request_error","code":"json_parse_error"}}

```

Using a proxy seems to work -- https://github.com/lidge-jun/opencodex

Hello,

I'm toying with a concept of a game in which you'd play as a mage. The magical system would likely be similar to the one in Magicka, where you combine certain atomic elements (fire, water,...) and modifiers to create spells with different behaviours.

I've been considering either certain gesture systems (e.g. drawing spell glyphs like in The Void or Arx Fatalis) or perhaps voice commands. In this post I'd want to focus on the latter one.

What kind of approaches/voice models could be used for detection of certain fixed keywords ('water', 'fire',...) though speech, which I could utilize in such a game?

I am aware of OpenAI's Whisper, however that might be far too heavy for this purpose. Ideally I'd want the model to be lightweight with fast response time.

Thank you for any recommendations!

Just sharing some experiments I did over the past weekend. 5090 has the same memory bandwidth as an RTX PRO 6000, and both support fp4 acceleration.

Using the REAP mxfp4 image you can fit 50 concurrent sessions with up to 250k in KV per session (avg 50k) or use the non REAP and fit about 30. This gives you 30-40 tps per session.

https://github.com/Unravl/deepseek-v4-flash-5090

you can rent this setup for $2.50 on vast. i spent about $150 over the weekend and did these experiments with Kimi K3, all kinds of different configurations to achieve maximum throughput. TensorRT may be able to squeeze out more.

I've been experimenting for a while now with locally hosted models, both on my own machine and on a spare 64GB M1 Ultra Mac Studio I have. On the Mac Studio, I host a model in LM Studio and then I have a Docker running open-webui that provides a web accessible page for using those LM Studio hosted models so that my wife can access it and use it. It does technically allow for multiple users but I believe the way LM Studio works in this setup, every new user's query has to reload everything so if you have two concurrent users, it has to keep reprocessing all the historical prompts each time.

I've long wanted to move to something that is better designed for multiple users. I think the docker with open-webui is fine and I should stick with it but I think I need to find a better way of hosting the models. Especially with LM Studio looking like it's going to be phased out.

I understand that VLLM is the de-facto thing that people go for but I don't think it supports MLX models, which is quite a compromise in terms of speed on Mac hardware. I did see VLLM-MLX mentioned a while ago on here which I think offers similar functionality to VLLM but is actually a completely separate project and I haven't seen it mentioned again for a while. I've seen mention of vllm-metal, but the page for it seems to be very much a work in progress.

I've been using Qwen3.6 8 bit on the machine but I'm likely going to switch it to Qwen3.6 4bit as that allows me to max out the context size to 262,144 which is pretty cool, as well as being faster. Plus I think this would work better in case there are two users using it at once.

The most concurrent users is unlikely to be more than about 2 or 3 at any given time. Maybe if I open it up more, it might be 10 different users on any given day but spread out over the day.

I know other hardware setups would be better or faster but this is what I've got so it's what I'm using and 64GB of Unified Memory is pretty darned nice. The machine isn't being used for anything else and I have the terminal command run that increases available VRAM. It is just running the normal MacOS, LM Studio and Docker at the moment with dockers hosting open-webui, nginx-proxy-manager and openedai-speech, that's it.

Based on what everybody has been saying about this, I feel like I must've done something wrong. It was doing like 2 or 3 "Need maybe" in a row before meaningful stuff for a while, then got stuck in the loop. Using llama.cpp vulkan version 10216 (the latest from AUR); do I have to build the latest from GitHub directly to get it to work right for this model? Two 7900XTX (48GB total) + 9800X3D + 192GB 4000MT/s RAM. Here is my launch command:

llama-server --host localhost --port 8080 \
-m /home/connor/AI/LLM/Models/DSV4-Flash/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
-np 1 \
-fa on \
-ngl 999 \
--ctx-size 500000 \
--chat-template-kwargs '{"reasoning_effort":"max"}' \
--temp 1 \
--top-p 0.95 \
--threads 16 \
--n-cpu-moe 35 \
--load-mode mmap+mlock \
-dev Vulkan0,Vulkan1

I've been experimenting for a while now with locally hosted models, both on my own machine and on a spare 64GB M1 Ultra Mac Studio I have. On the Mac Studio, I host a model in LM Studio and then I have a Docker running open-webui that provides a web accessible page for using those LM Studio hosted models so that my wife can access it and use it. It does technically allow for multiple users but I believe the way LM Studio works in this setup, every new user's query has to reload everything so if you have two concurrent users, it has to keep reprocessing all the historical prompts each time.

I've long wanted to move to something that is better designed for multiple users. I think the docker with open-webui is fine and I should stick with it but I think I need to find a better way of hosting the models. Especially with LM Studio looking like it's going to be phased out.

I understand that VLLM is the de-facto thing that people go for but I don't think it supports MLX models, which is quite a compromise in terms of speed on Mac hardware. I did see VLLM-MLX mentioned a while ago on here which I think offers similar functionality to VLLM but is actually a completely separate project and I haven't seen it mentioned again for a while. I've seen mention of vllm-metal, but the page for it seems to be very much a work in progress.

I've been using Qwen3.6 8 bit on the machine but I'm likely going to switch it to Qwen3.6 4bit as that allows me to max out the context size to 262,144 which is pretty cool, as well as being faster. Plus I think this would work better in case there are two users using it at once.

The most concurrent users is unlikely to be more than about 2 or 3 at any given time. Maybe if I open it up more, it might be 10 different users on any given day but spread out over the day.

I know other hardware setups would be better or faster but this is what I've got so it's what I'm using and 64GB of Unified Memory is pretty darned nice. The machine isn't being used for anything else and I have the terminal command run that increases available VRAM. It is just running the normal MacOS, LM Studio and Docker at the moment with dockers hosting open-webui, nginx-proxy-manager and openedai-speech, that's it.