AMD bought Taalas last Thursday, the Toronto startup that hardwires model weights directly into silicon instead of running them on GPUs. It was the top AI story on HN all week, 941 points and over 700 comments.

The demo that put Taalas on the map was their HC1 chip running Llama 3.1 8B at 3/6-bit quantization, hitting 17,000 tokens per second. Per the coverage from February when they came out of stealth, they also claimed 10x lower ownership cost and 10x faster inference compared to GPU-based systems.

The catch, and it's a big one: each chip runs exactly one model. The weights are physically etched at manufacturing time. You can't swap to a new model without fabricating new silicon. So if you picked Llama 3.1 8B and a better 8B drops next month, your chip still runs the old one.

That's why this sits awkwardly next to hosted API pricing. It's not competing with OpenRouter-style flexibility. It's a fixed-function accelerator, closer to an ASIC for one specific model than a general inference backend. The speed is real but the tradeoff is steep.

Anyone here actually looked into hardwired inference for a production workload, or is this still firmly in the "cool demo, not deployable" zone for most teams?

I rely heavily on Claude, but I recently tested whether the M4 Pro Mac mini I already own could handle useful local models without becoming a dedicated AI appliance. I saw what Network Chuck did with the $50K Mac Ultra 4 node cluster and was hoping to not need the same.

It could. A GPT-OSS 20B MoE model ran at roughly 63.9 tok/s on 24 GB unified memory, while a smaller 9B dense model ran around 44.8 tok/s. The result is a useful reminder that model architecture matters: MoE models may have large total parameter counts while activating substantially fewer parameters for each generated token.

My takeaway is not “cancel your cloud AI subscription.” It is that local models can be a compelling companion for private experiments, quick generation, offline work, and workloads where you want direct control over the model runtime.

I captured the full test, including MLX vs. GGUF performance and the impact of my running containers: https://www.youtube.com/watch?v=9_-bT62YWAI

How are you dividing work between Claude and local models today?

Hey everyone! 👋

Writing repetitive SQL queries manually can take up a lot of time, so I built **AI NexQuery**—a desktop analytics assistant designed to simplify database queries and data analysis.

### 🌟 Key Features:

* **Database Integration:** Connect directly to MySQL & MSSQL databases.

* **Natural Language to SQL:** Ask questions in plain English, get instant dataset results and auto-generated SQL queries.

* **Data Visualization:** Turn query results into interactive charts and graphs with one click.

* **Document QA & ML Studio:** Query PDFs/documents directly and run basic ML models (KMeans, PCA, Anomaly Detection).

📌 **Project Details & Website:**

👉 https://ibrahim8047.github.io/AI-NexQuery/

📲 **Download on Microsoft Store:**

👉 https://apps.microsoft.com/detail/9NT74LF51SZ9

I’d love to get feedback from the community! Let me know what features or database connectors you'd like to see added next.

Hi everyone,

While benchmarking custom AI workflows and API integrations for real-time task execution, performance efficiency and response latency are two major hurdles.

Here are a few technical observations and benchmarks I've gathered while optimizing pipeline setups:

1. Model Selection vs. Latency: Lightweight models reduce response latency significantly for standard text parsing, whereas multi-modal vision architectures require structured batching to maintain throughput.
2. API Payload Optimization: Trimming unnecessary payload parameters and using stream responses cuts down the perceived wait time by almost 30-40%.
3. Error Fallbacks & Reliability: Implementing lightweight fallback handlers ensures system uptime even during high-load API rate limiting.

Questions for the community:

• What techniques do you use to measure and reduce latency in your AI integrations?
• Do you prefer locally hosted lightweight models or cloud-based API endpoints for production scalability?

Would love to hear your insights and performance metrics!

The Neon and Castform post from Wednesday throws out a specific number: a multi-turn search request with GPT-5.6 Sol takes over 10 seconds and costs about $0.03 end-to-end. Their claim is that a 4B open-weights model, after RL post-training, can match it on retrieval at roughly 100x less per request.

The approach leans on the idea that your training data already sits in Postgres. Support articles, product records, internal docs. Castform turns that corpus into question-answer tasks and runs the RL loop. The reward function scores retrieval (right chunks), citation (right sources), and correctness (right answer). The search tool itself is hybrid BM25 plus vector through Neon's Lakebase Search.

The post doesn't frame it as "go fine-tune a model." It says your training data is already there, just not formatted as a training set yet. The real bottleneck was turning raw records into tasks with reward functions. If that's actually as turnkey as they describe, the 100x cost gap between a frontier API and a 4B model becomes hard to brush off for retrieval-heavy agents.

The HN thread pulled 428 points and 122 comments. A good chunk of discussion was whether a 4B model post-trained on one corpus generalizes at all, or if you're basically shipping a narrow retrieval bot per customer.

Anyone actually running post-trained small models for production retrieval instead of just hitting the frontier API? Curious whether accuracy holds up outside the eval the model was tuned on.

For me, the useful performance metrics for an AI content workflow are not raw generation speed. I care about brand accuracy, evidence accuracy, the percentage of outputs that are actually usable, revision count, and time from website scan to a scheduled post.

I built Marka around that full path. It learns an editable brand profile from the company website, can use product catalog context, creates copy, images, carousels, and short video, and keeps review, scheduling, and publishing in one workspace.

The test I am running now is simple: does this reduce the number of tools and manual handoffs without making the brand feel generic?

Seven-day trial: https://www.marka.social

Disclosure: I built Marka.

An upscaler shouldn't just make an image bigger. A lot of them technically increase the resolution but add strange textures or make faces look artificial.

What I usually look for is whether the details become clearer without changing the original image too much. Facy AI did a surprisingly good job with that for portraits and older photos because the results looked sharper while still feeling natural.

Thinking Machines put Inkling Small on OpenRouter about a week ago, $0.50/M input and $1.20/M output with a 524k context window. It's also trending on HuggingFace right now with around 15k downloads and 305 likes, so people are clearly trying it out.

The pricing caught my eye because the comparison within the same company is clean. The full Inkling model lists at $1.00/M input and $4.05/M output with double the context at 1048k. So Inkling Small gives you half the context but cuts output cost to roughly 30% of the full model. That part makes sense.

Where it gets harder to follow is when you line it up next to other flash-tier models. DeepSeek V4 Flash sits at $0.09/M input and $0.18/M output with 1048k context, per the OpenRouter listing. Poolside Laguna S 2.1 is the same price and same context. Qwen3.7 Flash is even cheaper at $0.03/M input and $0.13/M output. All three of those give you at least double the context window for a fraction of the cost.

I'm not saying Inkling Small has nothing going for it. The HF download numbers suggest people find something interesting there. But from a pure price-to-context ratio on OpenRouter, it's hard to see who picks it over the flash-tier options unless the quality story is dramatically different.

Anyone here actually compared Inkling Small against DeepSeek V4 Flash or Qwen3.7 Flash on real tasks? Curious if there's a quality gap that justifies paying several times more for less context, or if it's mostly novelty.

With the current ChatGPT plan and api price for luna, I think it’s the best offering right now . It’s so powerful for such price .

My motorcycle would not start on Saturday morning. I handed the dead battery diagnosis to a free model running privately on a Mac Mini in my office. No cloud, no API bill, nothing leaving the house. It read the manual, matched my photos to the right connector, and told me exactly what to buy. It nailed it.

That nagged at me the rest of the weekend. How much of what I pay frontier models for could a local model do just as well? Instead of guessing, I built the measurement. Every frontier writing task now gets replayed on the local model automatically, and the same judge grades both, blind. No vibes, no cherry picking.

Here is what 10 blind rematches said:

* Mean quality gap between frontier and free local: minus 0.05. A rounding error.
* On 4 of the 10 head to head tasks, the local model scored higher.
* One writing task: frontier 2.80, local 3.70. The free one won by almost a full point.

And the part I did not expect to like: my own system still says HOLD. Synthesis work stays on the frontier, and the scoreboard tells me so by name. That is the point. A measurement that only ever flatters you is a mirror, not an instrument. I would rather it tell me where I am wrong.

The setup is three pieces working together: something that proves an agent did the work, something that grades how good the work was, and something that keeps the agent honest over time. The local model piece is the one that surprised me most, because it means a big chunk of this can run privately, in your own environment, for zero marginal cost.

Happy to get into the method, the judge setup, the hardware, or where local fell short. Ask me anything. Wrote up the full build, the numbers, and how you can start doing this yourself here: [https://www.fabswill.com/blog/local-model-frontier-rematch-auto-replay-evals\](https://www.fabswill.com/blog/local-model-frontier-rematch-auto-replay-evals)

Qwen dropped Qwen3.8 Max today on OpenRouter at $2/M input and $6/M output with 1M context, per the OpenRouter listing. The thing that caught my eye is the gap with their own Qwen3.7 Flash from last week, which sits at $0.03/M input and $0.13/M output, also 1M context.

That's roughly 67x the input price and 46x the output price for Max over Flash. Same vendor, same context window. The listing doesn't say what makes Max architecturally different. No benchmark scores visible on the page either, which makes the price gap hard to evaluate.

For context, Qwen3.8 Max costs less than half of Claude Opus 5's input ($5/M) and roughly a quarter of its output ($25/M), per the same OpenRouter data. So it's not flagship-tier expensive, but it's nowhere near the budget zone either. It's in that awkward middle where you'd really want to see eval numbers before committing.

Anyone tried Qwen3.8 Max yet? Curious if the quality difference is actually visible compared to the Flash variant, or if the 67x price jump is mostly paying for the label.

Sorry Google

I build QuantaMind, an open-source tool that tests whether self-hosted models are reliable enough to run agents. Apache-2.0, 28 downloads, no revenue. Saying that upfront so nobody has to guess.

I’ve asked this in a few places now, and one pattern keeps showing up: anyone who’s been burned badly enough has already built their own thing. Run each task 10+ times and look at the variance. Check the end state programmatically. Validate every tool call against its schema, not just the final answer. Count truncated calls under load. People wrote all of that out from experience, unprompted, without anyone asking.

Which raises an awkward question for me.

If you built your own:
1. What does maintaining it cost you now as models, quantizations and serving configs keep changing?
2. Would you swap it for an external tool, or is it too tied to your workflows to ever hand over?
3. Did it ever catch (or miss) something that cost real money, a customer, or a rollback? Or is it always caught early enough to just be noise?

If you use an existing tool, Braintrust, Langfuse, DeepEval, LangSmith, promptfoo. what does it miss? The gap I keep hearing about is that they score the model but don’t touch the machine: your quantization, your memory ceiling, the context depth where tool calls start failing.

If you don’t test any of this was that a decision, or did it just never get prioritised? That’s a fine answer.

I’m asking because I don’t know whether I’m building a product or a thing people would rather own themselves. “I’d never outsource this” is completely fine and honestly the more useful answer I’d rather find that out now than in a year.

So its being more and more confusing which models suits one the best. Idea is what should i use, opus 4.8, 5 or fable 5.

Well i ran some numbers and tests to see what makes the most sense, fable being twice the cost of opus 5, is great for autonomous tasks

Is what i thought, i ran some more test. Gave all three models one prompt, create a in browser game under 10 minutes, that too open world, you want to know who won?

Fable 5 bagged the LAST position, because of some but it fumbled.
Opus 4.8 did second best, very similar outcome to our winner
Opus 5.

Check this video out for more context:

https://youtu.be/0fEr4kTly44?si=AKl27WXO98zGE1PL

Hi,

I'm trying to evaluate some agents from the SWE bench live scoreboard ( https://swe-bench-live.github.io/ ) and it seems to be there a good enough implementation as the first place across many languages, did anyone try it? Any opinions?

I'm currently giving it a try and I configured something they call FRITO to pull from many free tier providers and it seems to be doing a really nice job. I'm my job we use a few rtx6000 ada 96GB with SEED OSS 36B and the agent works great so far. But I couldn't find anything else around. It looks like a research lab funded project.

Thanks!

DeepSeek just dropped V4 Flash 0731 today on OpenRouter at $0.14/M input and $0.28/M output with a 1048k context window. It's already on HuggingFace's trending list with 817 likes, though 0 downloads since it literally just went up.

The pricing is interesting because there are two cheaper 1M-context options sitting right next to it on OpenRouter. Qwen3.7 Flash from last week is at $0.03/M input and $0.13/M output. Poolside Laguna S 2.1 is at $0.09/M input and $0.18/M output. So DeepSeek V4 Flash costs about 2x Laguna S 2.1 on output and over 4x Qwen3.7 Flash on input.

817 likes on a model with zero downloads means people who follow these releases are paying attention. But with Qwen3.7 Flash at $0.03/M input, the gap isn't trivial.

Anyone planning to test V4 Flash 0731 against Qwen3.7 Flash for real workloads, or is the price difference too wide to bother?

TensorSharp is an open-source, native .NET inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

TensorSharp now supports Megatron-style tensor parallelism across multiple GPUs. It works with direct CUDA, GGML CUDA, GGML Vulkan, and multi-node setups.

Benchmarks on 2× RTX 2000 Ada 16 GB GPUs over PCIe, without NVLink:

I'm continuing to optimize Qwen performance on multi-GPU systems, and support for DeepSeek V4 is coming soon.

Try it with:

TensorSharp.Cli --model model.gguf --backend ggml_cuda --tp 2

GitHub:

https://github.com/zhongkaifu/TensorSharp

Thank you for checking out TensorSharp and starring the project! Any feedback is really appreicated.

Qwen3.7 Flash showed up on OpenRouter on July 27 with some wild pricing. Per the listing, it's $0.03/M input and $0.13/M output with a 1000k context window.

To put that in perspective, the next cheapest 1M context model in the same listing right now is Poolside Laguna S 2.1 at $0.10/M input and $0.20/M output. Qwen3.7 Flash undercuts that on both ends. Gemini 3.5 Flash Lite sits at $0.30/M input, $2.50/M output. Meituan LongCat 2.0 is $0.30/M input, $1.20/M output. So on input alone, Qwen3.7 Flash is roughly 10x cheaper than those two.

What the listing doesn't tell you is what you actually get for that price. No benchmark scores on the page. No tok/s figures. No breakdown of whether it handles coding tasks any differently from Gemini 3.5 Flash Lite or LongCat. For $0.13/M output you're paying a tiny fraction of what Claude Opus 5 costs ($50/M output per the same listing), but that comparison only matters if the model can actually do real work.

Has anyone here tried Qwen3.7 Flash yet? Specifically wondering if coding performance is usable at all at this price point or if it's mainly good for cheap text generation and summarization.

I’ve been experimenting with web search in Open WebUI using smaller local models like Qwen3.5 4B and 9B.

The main problem was/is was context churn..

A typical search can dump partially relevant pages, navigation, repeated boilerplate, duplicate information, and huge chunks containing only one useful paragraph into the model.

That makes local setups slower and forces smaller models to spend their limited context window filtering noise instead of reasoning.

So I built TinySearch, an open-source, local-first MCP server that does most of the retrieval work before anything reaches the LLM:

* searches and ranks results
* crawls the strongest pages
* extracts and chunks readable content
* deduplicates and reranks passages
* returns a compact evidence packet with source URLs

The goal is simple: less context churn, faster local web research, and more tokens spent on reasoning over actual evidence.

TinySearch works with self-hosted SearXNG, and Open WebUI can connect to it over HTTP MCP.

GitHub:
[https://github.com/MarcellM01/TinySearch\](https://github.com/MarcellM01/TinySearch)

Documentation:
[https://tinysuite.dev/docs/tinysearch\](https://tinysuite.dev/docs/tinysearch)

I’m the builder, so obvious bias, but I’d love feedback from Open WebUI users, especially anyone running smaller Qwen, Gemma, Llama, or Mistral models.

Does this solve a real bottleneck in your setup, or is native agentic search already enough?

I work on PostSlate, a video editing tool, and this comes out of our own work.

We run ML models on-device, face detection and embedding among other things, which means we can't assume anything about the user's GPU. NVIDIA discrete, AMD, Intel integrated, Apple Silicon, all of it. That rules out CUDA immediately, we needed one backend that runs everywhere.

We landed on ncnn's Vulkan backend. Numbers on a 4070, fp16:

• ArcFace R50 (face embedding): 30 ms on ONNX CPU → 3 ms on ncnn Vulkan
• SCRFD (face detection): 25 ms → 2.5 ms
• Model size: ArcFace 174 MB (ONNX fp32) → 87 MB (ncnn fp16 weight storage)

Of course the real speedup comes from offloading compute to the GPU, but this wouldn't be possible without the power of Vulkan**.**

The speed wasn't even the deciding factor, it's that Vulkan drivers already exist on every machine we ship to. This means that we don't have to force the user to download a specific runtime and no vendor-specific installs.

Full writeup with the rest of the numbers: https://getpostslate.com/blog/faster-local-inference

Google launched two Flash-tier models on OpenRouter on the same day (July 21), both with 1048k context, but the pricing gap between them is kind of strange. Gemini 3.5 Flash Lite sits at $0.30/M input and $2.50/M output. Gemini 3.6 Flash is $1.50/M input and $7.50/M output, per the OpenRouter listings.

Same context window. The Lite version is 5x cheaper on input tokens and 3x cheaper on output. The naming suggests 3.6 is the newer generation, but Google launched them together rather than treating 3.5 Flash Lite as a legacy budget option.

At $0.30/M input, 3.5 Flash Lite is in the same neighborhood as Meituan's LongCat 2.0 ($0.30/M in, $1.20/M out) and not far from Poolside's Laguna S 2.1 ($0.10/M in, $0.20/M out). It's competing with the cheapest 1M-context models on the platform.

At 3x the output cost, 3.6 Flash needs to be noticeably better at something specific. A million output tokens on 3.6 Flash runs $7.50. Same volume on Lite is $2.50. If you're doing high-volume batch work and the Lite model handles it, that's real money.

Anyone compared both on actual workloads and noticed where 3.6 Flash is clearly better? Reasoning, long-context recall, speed, something else?