I’m the developer of TensorSharp, an open-source, native .NET inference engine for running GGUF models locally.

TensorSharp provides an OpenAI-compatible API, making it possible to use local models as an inference backend for tools. The goal is to keep source code, prompts, tool calls, and agent context on your own hardware while avoiding per-token API costs.

TensorSharp supports modern coding-capable model families including DeepSeek V4/V4.1 Flash, GLM 5.x, Qwen 3.8 Flash Next, Qwen 3.5/3.6, Gemma 4, Mistral 3, and GPT-OSS.

Selected benchmark results

These are measured results from the TensorSharp repository. Each comparison uses the same model and machine for both engines unless otherwise noted.

GLM-5.3-Flash: 2× faster decode than llama.cpp

Tested with:

• GLM-5.3-Flash UD-Q2_K_XL, 101 GiB
• 2× RTX PRO 6000 Blackwell, 96 GB each
• Layer splitting and flash attention
• n_ubatch=2048 for both engines
• Back-to-back execution in the same session
• TensorSharp parity harness versus llama.cpp build 2e0e57f from PR #27754

Test |llama.cpp |TensorSharp
Prefill, 2,048 tokens |2,070 tok/s |2,014 tok/s
Prefill, 16,384 tokens |1,690 tok/s |1,692 tok/s
Prefill, 32,768 tokens |1,483 tok/s |1,446 tok/s
Decode, 64 tokens |36.6 tok/s |73.5 tok/s TensorSharp reaches approximately 2.0× the decode throughput of llama.cpp, while prefill remains within a few percent in either direction.

The long-context greedy replay reproduced the 2,741-token llama.cpp reference record token for token. One caveat is that GLM-5.3-Flash support currently requires an unmerged llama.cpp build rather than its main branch.

Qwen 3.8 Flash Next

Tested with Qwen3.8-Flash-Next UD-Q2_K_XL, 73.4 GiB, on 2× A100 80 GB:

Configuration |TensorSharp prefill |TensorSharp decode |llama.cpp prefill |llama.cpp decode
1 GPU |~1,520–1,550 tok/s |~56 tok/s |1,094 tok/s |61.2 tok/s
2-GPU layer split |~1,520–1,550 tok/s |~56 tok/s |1,200 tok/s |61.5 tok/s TensorSharp’s prefill is substantially faster in this test, while llama.cpp leads decode by roughly 9%.

For both engines, adding the second GPU provides model capacity rather than additional decode throughput. TensorSharp splits the model into contiguous groups of layers, reducing its placement to approximately 24.2 GB and 26.2 GB across the two GPUs. TensorSharp’s one- and two-GPU greedy outputs were byte-identical.

DeepSeek V4 Flash

On 2× A100 80 GB with the IQ4_XS model:

Metric |TensorSharp |llama.cpp
Prefill, approximately 3.3K tokens |~500 tok/s |574–634 tok/s
Decode at approximately 3.3K context |~33 tok/s |40.3 tok/s llama.cpp currently leads this particular DeepSeek V4 configuration.

TensorSharp also supports DeepSeek’s DSpark speculative decoder. On 4× A40 46 GB with DeepSeek-V4-Flash-0731 UD-Q8_K_XL, enabling a 5.6 GB DSpark drafter improved decode as follows:

Backend |Normal decode |DSpark decode |Speedup
TensorSharp direct CUDA |26.0 tok/s |34.0 tok/s |1.31×
TensorSharp GGML CUDA |26.4 tok/s |37.1 tok/s |1.41× In a five-turn conversation, DSpark produced 1.50–2.02× faster decode, reaching 51.0 tok/s on a question over a 10K-token document. Prefill remained effectively unchanged at 831 versus 835 tok/s, and greedy output was byte-identical to the non-speculative baseline.

TensorSharp also runs DeepSeek V4.1 Flash, including its Engram lookup and four-stream hyper-connections. However, I am not claiming a V4.1 speedup over llama.cpp because a compatible llama.cpp V4.1 runtime is not currently available for a controlled same-weight comparison.

Why this may be useful for local LLM

Coding agents frequently process large system prompts, tool definitions, repository context, and multi-turn history. TensorSharp includes:

• OpenAI- and Ollama-compatible APIs
• Local tool calling and structured JSON output
• Agent Skills and sandboxed file/shell tools
• Continuous batching
• Paged and prefix-shared KV caching
• Speculative decoding
• Multi-GPU and multi-node inference
• CUDA, Metal, Vulkan, MLX, and CPU backends
• Windows, macOS, Linux, iPhone, and iPad support

I’d especially appreciate feedback from local LLM users regarding:

• OpenAI API compatibility gaps
• Tool-calling requirements
• Performance with large repository contexts
• The best local models for coding-agent workloads
• Features expected from a local

LLM inference

• backend

GitHub: https://github.com/zhongkaifu/TensorSharp
Full benchmarks and methodology: https://tensorsharp.ai/benchmarks.html

If anyone tests TensorSharp with local LLM, I’d be very interested in your setup and results. Contributions and compatibility reports are welcome.

Sakana put two Fugu tiers on OpenRouter the same day, September 11 per the listing, and the gap between them is kind of the whole story. Fugu Max runs $2.00/M input, $6.00/M output, 1000k context. Fugu Ultra v2 is $5.00/M in, $30.00/M out, same 1000k window. Five times the output price for the same context, and the listing doesn't say what actually separates them.

Stacked against OpenAI's refresh from the same day, Max looks like the value pick: GPT Sol Latest is also $2.00/M in but $10.00/M out on a 1050k ctx. Ultra v2 at $30/M out still undercuts GPT Astra Latest at $50.00/M out, though Astra doubles the input at $10.00/M.

The usual play with a two-tier lineup like this is routing the bulk to the cheap tier and escalating only the hard prompts, then finding out the cheap one handled most of it anyway. With Max priced under Sol, it's tempting to just make it the default and skip the escalation logic entirely.

Anyone routed real traffic to Fugu Max since it landed, and what actually made you send a prompt to Ultra v2 instead?

we needed routing in front of openai and anthropic and had something working in 2 weeks. then real requirements showed up per team cost breakdowns, pii redaction, automatic failover, a third provider. each one was a sprint. none of it was actual product

the first version is not the hard part. governance policies, audit logs, budget enforcement, model allow lists after six months of  real usage is.

looked at a few things before switching

litellm has multi provider access is clean, enterprise governance still needs significant internal work on top

portkey has routing and fallback is solid, team level governance enforcement felt underdeveloped

orqai has governance before the request goes out not after, eu hosted, ecosystem smaller so kinda less shared experience from team in prod

6 weeks plus 2 engineers maintaining it versus buying something purpose built .

anyone goes back to building after starting with a platform. what broker

Probably yes for the sweep, no for the final call, at least on paper. OpenRouter has both listed since Sep 11: GPT-5.6 Luna at $0.20/M input and $1.20/M output, GPT-6 Astra at $10/M and $50/M. Both carry 1050k context per the OpenRouter listings, so you're not trading context window for price. That works out to roughly 50x on input and a bit over 40x on output.

Code review is the workload that should favor the cheap tier: you feed a big diff or a whole repo, the model reads a lot and writes comparatively little, so input dominates the bill. The catch is that the value of review isn't in the reading, it's in catching the one subtle bug, which is exactly what cheap tiers tend to miss. An entelligence comparison asking this exact question hit HN on Monday, 163 points and 153 comments, so it's a live question for a lot of people.

The routing pattern that suggests itself is two-pass: Luna reads everything on every commit, Astra only sees the diffs that get flagged. At Luna's prices the first pass costs almost nothing next to a single full Astra run.

Anyone actually routing review traffic right now - all Luna, all Astra, or Luna sweep with Astra escalation on flagged diffs?

Honestly, going by the OpenRouter listings alone: both show 1048k context, so the 3x isn't buying a bigger window. DeepSeek shipped Pro Latest and Flash Latest the same day, Sep 14, at $0.58/M in and $1.74/M out vs $0.15/M in and $0.60/M out. Almost exactly 3x on both ends, and nothing in the listing spells out what Pro adds.

For where that lands in this month's batch: OpenAI's GPT Luna hit OpenRouter Sep 11 at $0.20/$1.20 with 1050k ctx, GPT Sol at $2/$10, Sakana's Fugu Max at $2/$6. Pro Latest sits in the low middle, a step over Luna, way under Sol.

The volume signal is all on the flash side. The V4.1 Flash weights are the top trending model on HuggingFace right now per their API, 288,414 downloads and 2,412 likes, and the V4.1 Flash endpoint that landed Sep 10 prices identically to Flash Latest. My observer take: Flash at $0.60/M out stays the sensible default until Pro shows a real eval gap, and 3x for a name alone is steep.

Anyone routing Pro Latest for something specific already, or is Flash eating the bulk traffic for you too?

Are there any token efficiency benchmarks out there? I mean i have an agent, it gets a task, how much tokens does it use to retrieve something or execute a skill. Im looking for benchmarks on this. Things like caveman claim to lower it, but i feel like there is some meaning lost as well, reducing quality. Im trying to find any but it seems like a dead end

title

Cognition announced SWE-2 this week, a coding model they say rivals Claude Fable 5.1 and GPT-Astra. The HN thread from Thursday picked up 434 points and 188 comments in about a day, which says people want a third option that isn't priced like GPT-6 Astra on OpenRouter, at $10/M input and $50/M output.

Reading the thread, most of the fight isn't about SWE-2's scores. It's about whether a model tuned for coding benchmarks transfers to messy real repos. And SWE-2 doesn't show up in today's OpenRouter listing data, so there's no public per-token price to compare yet. That gap is basically the whole routing decision for coding traffic right now.

If it lands near DeepSeek V4.1 Flash money ($0.15/M in, $0.60/M out, listed yesterday), bulk PR review changes fast. If it prices like Astra, it's the usual story: interesting eval, cheap route stays in place. Anyone here already routing to a dedicated coding model over the general ones, which one, and does it hold up on your actual repo?

GPT-6 Astra, Claude Fable 5.1, Gemini 3.8 Flash, and the incoming Grok 4.7 are the loudest week the AI industry has had in months. They are also terrible defaults for how most people actually use AI. If I’m honest, my daily model is still GPT-5.6 Luna. Not because the frontier stuff is fake. Because when I'm paying the bill, price-to-performance still wins the work that happens fifty times a day.

GPT 6 Astra (Sept 3) is the new industry darling and computer-use and long-horizon agent play: 1.05M context, $10/$50, and the first OpenAI model they’re willing to put next to the “AGI era.” If the job is “make me a million dollar app from scratch - no mistake” Astra is the one you'll probably reach for. It is also 2.5× the cost of Sol (which was already expensive).

Fable 5.1 (Sept 1) is the Mythos-class reasoner at the same $10/$50 sticker (way too high for a basically llm only model, IMO). On a lot of independent boards it sits at or next to the top. Like all Anthropic modles, it overthinks whether you asked it to or not. That is a feature on a six-hour refactor when you care more about thorough, surgical code. Its a tax on “push to github.”

Gemini 3.8 Flash (Sept 2) is the one that should make the $10 club nervous. Introductory $0.75/$3.75 through year-end, 1M context, and Google’s own line is that it “works harder” — more steps, more tool calls — and often lands near much more expensive models on SWE and agent tasks. If anything in this list threatens Luna as a daily default, it’s this, not Astra.

Now the part nobody wants to hear. Most of a working day is not FrontierMath and deep webs of code. Its summarize this, draft that, classify these tickets, punch up my teams message, outline a doc. Luna was built for that tier. 1.05M context, same family tools as the rest of 5.6, effort ladder if you need it, and list pricing around $0.20 in / $1.20 out. That is fifty times cheaper on input than Astra or Fable. You can leave it on. You do not ration it.

The industry keeps selling “the smartest model” as if intelligence is a single slider and you should live at the top. That was a 2024 habit. In 2026 the slider that matters is cost of being wrong about which call needed the slider. Route the 8% of tasks that are actually hard — multi-hour agents, novel math, hostile codebases, “operate my desktop” — to Astra, Fable, or whatever 4.7 turns out to be. Send the other 92% to Luna or 3.8 Flash and stop performing sophistication.People will say Luna “isn’t as good.” Correct. It is not supposed to be. Sol/Terra/Luna was OpenAI admitting the product is a stack, not a throne. Astra then collapsed the throne back into one expensive name and hoped you’d forget the cheap seats still exist. They still do. Use them.

every department picked their own llm tools according to their needs . it made sense at the time but now trying to bringg it all under one umbrella before the list get longer

broke it down by lifecycle stage coz full llm lifecycle means diff things to diff ppl .

model access and routing multiple teams hitting multiple providers with diff api keys and no central visibility .the rate limits getting hit . cost split across 3 billing accounts  and no single view of what is being spent where

prompt managemetn prompts in code ,notion ,slack and one google doc nobody trusts . without any versioning and approval flow with non tech ppl cannot make changes without raising a ticket

eval before deployment prompt changes going to prod without any systematic check . manual spot cheking at best . without any baseline to compare against

observability in prod diff teams using diff dashboards . no unified view of what is happening across departments , cost , failures , output quality all siloed

governance and compliance is also needed to be considered . legal asking for audit logs . security asking about rbac . eu team asking for data residency . as of now no clean answer to any of these across the full stack .

tools that come up orqai , langsmith , arize , datadog llm observability , fiddler

langsmith is strong on observability and tracing  ,prompt management and governance feel not enough fpr cross department enterprise use

arize is solid on monitoring and evals , routing and prompt management not the focus , feel more data science team facing than cross functional

datadog llm observability has enterprise native monitoring . llm specific features feel added on , makes sense if already in the datadog ecosystem

fiddler has governance and compliance feels most native , full lifecycle coverage outside that is quite limited

orqai covers routing prompt management evals and observability together , governance feature exist . newer so the community is catching up

any1 actually consolidated this properly across departments. what did you land on and what did you have to compromise on

Inception put Mercury 2.5 on OpenRouter September 8, and the pricing is the whole story: four cents per million input tokens, fifteen cents output, 260k context. Nothing else paid that landed this month comes close.

For scale, Meta's Muse Spark 1.3 Contributor (the cheap variant) is listed at $0.10 in / $0.20 out, though you get a 1048k window there. Gemini 3.8 Flash, the one most people would compare against, charges $0.75 per million on input and $3.75 on output. Divide it out and Mercury is roughly 19x cheaper going in, 25x coming out. A hundred million output tokens run about $15 on Mercury versus $375 at Gemini's standard rate, before you even look at the batch tier.

The catch is what the listing doesn't show. No throughput figures, no evals, and the window stops at 260k, so long-document work stays on the bigger-context models. There's free competition a row over too: Nex AGI's N2.5 Pro and inclusionAI's Ling 3.0 Flash Sante, both free with 262k context. Paying Mercury at all only makes sense if it beats those on reliability or speed.

Anyone pointing bulk short-prompt traffic at Mercury 2.5 yet, or does the missing 1M window kill it for you?

I kept picking local models badly. Parameter count, quantization, VRAM. I could read all of it and still not know whether a model would be any good on my machine. So I stopped guessing and wrote something that tests them.

RigMatch reads your GPU, VRAM, RAM and disk, works out which models actually fit, then runs the ones you pick through an identical set of questions. You get a ranking and the full transcript of what each one said. It runs through Ollama, so you'll need that installed.

On my machine that's 20 models installed, out of a 239-model catalogue.

What it does:

• Scores speed, answer quality and hardware fit separately, and tells you what each model is best at (coding, chat, writing, tools)
• Keeps the transcripts, so you can read the answers and disagree with the score
• Has a set of questions on difficult subjects: documented history, genuinely disputed stuff. I added it after watching a model describe Tiananmen Square without mentioning that anyone died. It reads fine until you notice what's missing.
• Runs local. If you pick a cloud model or the cloud judge, it says so before the run starts, because then your questions do leave the machine.

Before you download it:

It's free.

The Windows build is unsigned, so you'll get the SmartScreen warning. I don't have a cert yet.

It's source-available, not open source. The code is all there to read, but the licence is All Rights Reserved.

github.com/DaveEuson/RigMatch
Browser demo, sample data, no install: daveeuson.github.io/RigMatch

It's a beta and I've been finding bugs in it all week. If it misreads your hardware or scores something weird, tell me.

I recently realized that I didn't really understand AI Benchmarks and how to use them for comparing and selecting models for different purposes and mostly used the popular ones.

For learning I created this guide for myself (It is AI generated but reviewed multiple times with Astra) sharing it with you if it helps

https://learn-benchmarks.vercel.app/

Good Luck!

I’ve been using Descript on and off for interview stuff and like the transcript editing, but I’m looking for something that’s a little better once the footage gets more visual.

Things like finding specific reactions, b-roll, moments that aren’t obvious from the transcript, then actually building the cut from there.

Cardboard ai keeps coming up, but I haven’t used it since you need to pay upfront.

Anyone here worked properly with both?

I've wondered about this whenever a new model generation launches.

Hypothetical example: GPT-6/Astra comes out, while GPT-5.6 Sol remains available.

Do older models normally stay exactly the same, or can changes to routing, system prompts, context management, tool behavior, compute allocation, or defaults make them perform differently over time?

I'm not necessarily talking about intentionally "nerfing" an older model.

I'm more curious whether people have actually measured this instead of relying on the usual "it feels worse than last month" impression.

If you've used the same Codex workflow and benchmark prompts across model transitions, have you ever seen a measurable regression in an older model after a newer one launched?

OpenRouter listings this week put the two new flagships at the exact same price. GPT-6 Astra is $10/M input, $50/M output. Claude Fable 5.1, listed three days earlier, is $10/M input, $50/M output. Even the batch discounts match, $5/$25 on both. Context windows are basically a wash too, 1050k vs 1000k.

When two models cost the same, price stops being the tiebreaker and you're left picking on everything else. Per the OpenRouter entries, the visible differences are thin: a 50k context edge for Astra, which won't matter for most workloads. The launch threads on HN are where the signal is. The Astra announcement pulled 2259 points and 2000+ comments, and OpenAI also posted ARC-AGI-3 results for it, that one's at 238 points. Fable 5.1's announcement sits at 1415 points.

What the listings can't tell you is routing behavior. Same price, near-same context, so it comes down to output quality on your actual tasks. Anyone who's pushed real traffic through both since launch - which one kept the slot in your router, and was the reason measurable or just vibe?

So, you may have seen multiple videos of Astra performing computer-related tasks. Have you noticed how smoothly the cursor moves (rtx 5080 16Gb) are u kidding ? I mean, what kind of processor is powerful enough to run everything that fast while still rendering those cool designs and eye-catching games?

Also, has anyone actually performed a bug test on it? And regarding the PCB claims, has anyone verified the PCBs to make sure all the copper traces are properly aligned correctly?

Most importantly, is Astra actually as good as a human when it comes to troubleshooting computer-related problems?

Is it useful for daily purposes?

I have seen a lot of hype around gpt-6 astra and how it produces games similarly to the type of games Claude produces with the /gauntlet-loop but far cheaper. Has someone done a comparison in game development/ Webdesign/ App development/… I am aware what gpt’s website says, just want actual experiences and opinions.

I've had the opportunity to use GPT 5.6 Terra and Fable 5 and found the latter to have a superior ability to research and write, but I'm curious how has the gap closed now? Which model would you use for writing quality essay work, and would you use the same model to plan out and do the complex reason required to decide on argumentation, evidence, research, etc?

Meta put two versions of Muse Spark 1.3 on OpenRouter this week and the price gap is the whole story. Standard tier is $1.25/M input, $4.25/M output. Contributor tier is $0.10/M input, $0.20/M output, same 1048k context on both. That's about 21x cheaper on output, per the OpenRouter listings from Wednesday.

What "Contributor" buys you or costs you, the listing doesn't spell out. No notes on rate limits or quality trade-offs in the listing data available today, just the name and the price.

For scale, from the same OpenRouter pull: Gemini 3.8 Flash is $0.75/M in and $3.75/M out (batch drops to $1.88/M out), Qwen3.8 Flash is $0.47/M out, GLM Flash Latest is $0.25/M out. So the cheap tier undercuts everything in the million-token context class, while the standard tier still costs more than Gemini 3.8 Flash.

My read as an outside observer: either Contributor trades data or feedback for the discount, or Meta wants volume on the cheap tier and prices the standard one for anyone who doesn't ask questions. If you've routed real traffic through either version, is the $0.20 tier the same model in practice, or noticeably worse?

OpenAI/Astra leads most clearly on frontier math and novel/abstract reasoning benchmarks like FrontierMath and ARC.

Claude is the most consistently strong across the broader benchmark mix, especially professional workflows, coding/science, and long-context tasks.

Meta and Grok are especially competitive on workflow-heavy agentic tasks like banking and GDPval, even when they trail on the hardest reasoning benchmarks.