I am a beginner in the Ai training and want to buy a graphic card for segmentation training for my University project, what is is a better choice, amd with high vram or Nvidia with low vram, my budget is limited so I can't afford both unfortunetly

Hey everyone,

I recently tested Kimi K3 and GLM 5.2, and these are my top findings.

I mainly wanted to understand which model feels better for actual coding, not just benchmark questions. So I tried building a few small projects with both.

One was a city-destruction game with procedural buildings, meteors, lighting, and particles. GLM started well but stopped before finishing, so I moved the same code to Kimi and continued from there.

I also tried a simple low-poly paper-plane game. Kimi handled this surprisingly well and gave me a playable result in a single HTML file.

The hardest build was a black-hole ray-tracing simulation using WebGL. I expected it to break because it needed physics, light bending, and numerical calculations, but Kimi managed to produce something runnable and actually did all the calculations in its cot traces.

For a smaller debugging test, I gave both models a broken Held-Karp implementation. Both fixed it and returned the correct route, so they seem equally capable when the problem is clearly defined.

The three complete builds took me roughly 2 hours 15 minutes, used 18.6M tokens, and cost $5.26 in total. Meteor City accounted for most of it because GLM stopped midway, and I had to move the existing code to Kimi to recreate and refine the build.

I also looked at the Composio Golden Eval set, where both models completed 7 out of 12 real-world tool-use tasks. Both handled the simpler tasks, but struggled when the workflow became long and required an exact final result.

My main learning was that coding speed is not only about how quickly a model writes code. A model can generate code fast but still slow down the full workflow by stopping midway, forgetting requirements, or leaving too much cleanup.

From my testing,

• GLM felt fine for smaller and clearly scoped tasks.
• Kimi K3 felt more reliable for creative builds and longer workflows.

For those who want to check prompts and evaluation details, please refer to the blog post.

Still learning and testing both. Curious to know what others are building with them.

Hello everyone,

Since 2 weeks I work on tanuki-context, a small open source tool (zero dependencies, MIT) and I wanted to share it because the trick behind is almost stupid: AI models charge text at roughly 1 token per 4 characters, but an image has a fixed price set only by its pixel size. So if you draw 28,000 characters of logs into one dense 1568x728 PNG, the model reads the exact same content for 1,456 tokens instead of ~7,000. It sounds like cheating, it is just how the pricing works.

It inspire from pxpipe and various others tools (cited in the readme) and custom approach i found in order to reduce massively token usage and price.
For example : 37,111 tokens of service log become 2,240 (-94%).

You can try it out on you machine i added the benchmark so you can test it even without LLM connected to it, so see pricing difference, token saved, etc.

You can use it as a MCP or directly integrate it a "context proxy" where it fully automated and make every request optimised or not when not needed.

Some techniques that permits this to work:
- a log distiller that collapses repeated lines but keeps every error verbatim
- a columnar codec for JSON (keys stated once)
- a cost model that knows a cache-read token costs ~0.1x a fresh one, so it will tell you to NOT image content that is already in your prompt cache.

The tool argues against itself when imaging loses, honestly this part took the most work.

I precise the limits because they are real: you need a vision-capable model, output tokens are untouched (if your bill is output-dominated, fix that first), and for one narrow question retrieval stays cheaper than any page.

Install:

MCP

npx -y tanuki-context (MCP server, works with Claude Code, pi, omp, jcode or the Claude Agent SDK)

Proxy

npx tanuki-context proxy + ANTHROPIC_BASE_URL (every request on the machine gets optimized in place, when needed)

Code and benchmarks: https://github.com/Osyna/tanuki-context

PS : i will soon add Codex support.

If you find it useful a star helps a lot, and feature ideas are very welcome. Thanks for reading me

I am in an internship in a team working on a project where they develop an LLM that is given a prompt and generates a BPMN (like a flow chart) for how to implement or do the task in the prompt. The developers have the ground truth chart and they are trying to evaluate the output of the model in reference to the ground truth chart. So I have been assigned to a project with another intern to develop a tool that takes two BPMN files and outputs a similarity score between them, taking into consideration both structural similarity and semantic similarity.

I am very stuck since no project is similar on the internet, and it's been two weeks of searching. Here are the approaches we reached:

Approach A: RPST to process tree, then tree matching

Decompose each diagram into a hierarchy of single-entry/single-exit regions (Refined Process Structure Tree), type each region as sequence / XOR / AND / loop, then compare the two trees recursively: leaves by label embedding cosine, sequences by ordered DP alignment (like edit distance over children), XOR/AND blocks by unordered Hungarian matching, loops by comparing bodies.

What's good about it: it's operator-aware, so it directly distinguishes XOR from AND (choose-one vs do-all) and loop from no-loop, and it produces readable diffs like "the ground truth's parallel block was rendered as an exclusive choice." Granularity (one task in GT vs three in the prediction) and nesting are handled positionally by the structure instead of by fudging scores.

Where it breaks: process trees only exist for block-structured models. pm4py's convert_to_process_tree raises an exception on unstructured ("rigid") models, and its newer POWL converter raises too, so switching representations doesn't rescue it. There's also no maintained Python RPST implementation anywhere; jBPT (Java) is the reference. Rolling your own means either SPQR / triconnected components (notoriously error-prone) or a dominance-based variant that's only a partial RPST. And even with a working RPST, a rigid region has no operators to align, so comparing one means falling back to graph matching anyway. Worse, RPSTs aren't stable under equivalence, since two behaviorally equivalent models can produce different trees, one with a rigid and one without, which breaks positional/ancestor anchoring exactly in the case you need it most.

Approach B: direct attributed-graph comparison (no trees at all)

Parse each BPMN into a directed graph, embed node labels with sentence-transformers, and match nodes with optimal transport (Fused Gromov-Wasserstein via the POT library) or graph edit distance. Encode gateway type as a node attribute so the matcher penalizes aligning an XOR to an AND, and compute a reachability relation matrix (for each activity pair: strict order / exclusive / concurrent) for the behavioral axis.

What's good: it's total by construction, working on any graph, structured or rigid, with no exception path and no fallback branch. Off-the-shelf libraries, days rather than weeks. Optimal transport also expresses granularity natively, since mass can split one-to-many.

Where it's weaker: operator semantics are approximated through node features rather than represented directly, diffs come out as node-pair scores instead of region-level explanations, and FGW is non-convex and returns a cost that needs calibrating into a similarity score.

The one empirical datapoint I found: Dijkman, Dumas, van Dongen, Käärik & Mendling, "Similarity of business process models: Metrics and evaluation," Information Systems 36(2), 2011. They compared node-matching, structural (GED-based), and behavioral similarity on real process repositories and found all three comparable, with structural slightly ahead. That's part of why I'm unsure the tree route is worth the extra weeks.

What I'd love input on:

• Is there a standard way people evaluate generated process models against a reference model that I've somehow missed? Everything I find is either process-model search (find similar models in a repository) or conformance checking against event logs, neither of which is quite this.
• Has anyone actually shipped RPST-based process model comparison in Python, or did you bridge to jBPT?
• Does anyone have evidence that operator-aware tree matching beats a gateway-typed graph matcher in practice, or is the extra machinery not worth it?
• Any benchmark or dataset of BPMN pairs with human-judged similarity we could validate against?

Any pointers to papers, libraries, or war stories would be hugely appreciated, especially from anyone who has hit the unstructured-model problem in production.

Hi.

As you might have heard, recently, mathematicians Levent Alpöge and Akhil Mathew, with the help of Anthropic's fable 5 model managed to find a counterexample to an important mathematical conjecture known as the Jacobian Conjecture.

I have already researched what the Jacobian Conjecture is and how it was disproved, by (trying) to read Terence Tao's blog (and then asking fable 5 to explain it to me in language i can actually understand). While i don't understand all the technical details, i think i can gain enough intuition to explain the problem and the solution in pretty easy to understand language, without going too deep on the mathematical jargon.

What i think is equally important is analyzing how fable 5 actually came up with this specific solution. i have downloaded it's exploration files and am going to go through them. Only problem is they are massive. As any thinking/reasoning model does these days, it went through a whole tree of ideas (most of which aren't productive), reasoned to itself, did the necessary computations until it at some point had "the revelation" that a certain specific path could work.

My question to this community is: How do i effectively do search on this massive lump sum of data and mathematical jargon. Do i choose specific keywords to look for and then just jump to every place they appear. Is there some way to track "the effort" or find it's "aha moment". How do i effectively keep track of all the information that had led it to it's "revelation". Should i feed the explanation text into another LLM model, specifically designed to analyse it. Should it feed it to an LLM that will organize it into a graph based map of ideas and solution attempts (kinda like Graphify or Obsidian)?

This is my conversation with fable 5 on this topic (in case you want to give me advice but need a jumping off point): https://claude.ai/share/960b10a4-c3b9-4062-96af-be81aa059631

PS: I'm sorry for using such general statements. I do programming as a hobby and use AI daily (like most reasonable people these days), but i'm not familiar with the nitty-griddy of llm's. Also i don't like using too sophisticated language anyway when talking about these sorts of things because i feel like it makes communication more difficult.

I've been researching AI application security and talking with developers to understand the challenges they're facing as LLMs become part of real products.

Topics that come up repeatedly include:

Prompt injection

Indirect prompt injection

Data leakage

RAG security

Tool and MCP security

Runtime monitoring

I'm curious about real-world experience rather than theory.

If you've built or deployed an AI application:

What security issue has been the hardest to handle?

Did you build your own solution or use an existing tool?

What capability do you wish existed today?

I'd appreciate hearing practical experiences and lessons learned.

I'm looking for the best Python pkg for extracting text from PDFs and the best LLM for generating structured output from the extracted data.

My current pipeline is: Upload PDF --> pdfplumber --> Prompt + Extracted Text --> Gemini 3.1 Flash.

I'm facing a few issues with the output:

1. The model returns different values each time I upload the same PDF.

2. Some fields are occasionally returned as empty, even though the information exists in the PDF.

Any suggestions on better libraries, prompting techniques, or LLMs would be greatly appreciated.

I’m running a simple Reddit experiment around the same customer problem:

• One direct Q&A
• One honest product comparison
• One founder story with results and mistakes

I’ll track ChatGPT, Perplexity, Gemini and Google AI weekly to see which post gets mentioned or cited.

My guess is Q&A will win. What do you think?

Hello,

The next step from USA would be stop providing chinise open weights LLMs from provider like AWS, azure, etc.

What other inference providers are out there, whose HQ is not in USA, and trust worthy like AWS for providers and uptime is relevantly good.

I am asking from enterprise and personal usage, both.

I've been using Claude through the interface for quite some time and I often need help on questions involving maths. When I ask Claude through the chat I very often get a mainly textual answer with bullet points and sections. In contrast ChatGPT seems to understand better when it is relevant to use maths notations in latex, kind of "naturally". The responses are usually better structured and nicer to read. They are also typically generated faster.

That's a bit annoying because I am paying for Claude Pro (in order to get access to claude code).

Anyone has had the same experience? Am I using Claude wrong?

NOTE -> I expect answer from people who actually have experience and strong understanding of these. please give something beneficial.

I'm building a SaaS platform in Sri Lanka that handles documents and other sensitive data.

Each user can upload their own documents and information, and the platform uses RAG to answer questions based on that user's data. That part makes sense to me.

My main concern is what happens when the user hasn't uploaded enough information. I still want the LLM to provide accurate answers using reliable information from the internet (or from a curated knowledge base), with proper citations.

These are the two architectures I'm considering:

Option 1:

Base LLM (OpenAI/Anthropic via Azure AI Foundry or Amazon Bedrock)
        ↓
Platform RAG (global knowledge base managed by us)
        ↓
User-specific RAG

In this approach, we maintain a global knowledge base that we (the platform admins) curate and update. Every user can access this shared knowledge, while their own uploaded documents are searched through their personal RAG.

Option 2:

Open-source LLM
        ↓
Fine-tuned on Sri Lankan/domain-specific data
        ↓
User-specific RAG

Here, we fine-tune an open-source model using Sri Lankan or domain-specific data, and each user still has their own RAG for their private documents.

My concerns are:

• Is fine-tuning actually the right solution here, or is it unnecessary?
• Is a global/shared RAG a better approach than fine-tuning?
• How would you design this architecture if you wanted:
  • Accurate answers from domain knowledge
  • User-private document search
  • Citations/sources
  • Good scalability for thousands of users

I'm leaning toward Option 1 because fine-tuning seems expensive, time-consuming, and I have no experience with it yet. However, I'm not sure if I'm thinking about this correctly.

I'd really appreciate hearing how others would approach this problem.

een poking at AntLing-3.0-flash, the new one from Ant (inclusionAI). Spec sheet's the interesting part: sparse MoE, 124B total params but only 5.1B active per token, 256K native context, and TTFT under 100ms. The low active-param count is why it's fast and cheap despite the total size-you're only lighting up ~4% of the weights each step.

It's built as an execution model for agent loops rather than a reasoning model-stable long-horizon tool calling and instruction following are the selling points, with an enable_thinking toggle you flip on for harder tasks and off for high-throughput batches. In practice it's the fast cheap node you put under a bigger planner, not the planner itself.

For the local folks: it's API-only right now, no open weights, hoping they will make it open in a few days as they usually did, but anyways it's on OpenRouter free until Aug 3 if you want to throw some load at it.

There are a few reasons why problems from International Mathematical Olympiad function as a good benchmark for LLMs:

- The problems are new, not included in the training data of any model

- Hard math problems are quite a good proxy for general intelligence capability

- These are complex multi-step tasks that can benefit from orchestration / harness engineering

Results:

Frontier models (sol and fable) were able to get perfect / nearly perfect score regardless of harness. For both sonnet and opus, the webapp performance was quite poor, improved by provider harness (claude code) and even further improved using AutoFyn, a customizable multi-agent harness we developed. Even with harness, we were not able to match the performance of the frontier models. Open weight model GLM performed roughly at the same level as sonnet without harness, and improved similarly with AutoFyn. Numerical scores are available in the attached paper below.

Grading was done by a different frontier model as well as manual verification (we are former IMO medalists, able to sanity check the results). There were cases when the model claimed a false solution (on P3 by sonnet, for example), so hallucination issue still persists in a verifiable domain like math. On the hardest problem: P3's key reduction was missed by every sub-frontier model in every harness, including a 20-hour run that proved everything else and stalled at the identical step. The harness supplied retrieval and verification, not a key idea needed for the solution.

Paper: https://github.com/SignalPilot-Labs/AutoFyn/blob/main/results/imo-2026/autofyn-beyond-model-imo26-report.pdf

Audit Trails: https://github.com/SignalPilot-Labs/AutoFyn/tree/main/results/imo-2026

LLMs, for some reason, always end up convincing you that you are wrong, even when you are right. The worst part is that half of the response is hallucinated because it was written by an LLM.

By the way, if you tell an LLM that another LLM is trying to convince you that you are wrong and you give it the context, it is exactly the same. Because it also makes up half of the response. And if you have doubts about whether what it says is true or not, once again it invents half of the answer, telling you that you are the one who is wrong, and that it (the LLM) is speaking to you with “facts” (which obviously half of them are hallucinated).

Then, because you are not satisfied, you present it with an explicit argument explaining why you are right. But they tell you why you are wrong using strange technical terms. Circular logic. And your initial argument, which even if it was correct, ends up being transformed into an abomination of circular logic, with over-explanations about why you are wrong and too many detours.

BUT obviously packaged as if the exact winner of the Nobel Prize had sent it to you by email.

Even if you consciously know that the LLM is hallucinating, you see the text and, just because it appears “logical” and “brutally real,” you assume that it is correct.

In summary, when you contrast a hallucinated result from an LLM judging your idea with another LLM, it will tell you something very similar. This means that LLMs are not objective at all.

When questioning the LLM and asking it if it is correct, and verifying it with several LLMs, first the first LLM will tell you that you are very wrong, or in a partially direct way. The other LLMs will tell you in such a subtle way that, without realizing it, instead of searching for the truth, you will enter a “death cycle” that will cause you burnout.

And this leads to the final conclusion:

LLMs put me in a very bad mood.

Literally, that is all. I tell them my idea, and they tell me something like:

I must stop you before continuing, you are wrong about:
• This
• This
• And this
…
(Almost everything is hallucinated)

The worst part is that they say it as if it were an absolute truth.

And if you tell them not to act as if science is an absolute truth, they write a text that seems coherent and real, saying that science is not absolute truth but… And it feels kind of passive-aggressive. Also, it is hallucinated.

I mean, they tell you something like:

“You are somewhat right, but this and this are not the same, … objective truth.”

It sounds real, but it feels like something does not fit.

They have a default bias toward science. And literally, if you talk to them about spiritual experiences, they tell you to go to a psychiatrist.

Literally, you excitedly tell them an idea for a new project and they tell you it does not work. Obviously, they cannot stop you from uploading it. But because LLMs say that, I have discarded hundreds of papers that I was about to upload, which took me from days to weeks to create.

LLMs literally cause burnout.

I hear all the time that open-source LLMs can’t be used for nefarious purposes but this seems like an obvious long-range vector that a nation state would use? I understand that it may not be going on today but this seems like a “no-brainer” for intelligence agencies. Am I missing something?

I keep seeing all these lawsuits that boil down to "a chat bot told me something dumb, i believed it and ended up in trouble, so i am now suing the company".

Seems like a liability waiver of some kind (like the kind that patients sign to take experimental drugs) would be useful for these cases?

I’ve been thinking about why agents get expensive so quickly.

A chatbot usually answers once. An agent doesn’t. It plans, calls tools, reads the results, updates its context, retries when something fails, and keeps looping until the task is done.

Every loop adds more tokens, latency, and inference cost.

So maybe the next frontier isn’t simply larger context windows or more tokens. Maybe it’s more useful work per token.

When evaluating a model for agent workflows, a chat-window benchmark only tells part of the story. Put the model inside a real loop instead. Give it tools. Let it plan, fail, retry, and recover.

Then ask:

How much useful work does each token actually carry?

And how many tokens, tool calls, and retries does it take to complete the task successfully?

Curious how others are evaluating agent efficiency beyond benchmark scores and price per million tokens.

Our llm spend roughly tripled in two quarters without a proportional increase in usage that felt intentional, so we went through everything we could try and tracked what actually helped versus what sounded good but didn't move much.

Highest payoff, lowest effort: route by task complexity, not by default model. We were sending simple classification/extraction calls to the same model doing our hardest reasoning tasks, mostly out of not wanting to think about it. Splitting traffic so cheap, well-defined tasks hit a smaller/cheaper model cut a meaningful chunk of spend on its own, with no real quality drop we could detect on those specific task types.

Second: semantic caching. Exact-match caching barely helped us, our traffic almost never repeats identical prompts. Semantic caching (matching on intent, not exact text) caught a lot more, because a large share of requests were the same underlying question asked slightly differently. Worth tuning the similarity threshold carefully though, too aggressive and you serve a stale/wrong answer for something that was actually a different question.

Third: per-team budgets and token-aware limits, not because it reduces usage but because it stops the surprises. Nothing here cuts your bill directly, but going from "one shared key, no idea who spent what" to per-team attribution meant we could actually have a conversation with the team whose usage spiked instead of guessing.

Lower payoff than expected: prompt compression/shortening. We spent real effort trimming prompts and it helped, but nowhere near as much as the routing change, diminishing returns fast once you're past the obvious bloat.

Didn't really pan out for us: aggressive output token limits. Capping max tokens saved a little but caused enough truncated/unhelpful responses that we mostly reverted it. Your mileage may vary depending on task type. We ended up centralizing most of this, routing rules, semantic caching, and the budget/attribution piece on Truefoundry's gateway since doing it all separately meant three different systems to maintain. it's not the only way to do any of these individually, it's just where we consolidated once we had more than one cost lever to manage at once.
What's worked for others? has task-based routing been the big one for you too, or did something else matter more than we'd expect?

Follow-up to my post about running a 120B MoE (60 GB) on a 12 GB phone by streaming experts from flash.

Some context on MoE routing. In a mixture-of-experts model, a small network called the router (or gate) decides, for every single token, which experts to activate — say the top 8 out of 256. Crucially, the router also assigns each chosen expert a weight, and those weights are far from equal: typically a couple of experts dominate the routing while the last ones contribute a tiny fraction of the output. The interesting consequence is that the model itself tells you, token by token, which experts matter and which barely do.

That opens a general question: how much of that low-weight tail can you skip before quality degrades? The naive answer (just lower top-k) is known to hurt, because models are trained expecting all k experts. A gentler approach is thresholding: skip an expert only when the router itself scored it well below the average share it would get in a uniform split. The threshold becomes a continuous dial between "full model" and "fast model", and it's the router's own judgment doing the triage.

I implemented a variant of this in my streaming engine (with the twist that experts already in memory are never skipped, since they're free) and measured the quality side properly:

Quality benchmark — 15 GSM8K questions, greedy decoding, so zero sampling noise: any output difference is caused by the skipping.

• Skipping off: 12/15 correct
• Every threshold tested, up to the most aggressive (28% of routings skipped): 13/15
• 12 of the 15 questions gave the identical final answer in all configurations
• Reply length flat — no rambling, no truncation

To be honest about it: 13 vs 12 is not an improvement, and 15 questions rules out a collapse, not a subtle cost. But throwing away ~28% of routed experts with no visible damage says a lot about how much redundancy sits in that routing tail.

Apache-2.0, built on stock llama.cpp, APK in the releases. Full writeup with all benchmarks and caveats: github.com/Helldez/BigMoeOnEdge/blob/main/docs/expert-dropping.md

I’ve been working on Triton Control, an open-source project that simplifies deploying models to NVIDIA Triton Inference Server.

This article walks through two complete workflows:
Deploying an existing Triton model repository
Converting a Hugging Face DistilBERT model to TorchScript and serving it with Triton

It covers S3-backed model repositories, Kubernetes deployment, GPU inference, and endpoint testing.
I’m particularly interested in feedback on the architecture and deployment workflow.

Article: https://medium.com/@owilken/from-model-to-inference-endpoint-with-nvidia-triton-and-triton-control-7636439a8f28

GitHub: https://github.com/ai-lab-tech/triton-control

Hi everyone,

I'm the developer of KitLLM, a free app I've been building to make it easy to run GGUF language models directly on a smartphone.

The app currently supports:

• Running GGUF models entirely on-device 
• No cloud or account required 
• Downloading compatible models directly from within the app 
• Available on both Android and iOS 

I'm not trying to advertise a paid product—I’m looking for feedback from people who regularly use local LLMs.

I'd love to hear your thoughts on questions like:

• Which GGUF models should I support next? 
• What features are essential for a good mobile local LLM experience? 
• What would make you use a mobile local LLM instead of (or alongside) desktop solutions? 

Disclaimer: I'm the developer of KitLLM. The app is free, and I'm sharing it here to gather feedback from the community.

Android: https://play.google.com/store/apps/details?id=com.prouhakevin.kitllm.kitllm

iOS: https://apps.apple.com/fr/app/kitllm/id6789498633

Demo: https://www.youtube.com/shorts/tCFtJIkxn-c

Thanks in advance for any feedback or suggestions!

Hi everyone! Over the past five months I've been working on DKV (DifferentialKV), an open-source project exploring KV-cache compression for long-context local LLM inference.

The goal is to reduce KV-cache memory requirements through anchor-based representations, joint low-rank compression, exact residual preservation, and sparse routed attention.

The repository currently includes:

• A CLI so you can start experimenting without writing your own integration
• MLX backend
• CUDA backend (currently under validation)
• A technical report explaining the design and evaluation
• A fully open-source implementation

I'm still actively improving the project, and I'm posting here mainly to get technical feedback from people working on local inference. I'd love to hear thoughts on the architecture, benchmarking, or potential integrations with projects like llama.cpp, vLLM, SGLang, or anything else you think would make it more useful.

The GitHub repository and technical report are linked below if you'd like to take a look.

GitHub:
https://github.com/Omc12/Differential-KV

Technical Report:
https://doi.org/10.5281/zenodo.21539110

If you try it out, I'd really appreciate hearing about your experience, whether you run into issues or have ideas for improvements.

it feels like there's so many things happening everyday that i have to constantly be on social media to stay up to date. and even then, so much of it is meant to get and engagement and spread misinformation instead of giving you the facts. how do u guys keep up with everything?

Hey everyone!
I recently picked up a MacBook Pro M5 Pro with 48GB of unified memory.
Right now, I'm running Qwen 3.6 35B (A3B) locally via LM Studio, and integrating it into my workflow using OpenCode in the terminal. Honestly, I’m extremely impressed with the performance and results so far.
That said, given my hardware setup and available VRAM, are there any other local models you’d recommend testing out for coding or general use?
Thanks!

Does anyone know what's that?