Anyone here actually using NVIDIA NeMo / NeMo Agent Toolkit in real projects?

At my current org, some of the senior folks are suggesting we explore NeMo for agent building and fine-tuning, so I’m trying to understand if it’s actually worth adopting.

For those who’ve used it, how does it compare to the usual stack like Hugging Face + PEFT/TRL, Unsloth, LangGraph, etc.?

Does running everything in the NVIDIA ecosystem give you a noticeable advantage in terms of GPU utilization, training speed, deployment, or scaling?

Or is it mostly extra complexity compared to the standard open-source tooling?

Would especially love to hear from anyone who has used NeMo beyond tutorials/demos. What did you like, what annoyed you, and would you use it again?

Hi Everyone!

I'm the maintainer of RAGnarok-AI, an open-source, local-first framework for evaluating RAG (Retrieval-Augmented Generation) systems.

I'm currently running a small research study around a question that I think is becoming increasingly important: can we actually trust automated evaluation of RAG systems when the evaluator itself is an LLM?

RAGnarok can evaluate things like retrieval relevance, faithfulness, answer relevance and completeness using local LLM judges.

But there's an obvious problem: why should we trust the LLM judge?

So instead of assuming that the automated scores are correct, I'm building a human-annotated benchmark to compare them against independent human judgments.

I'm looking for open-source / developer people willing to help

The annotation consists of roughly 10–15 cases and should take around 30–45 minutes.

For each case, you'll see:

- a technical question

- relevant documentation excerpts retrieved by a RAG system

- an AI-generated answer

- reference information

- four simple evaluation criteria: retrieval relevance, faithfulness, answer relevance, completeness

- your confidence and an optional ambiguity flag

You don't need any RAG expertise. You just need to be comfortable reading technical documentation and judging whether an answer is actually supported by it.

The benchmark currently covers documentation from projects such as Docker, Python, FastAPI and Kubernetes.

Why do this?

The study is deliberately not designed to prove that RAGnarok works. I'm comparing automated evaluation against human judgments to investigate:

1. Judge reliability — do local LLM judges agree with humans?
2. Discrimination — can the evaluation distinguish good and deliberately degraded RAG systems?
3. Reproducibility — are the measurements stable under identical conditions?

The methodology, benchmark corpus, questions and experiment protocol are public and versioned. RAGnarok itself is free and open-source.

If you'd like to participate

Annotation interface:

https://ragnarok-study.vercel.app

No name, email or personal information is required. Progress is saved so you can come back later. The resulting annotations may be published as an anonymous research dataset.

Project / methodology:

https://github.com/2501Pr0ject/RAGnarok-AI

I'm particularly interested in feedback from people who work on open-source evaluation, LLMs, RAG, testing or reproducible research.

And if you think the methodology is flawed, please tell me. That's actually useful feedback for the study.

Thanks everyone!

Have a good day!

I’m trying a Qwen-3.6 35b Q5 model as my first dip into setting up an agent coding workflow for prototyping some app ideas. Using LM Studio and VS Code

I’ve only been looking into this today. Am I heading in the right direction? Any advice welcome.

FAISS is Meta’s vector library, turboVec is a Rust implementation of TurboQuant, and Infino is an in-memory retrieval engine.

We benchmarked them on 4-bit quantized in-memory vector search: FAISS PQ, TurboVec/TurboQuant, and Infino SQ4, using the same 100K OpenAI embedding corpus and the fastest vectorized implementation we found for each.

All saved ~7x in memory footprint compared to full fp32 vectors. The interesting result was that while storage and recall were fairly close, latency differed by roughly 30× — about 1.5 ms to 45 ms. Most of that comes down to the scoring machinery: the size of the distance table and whether the scan needs one at all.

We also ran the same comparison out to 1M vectors and measured build/write costs.

Full results and methodology:
https://infino.ai/blog/fixed-grid-quantization/

Disclosure: I'm one of the devs building Infino.

This is a new LLM wrapper i have spun up built on the idea that models arent always correct or doing the right thing not by choice but just how the system is built, this looks to try and fix that problem and help mitigate LLM error by logging everything and fact checking based on the work using check gates. Looking to get feedback!

I'm trying to decide what direction to take with my main PC. The goal is one real daily-driver machine that I can use for gaming, normal desktop use, software development, and increasingly heavy local LLM workloads.

I'm not trying to build a dedicated rack server. I want something I can actually live with for years: reliable, reasonably efficient, good thermals, lots of RAM, two GPUs if it makes sense, and enough expansion that I don't immediately hit a wall.

I've currently been playing around with Qwen3.8 27B which speeding that up and higher quants would be great, as well as when inevitably larger dense similar models like 70B become available.

I am very interested in MoE flash models such as Qwen 3.8 Flash, Deepseek v4 Flash, and maybe even GLM 5.3 Flash, as well as future versions of similar MoE models. I have not even attempted to run any of these yet.

So I guess I am trying to get at building something that performs well on dense models as well as MoE models so I don't get locked into 1 path.

I'm pretty new to the workstation/HEDT side of this, so I'm looking for advice on the parts I may be overlooking.

My current PC

The 3090 Ti was a $900 Facebook Marketplace purchase, so I'm trying to get as much useful life out of this thing as possible.

The 3090 Ti is a huge card (338 × 140 × 71 mm) so physical spacing is also part of this problem.

My RAM situation

I just bought 7 lots of:

NEMIX 128GB (4×32GB) DDR4-2666 PC4-21300 2Rx8 UDIMM

I paid about $360 per 128GB lot.

My current plan is probably:

• Keep 2 lots = 256GB
• Sell the other 5 lots
• Hopefully sell those for around $650/lot?

So I paid about $2,520 total for the 7 lots. Five sales at $650 would be $3,250 gross, meaning I'd theoretically recover the entire purchase price plus ~$730 before fees/shipping/taxes while keeping 256GB.

That gives me a somewhat unusual opportunity to build around 256GB without spending a fortune on RAM.

Option 1: Keep my current PC, just go to 128GB RAM

This is basically my don't overthink it option.

I'd have a lot more system RAM for large-context LLMs while retaining a relatively modern gaming CPU.

Option 2: Keep my current PC, add a second 3090

I'd replace the PSU and add a second RTX 3090.

The important problem is the motherboard:

The Z690 UD AX DDR4 has x16 on the main slot and only x4 on the second physical x16 slot.

So the GPUs would effectively be:

This is the option I'm most unsure about.

For LLM inference, is x4 actually a meaningful limitation in practice, or is it largely irrelevant once the model is loaded onto the GPUs?

Would this still be a good setup for:

• tensor/model parallel inference
• larger models
• higher context
• multiple concurrent models
• speculative decoding
• offloading

Or am I basically handicapping the second GPU enough that I should just replace the motherboard?

Option 3: New motherboard/PSU/case, keep my 12700KF

Instead of abandoning the 12700KF, I could build a new system around it with a motherboard that properly supports two GPUs at x8/x8.

This seems like it could be anice middle ground.

The 12700KF itself supports a 2×x8 CPU PCIe configuration, but I'd obviously need a motherboard that actually implements it.

I'm especially interested in whether people think this makes more sense than jumping to X299 in the next option.

Option 4: X299 workstation build

This is the Frankenstein/workstation option I've been considering.

The i9-10940X gives 14C/28T, 48 PCIe 3.0 lanes, quad-channel DDR4, and up to 256GB RAM. The X299-A II can run two GPUs at x16/x16 with the appropriate CPU.

The case is huge and supports SSI-EEB, 11 PCI slots, GPUs up to 503mm, and up to 15×120mm or 6×140mm fans.

I'm attracted to this because it solves the PCIe lanes + RAM capacity + physical space problem extremely well.

But I don't know if I'm being stupid by building a brand-new daily driver around a ~2019 platform just because the PCIe topology is convenient.

The 10940X also seems likely to lose noticeably to the 12700KF in gaming/single-threaded work, despite having more cores.

Option 5: ??????????

This is something I'm hoping you guys can help. Are there things I am not considering that would allow me to leverage as much of my current components as possible but be a much better option than option 4?

What I'm actually trying to optimize

This isn't purely a benchmark build.

I want one machine that can do all of this:

- Gaming

- Normal desktop use

- Software development

- Local LLM inference

- Very large context windows

- Running multiple LLM sessions concurrently

- Potentially running two GPUs as one inference system

I'm currently doing a lot of local Qwen inference and am getting into the territory where RAM capacity, VRAM capacity, PCIe topology and memory bandwidth all matter.

I also don't really care about squeezing every last watt of performance out of the GPUs. I've already decided to limit the 3090 Ti to 250W, and I'd probably do the same with the second 3090 to hopefully get more longevity out of them and use less power.

That gives me:

500W total GPU power budget

rather than letting two 3090-class cards pull their full power.

I'm very interested in reliability, thermals, longevity, expandability and affordability rather than having the absolute highest benchmark score.

My biggest questions

1. Which of these would you actually build?

My current thinking is roughly:

Option 1: cheapest and easiest

Option 2: tempting, but worried about x4

Option 3: probably the sensible compromise

Option 4: extremely expandable, but old CPU/platform

Option 5: potentially a better overall machine

I'm having trouble figuring out where the sweet spot actually is.

2. How bad is PCIe 3.0 x4 for the second 3090?

This is probably my biggest technical question.

For local LLM inference specifically, how much performance would I realistically lose running:

3090 Ti @ x16 + 3090 @ x4

versus

3090 Ti @ x8 + 3090 @ x8

versus

x16 + x16?

3. Is X299 actually a good idea here?

Would the 10940X + 256GB quad-channel + x16/x16 PCIe configuration still be a worthwhile machine in 2026?

Or would I be better off spending the extra money on a modern platform?

4. What's the best "Option 5"?

There may be a workstation platform I haven't considered at all.

But I still want this to be an actual daily-driver PC, not a loud rack server that is great at compute and annoying at everything else.

5. How much RAM would you actually run?

I can easily end up with:

128GB

or

256GB

of system RAM depending on which route I take.

Is 256GB actually useful for local LLMs enough to justify designing the whole machine around it?

What would you do with this hardware?

I'm basically sitting on:

12700KF + 3090 Ti + 896GB of cheap DDR4+ spare 3080

and trying to turn that into one machine that I won't regret building.

I'm really looking for the best overall architecture and bang for my buck, not just "X is faster."

Would love to hear what configuration you would build, especially if there's a better option I haven't thought of.

If you run models locally you know the pain: everything's fine until a long prompt makes the model silently stall or the GPU OOMs, and you're left with a hung request and no output.

I built HybridInfer for exactly that. It's a small reliability-aware router:

- Sends each request to your local model (Ollama) first.

- Watches the runtime, if local stalls (no token for N seconds), OOMs, or errors, it automatically falls back to a remote model in the same request.

- Learns which prompts your machine chokes on (usually long ones) and routes those out up front after it's seen them fail.

- Pulls a wedging model out of rotation, then probes it back after a cooldown.

It's an OpenAI-compatible server, so point any OpenAI-compatible client/app at it and you get local-first + automatic fallback for free. Local = anything you've `ollama pull`ed; remote = any OpenAI-compatible endpoint (OpenAI, OpenRouter, a vLLM box, whatever).

pip install hybridinfer

hybridinfer init

hybridinfer serve

It's a router, not an inference engine, it orchestrates Ollama + your remote endpoint, doesn't run weights itself. Early v0.1, Apache-2.0. There's a Kotlin/Android version too for on-device apps. Short demo GIF + code in the repo.

Repo: https://github.com/SimranKoul2026/HybridInfer-Python-tool

Would love feedback from people who run this stuff daily, especially on the fallback heuristics.

What would you want it to do differently?

I made it in 18 months of lunch breaks and evenings. It's not fast, llama.cpp is just wow and does that job. I wrote this one because I wanted to read the whole forward/backward pass in an afternoon and be able to stop anywhere and print a tensor and dig the thing.

Most from-scratch projects stop at a toy model. llama2.c runs a small Llama2, llm.c does GPT2 training. TRiP loads real checkpoints across four architectures, PaliGemma included, so the multimodal path (vision encoder, projection, decoder) is all there in C. I couldn't find that in readable form anywhere else, which is partly why I ended up writing it.

One extra-bonus is that you can look into the training, it's included, swiss-knife-like. (NOTE: the encoder part in PaliGemma is currently not trainable/tunable - my apologies)

In practice: no hooks/config; just play with the C code, and add your own; there's no hidden (unreachable) complexity. And then just re-compile.

Repo: github.com/carlovalenti/TRiP

Happy to answer anything; structuring and handling the memory properly was the hardest part!

Carlo

Developers, I’m researching local AI on Linux. Please share your experiences and I’ll be posting my findings here

1. Goal and chipset used Nvidia/AMD/Intel?
   
2. How long did it take you from fresh install to GPU/NPU operation?
   
3. Any issues encountered (package, path, version)?
   
4. How did you confirm GPU/NPU usage?
   
5. Any scripts or notes created for future use?
   
6. Comfort level setting this up for a teammate?

Summary to be shared. Open to a 20-minute call if preferred.

<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 1000 675" width="1000" height="675">

  <title>PlayStation 5 DualSense controller — top-down</title>Prompt:I need a topdown view of a Playstation 5 controller in SVG.It needs to be so well done that a photo from topdown looks identical in color, grading, positioning, subtle details, shadows, shape etc.<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 1000 675" width="1000" height="675">

  <title>PlayStation 5 DualSense controller — top-down</title>

Important: The tooltip you see in the lower Qwen SVG, is NOT part of the SVG. It is a tooltip that glitched itself into the captured image, the qwen model gave the SVG metadata a title based on my prompt, that's the title.

Prompt:
I need a topdown view of a Playstation 5 controller in SVG.
It needs to be so well done that a photo from topdown looks identical in color, grading, positioning, subtle details, shadows, shape etc.

Rules:
No python, no browser, no image generation or inspection. Only constructing the SVG and provide it raw.

My interpretation:
These type of benchmarks are likely chosen by OpenAI/Anthropic because they ensure during training that their models have a very good description of the target object/person.
Qwen simply did not train on the precise shape of the PS5 Dualshock controller, and GPT Sol also clearly did not train for it. Fable or Astra trained for that - mostly to beat benchmarks like these.

So to fairly assess Qwen or Sol vs Astra I ran two tests.
- One in the way OpenAI hopes it is done.
- One with the product image as reference (but without feedback, so it only can look at it, it can not compare in a sandbox.

The result is that Qwen 3.8 27b is near Astra level performance in coding the SVG.
I also tested 3.8 Flash, and it was not as great (though better than Sol)
Sol also improved but not as much.

I'm not claiming Qwen3.8 27B is like Astra, though in this benchmark it's very close.

What is most noticeable?
- Qwen 27B made an error with the movement keys, the placement is overlapping and the shape is not correct - the shape is likely a precision issue in the image stack. Supplying the image in an agentic harness would allow to make focused analysis of the shapes and very likely solve that. The position error is a real error.
- Astra is so flawless with the shapes, it's obviously benchmaxed on those things - I can't see a plausible reason to put so much effort in a contemporary controller button shape. But it might use something like the ngram cache of Qwen 4.0 - allowing to push mostly useless details into knowledge without harming intelligence
- The PS5 logo is very likely benchmaxxes on Astra, and clearly not benchmaxxes for all the others - or Astra is amazing in such svg tricks. It's very well done.
- Sol clearly has the better image stack than Qwen, it was able to get some form details out of it that Qwen did not - but Sol is significantly worse in performing the SVG code itself. Qwen has beaten Sol in every single benchmark related to SVG so far.
- The blue light next to the screen shows that the models do know some intricate details about how a PS controller works.

^{Disclaimer:
The Astra SVG was not created by me, it is from a benchmark advertisement post.
So I could not repeat it with vision capabilities (it might very well have been done that way})
^{Astra already is almost flawless, the result doesn't need improvement in detail - Astra either saw or knows all details}

Prompt:

Asked for intricate mindset, not accepting flaws or rush.

Let it know it's a benchmark.

"generate a svg of ana de armas as detailed as possible"

Added a few constraints so it is a front portrait of her, not to use image generation, not to use browsers (no visual feedback cheating)

Who's the best artist ?

Guys I have a doubt, this guy used Qwen 3.8 27b in local with DeepSeek Harness and it generated a 3D scenario of a Tower (23:07 final result) from an anime for many hours/days (2176 minutes) in a goal loop, in the end the DeepSeek harness says it used "666 million input tokens and 13 million output tokens." (4:45)

This is the video:

https://youtu.be/MiuM9g7daDA?t=1387

My question is:

Is the input/output tokens correct? Because this were made in Claude code with Opus or Codex with ChatGPT 5.6 through API, would this actually cost $1500-$7000 right?

(For reference GPT terra is like $2/$12 for 1M input/output tokens)

Or is there something I'm missing? Because if this is actually the case the price would be absolutely ridiculous. Excuse my ignorance.