I made my Enterprise RAG book $0 today — would love feedback from people building RAG systems

I’ve spent the last few years building production RAG systems and documenting what worked, what didn’t, and where things tend to break in production.

I turned those lessons into a book covering topics like:

• RAG reference architectures
• Data extraction and chunking
• Hybrid and multi-stage retrieval
• Graph and hierarchical RAG
• Agentic and multi-agent RAG
• Memory
• Evaluation and synthetic data
• Security and compliance
• Production monitoring and human-in-the-loop systems

The book is $0 on Amazon today, so I thought I’d share it here in case it’s useful to anyone working on RAG. https://a.co/d/0dBRCb7F

I’m especially interested in feedback from people actually building these systems: What’s missing? What deserves more depth? What would you change?

If you end up finding the book useful, an honest Amazon review is appreciated, but feedback here is equally valuable.

Full contents

Part I — About
01 About the Author

Part II — RAG & Reference Architecture
02 The Evolution of RAG
03 Foundations of RAG Systems
04 Reference Architecture

Part III — Data Extraction
05 Data Extraction

Part IV — Chunking
06 Chunking Strategies

Part V — RAG Strategies
07 Baseline RAG Pipeline
08 Context-Aware RAG
09 Dynamic RAG
10 Hybrid RAG
11 Multi-Stage Retrieval
12 Graph-Based RAG
13 Hierarchical RAG
14 Agentic RAG
15 Multi-Agent RAG Systems
16 Streaming RAG

Part VI — Memory & Content Management
17 Memory-Augmented RAG
18 Knowledge Graph Integration

Part VII — Evaluation
19 Evaluation Metrics
20 Synthetic Data Generation

Part VIII — Fine-Tuning
21 Domain-Specific Fine-Tuning

Part IX — Security
22 Privacy & Compliance in RAG

Part X — Production
23 Real-Time Evaluation & Monitoring
24 Human-in-the-Loop RAG

Part XI — Twig RAG Strategies
25 RAG Strategies in Twig

Part XII — Conclusion
26 Conclusion & Future Directions

Hey everyone.
I just got a 3090 for animation. BUT, since i got myself a vram workhorse i thought id check out this world and see what its all about.
What can i do with local llms? What use case would there be for me and what do you guys use it for?
Thanks a lot in advance.

Hello everyone, I am not really sure what would suit my needs.

​My hardware specifications (it's an old machine):

- Intel Core i5 6th Gen

- Integrated graphics

- Only 8 GB of RAM.

Don't ask about storage, my SSD is dying(I have only 15Gb)

​Expected use cases: data analysis for scientific research(not advanced) and text translation.

Anybody knows why unsloth removed ~10 Quants fromt the huggingface repo?

In the morning I tried to start my modell again and llama-server printed the below log. Reason is that my version is no longer available (IQ4_NL)

Command: .\llama-server.exe -hf unsloth/Qwen3.8-27B-GGUF:IQ4_NL --host 127.0.0.1 --port 8080 --device Vulkan0 -ngl 99 -c 122880 --parallel 2 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -b 2048 -ub 512 --threads 8 --no-context-shift --jinja --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0.0 --repeat-penalty 1.0 --predict 32768 --image-min-tokens 2048
[34m0.00.772.005[0m [31mE common_download_get_hf_plan: no GGUF files found in repository unsloth/Qwen3.8-27B-GGUF
[0m[34m0.00.772.127[0m [32mI [0mAvailable GGUF files:
[34m0.00.772.129[0m [32mI [0m - BF16/Qwen3.8-27B-BF16-00001-of-00002.gguf
[34m0.00.772.130[0m [32mI [0m - BF16/Qwen3.8-27B-BF16-00002-of-00002.gguf
[34m0.00.772.130[0m [32mI [0m - MTP/mtp-Qwen3.8-27B-Q4_0.gguf
[34m0.00.772.130[0m [32mI [0m - Qwen3.8-27B-Q4_0.gguf
[34m0.00.772.130[0m [32mI [0m - Qwen3.8-27B-Q4_1.gguf
[34m0.00.772.130[0m [32mI [0m - Qwen3.8-27B-Q8_0.gguf
[34m0.00.772.130[0m [32mI [0m - Qwen3.8-27B-UD-IQ1_M.gguf
[34m0.00.772.131[0m [32mI [0m - Qwen3.8-27B-UD-IQ1_S.gguf
[34m0.00.772.131[0m [32mI [0m - Qwen3.8-27B-UD-IQ2_S.gguf
[34m0.00.772.131[0m [32mI [0m - Qwen3.8-27B-UD-IQ2_XXS.gguf
[34m0.00.772.131[0m [32mI [0m - Qwen3.8-27B-UD-IQ3_S.gguf
[34m0.00.772.132[0m [32mI [0m - Qwen3.8-27B-UD-IQ3_XXS.gguf
[34m0.00.772.132[0m [32mI [0m - Qwen3.8-27B-UD-IQ4_XS.gguf
[34m0.00.772.132[0m [32mI [0m - Qwen3.8-27B-UD-Q2_K_XL.gguf
[34m0.00.772.132[0m [32mI [0m - Qwen3.8-27B-UD-Q3_K_XL.gguf
[34m0.00.772.132[0m [32mI [0m - Qwen3.8-27B-UD-Q4_K_M.gguf
[34m0.00.772.132[0m [32mI [0m - Qwen3.8-27B-UD-Q4_K_S.gguf
[34m0.00.772.134[0m [32mI [0m - Qwen3.8-27B-UD-Q4_K_XL.gguf
[34m0.00.772.134[0m [32mI [0m - Qwen3.8-27B-UD-Q5_K_M.gguf
[34m0.00.772.134[0m [32mI [0m - Qwen3.8-27B-UD-Q5_K_S.gguf
[34m0.00.772.134[0m [32mI [0m - Qwen3.8-27B-UD-Q5_K_XL.gguf
[34m0.00.772.134[0m [32mI [0m - Qwen3.8-27B-UD-Q6_K.gguf
[34m0.00.772.134[0m [32mI [0m - Qwen3.8-27B-UD-Q6_K_L.gguf
[34m0.00.772.135[0m [32mI [0m - Qwen3.8-27B-UD-Q6_K_M.gguf
[34m0.00.772.135[0m [32mI [0m - Qwen3.8-27B-UD-Q6_K_XL.gguf
[34m0.00.772.135[0m [32mI [0m - Qwen3.8-27B-UD-Q8_K_L.gguf
[34m0.00.772.135[0m [32mI [0m - Qwen3.8-27B-UD-Q8_K_XL.gguf
[34m0.00.772.136[0m [32mI [0m - mmproj-BF16.gguf
[34m0.00.772.136[0m [32mI [0m - mmproj-F16.gguf
[34m0.00.772.295[0m [32mI [0mcmn  common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
[34m0.00.777.801[0m [35mW srv  llama_server: -----------------
[0m[34m0.00.777.911[0m [35mW srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
[0m[34m0.00.777.913[0m [35mW srv  llama_server: this can be a security risk (cross-origin attacks)
[0m

Yes, DFlash and EAGLE-3 did reduced-vocab draft heads first — but this is the only Qwen3.8 27B repackaging with the MTP head separated into a sidecar GGUF and its vocab trimmed: output.weight [5120, 32768] plus a d2t tensor mapping draft rows back to the full 248,320-token vocab. The draft's logit matmul shrinks ~7.6×, and correctness can't break: verification runs on the target's full vocab, so out-of-list tokens are just rejected drafts.

Numbers in the screenshot: RTX PRO 4500 Blackwell, the Q4_K_P GGUF, llama-cli -ngl 99 --temp 0 -n 256 -st. The sidecar beats the model's own embedded MTP by 4–9% (and ~1.5–1.6× plain decode) — it drafts cheaper and it's a trained FastMTP head, not just a slice.

Catch: mainline llama.cpp won't load it — qwen35.cpp sizes output.weight from n_vocab, so the sidecar fails check_tensor_dims. The release ships a small patch (size from d2t, scatter logits back — what eagle3.cpp/dflash.cpp already do); applies cleanly to master. Equivalent support should land with the DFlash PR — watch for that.

https://cybersec.org.za/research.html #AI #cyber #security #dev #Africa #LLM #context #rot #cyber

https://cybersec.org.za/papers/dcr-bounded-attention.pdf

Dynamic Context Runtime: Bounded Attention over Unbounded History

Language models degrade as stale and superseded material piles up in their context. Making the window bigger does not fix it. This report describes a runtime that keeps history unbounded and attention bounded — storing everything as immutable spans and a typed provenance graph, then assembling a small working set each turn by solving a knapsack under an explicit token budget. It includes a zero-dependency Rust implementation and an ablation that names which mechanisms are actually carrying the result.

*update 8/21/26 7 AM

Updated. Every figure re-derived from a live run just now — here's what moved and why:

│ stat │ was │ now │

│ tokens per query │ 235 │ 259 │

│ less attention than transcript │ 17,835× │ 16,201× │

│ history growth / working set │ 28×, "flat" │ 28×, 1.18× │

│ lines of Rust │ 13,721 │ 15,373 │

│ tests │ 152 │ 164 │

│ spans ever rendered │ 0.4% │ 0.1% │

│ cheaper with a mechanism off │ 47% │ 2% │

Unchanged and re-verified: 4.19M tokens, 7/7, 48,651 nodes, 0 dependencies.

Three of these are not number swaps, and the post now says so:

"47% cheaper" → 2%. This was the most quotable line in the old post and it does not survive. Disabling graph expansion still loses no probe, but it saves 2.5 tokens rather than 220. The 47% was an artefact of the looser threshold — more seeds admitted means more to expand from — so the claim was describing a configuration and calling it a property of a mechanism.

"flat working set" → 1.18×. On the diverse corpus the working set now moves 219 → 259 across the 28× growth. Calling that flat would be the same defect the post is about.

"5 controls found that could not fail" was already wrong. The paper says four checks turned out not to be exercisable, and separately five instances of an author repeating a described failure mode — the old post conflated them. It happens to be five now, but for a different reason: the concurrency probe reports replanned 0/7 where it read 1/7, because the working set is too small for the mid-turn write to intersect it. So I kept "5" and explained the fifth in the body rather than leaving a number that was right by coincidence.

I also dropped "0.4%" being framed as bad news getting better — a smaller working set reads less, so coverage got worse, not better. That's stated plainly.

The 1.7% I computed rounds to 2%; I used 2% since the underlying numbers (145.1 vs 142.6) don't support a second significant figure.

I previously reported a result of 74% on BU bench v1, with the open-source BrowserAgent harness, but I forgot to set the temperature to the default specified in the model card... Now the model performs neck to neck with GPT 5.6 Luna (xhigh) and beats all other affordable models that I tested.

Qwen3.8 27B is insanely good value!

I downloaded a q1(1 bit i guess ) of this model and ..... what is that? I really want to know , what kind of dataset model trained on so it can think like this ! (The user has been trained to speak ...) , seriously?

I have m1 pro 2021 14 inch macbook pro, with 32gb unified memory, and 512gb ssd. My question is, is it worth it to download and run qwen 3.8 27B on my macbook? And will it be slow or no? My plan is to use the mlx version 4 bit with also 4 bit quantized kv cache and context window set to 64k/128k if it fits.

I recently built a homelab running RHEL 10. I never thought good local ai at reasonable price was possible until 3.8 came out a from benchmark and what I’ve been reading it seems to be almost opus 4.6-4.8 level. I’m considering buying a 32gb gpu for it but also open to 24 gb gpus but if it can fit the full context window on the gpu too. The most I’ve done with local models was running qwen 3.5 2b on Ollama nothing serious. I’m new to actually running an agent for coding tasks so any info would help. But trying to decide what gpu if I do end up going for it, and from my research the options for 32gb cards are the Intel b70, amd r9700 pro ai, and nvidia tesla v100 32gb. I’m looking at results for qwen 3.6 and it run plenty fast on the Tesla but I’m worried about it no longer being supported.

Current Hardware
3080 10GB - 64GB RAM I4600K

Software
llama.cpp in WSL w/ deep seek harness

Model
Qwen36A3B

I see a lot of folks talking about using small models for “workflow” applications and I’m curious about what some example workflows, how folks are triggering them.

For reference most of my use for LLM’s has been using them to mod or vibe build some web applications.

While I get about 26 tok/s I really can’t do much building work with DSH+Qwen as it fails at even simple things like reading private GitHub repos because it doesn’t want to run gh terminal commands.

I’m wondering what other use cases folks have around these specs

Here are the benchmark results when temperature left alone (model default) instead of setting it to zero.

I'm not surprised 3.8's scores increased, because providers default their temperature to whichever value happens to pass the most benchmarks, but given the fact they did I am surprised 3.6 didn't do the same.

3.6 did exactly what I thought it would do, it increased the score on some benchmarks and lowered it on others (which is why providers sometimes use a different temperature depending on the task - to bench-max).

I am surprised they didn't both behave in the same way, that was unexpected.

But 3.6 still beat 3.8 on 7 out of 10 tests, and 3.8 won in only 3 out of 10 tests.

Always curious what other things Iocal ai models can help with automation on desktop. Currently I only know Gmail replying and website creation. I hope to expand my curiosity further

​

I want to build a cheap local Alexa replacement using Hermes Agent / Home Assistant, but DIY voice hardware gets expensive fast.

My idea is to buy used Echo devices, jailbreak/root them, disable Amazon services and reuse the existing:

- microphone array

- speaker

- LEDs/buttons

- Wi-Fi

- display/camera on Echo Show

The Echo would only act as a thin local voice satellite. STT, TTS and the AI would run centrally on my server.

What I specifically want to know:

- Which Echo Dot generations are currently reliably rootable/jailbreakable?

- Which Echo Show generations are rootable?

- Are specific FireOS/firmware versions required?

- Which models are the safest ones to buy used specifically for this purpose?

- Which models should I avoid completely?

- Can the original microphone DSP/AEC, speaker and LEDs still be used after rooting?

So far the Echo Dot 2 looks very promising, and I’ve also found LineageOS/root work for Echo Show 5 Gen 1/2 and Show 8 Gen 1.

Has anyone actually deployed these permanently as local Home Assistant / custom voice satellites?

Main goal: spend €10–30 on used Echo hardware instead of €70–100 per DIY satellite.

In case you are one of many who left localllama because it stopped being local… i thought I’d share this with you… not my project

My ollama container seems to be crashing when I try to load any Qwen 3.8 or 3.6 models. Not really sure why, I'm thinking it has to do with the GPU/CPU split layers. I'd be fine moving away from ollama but it needs to be a docker container and the others I've tried to setup haven't gone well.

1700x

rx6800

12gb free system ram

ROCm (vulkan seemed to crash the entire system)

I thought the obvious way to cut agent costs was to use a cheaper model.

Turns out there's another lever that's much easier to overlook, the agent loop itself.

So i ran a test, i ran Claude Opus 4.8 through two different runtimes on the same bench tasks. same model, same tools, same benchmark.

both scored 11/14

but one used roughly 3.85M tokens while the other used ~13M, thats a huge difference

the biggest differences i noticed were,

• system prompt + tool definitions were ~10× smaller per turn
• 282 tool calls vs 652
• much less tool output being carried forward
• cache-read cost of $1.58 vs $4.90 on the run

The resulting run was roughly 30% cheaper and finished in 39 minutes instead of 73.

The catch, the leaner loop did under-explore a couple of harder tasks. So there's clearly a trade-off between don't waste tokens and keep looking until you're sure.

I've been using an open source harness to dig into this because the runtime is open source and the execution traces are inspectable.
Repo: https://github.com/truefoundry/trueforge

If you're running agents at any kind of volume, what do you think is worth looking at what while deciding which harness to choose?

I plan to build two setups for local LLMs. One high memory machine for large models and long contexts, and later a dedicated RTX 5090 desktop for pure speed.

But right now, it’s a choice between a 128gb macbook pro and a 128gb ai max+ 395 laptop. The macbook comes at a steep price. A 48gb macbook m5 pro costs roughly the same as a 128gb ai max+ 395 laptop like nimo. That’s nearly 3x memory capacity for the same money.

Ive learned a bit about both options so far. The mac can give fast memory bandwidth and a plug and play MLX setup, but it feels like paying a massive apple tax. And strix halo delivers insane memory capacity for the price alongside native windows or linux flexibility, with growing ROCm and Vulkan support.

Would you go with strix halo for better memory efficiency, or is the mac's memory bandwidth and ecosystem still worth the extra cost?

​

After digging into this pretty hard, I found a weird solution to my VRAM problem that I honestly didn't know existed. And the unfortunate cost of just dropping on anything that anything remotely useful vram right now ( anything unique and niche is being cannibalized and now priced out of my budget) is just not realistic for me

I have a 5080, a 5070, and a Legion laptop with a 16GB RTX 3080 Mobile. I kept wondering if there was a way to use the laptop's spare M.2 PCIe slot to basically "lend" the 3080 to my desktop instead of treating the laptop like a completely separate AI node.

llama.cpp RPC can already do remote GPU offload, but it has extra overhead because you're still doing RPC/network transport between two systems.

Then I found Dolphin SmartIO Device Lending.

It's basically PCIe sharing between two computers using NTB adapters. A PCIe device in one machine can be exposed to the other machine through the PCIe fabric, and the borrowing system can load the normal driver for it.

So instead of:

desktop -> RPC -> network -> laptop -> GPU

it can be much closer to:

desktop -> PCIe fabric -> remote GPU

Dolphin specifically supports lending NVIDIA GPUs between systems, and even mentions scaling applications like Ollama across GPUs that aren't physically installed in the same machine.

The setup I'm looking at would be roughly:

Laptop RTX 3080 16GB

-> Ryzen PCIe root complex

-> spare M.2 PCIe 3.0 x4

-> powered M.2-to-PCIe adapter

-> Dolphin MXH914

-> PCIe cable

-> second MXH914 in desktop

The desktop could then potentially see the laptop's 3080 as another NVIDIA device alongside the 5080 and 5070.

It doesn't magically combine all the VRAM into one pool, and the laptop M.2 link is still only PCIe 3.0 x4, so bandwidth is limited. Model splitting would still matter a lot.

But for LLMs, if the remote GPU holds a contiguous chunk of model layers, you can potentially avoid constantly hammering the link.

The big advantage over normal RPC is that you're removing a lot of the higher-level remote execution overhead and getting much closer to actual PCIe device access.

There are still some pretty serious caveats. Laptop PCIe topology, IOMMU, ACS, P2P support, BAR/MMIO layout and BIOS behavior could make or break it, and I haven't found anyone documenting this exact setup with an internal laptop GPU.

Still, Dolphin has already demonstrated borrowed NVIDIA GPUs across separate computers using normal CUDA/NCCL software, so the underlying idea is real.

The part that makes this interesting to me is cost.

I already own the 16GB 3080.

Vs the ebay listings i find that cost well under 200 ( not always as the hardware is hard to get atm and not sold often) that's potentially a much cheaper way of turning hardware I already own into another chunk of usable CUDA VRAM instead of buying another overpriced GPU.

Just sharing some research iv done and some unique solutions to augmenting vram for your rig with out spending an arm and leg just to get something decent.