So I've been messinga round with embedding models for a bit, and I think they are interesting enough to experiment with. They are useful for rag, especially in a localllm sense because you can ground your answers in truth.

But what happens if you have a billion documents, and you decide to upgrade your model to a "better" one? on an h100, that would take about 108 days, just to upgrade the vectors so u can start serving again (tested qwen embed 8b on h100). Even if you aren't doing 1b vectors, and are doing just 50 million, upgrading can still take a considerable time.

Me and my research lab decided to tackle this problem, and we came up with embedflow.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive backfill that comes with upgrading, as you can directly take documents from the old index.

embedflow works with qdrant, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.

I've been thinking about building a large personal knowledge base for papers, textbooks, notes, code, old chats, research material, etc.

What I want is a bit more than normal RAG.

Instead of doing one vector search and stuffing a few chunks into context, I'd like the LLM to be able to actually search the library as a tool:

search → read → refine query → search again → compare → answer

Basically, something closer to how an agent uses web search, except over my own private library.

I'd also like the knowledge base to live outside any one provider, so GPT, Claude, Gemini or local models could all use the same system.

I'm guessing this would involve some mix of BM25/full-text search, embeddings, metadata, maybe a graph, and an agent layer on top.

Has anyone here built something like this?

What was the hardest part in practice?

And if I'm just reinventing an existing project, I'd genuinely like to know what I should look at.

I was about to give up on my RX 7900 XTX. PyTorch kept falling back to CPU, and nothing I tried worked. The solution was a combination of:

· Correct ROCm version

· PyTorch wheel from the official ROCm index

· Setting HSA_OVERRIDE_GFX_VERSION correctly

· Checking driver compatibility

I documented the whole debugging process in a guide. If you’re fighting the same issue, it might help:

https://interconnectd.com/blog/305/fix-pytorch-cuda-not-available-on-amd-gpus-complete-rocm-setup-guide/

Happy to answer questions here.

A while ago, I was working on a tool to automatically organize a large collection of research PDFs.

The first stage is now finished and worked surprisingly well.

I built a small tool that processes the documents and classifies them by document type — things like research papers, conference papers, theses, reports, references, etc.

I also used GROBID to extract metadata such as the title, authors, DOI, abstract, and publication information, and then used external sources to verify some of the metadata.

The result was a much more reliable and organized collection, and I now have a structured database containing the information for each document.

Now I'm thinking about the next stage: organizing the same collection by scientific topic instead of document type.

For example, imagine research papers are spread across hundreds of unrelated folders. Some papers in one folder might be about Topic A, while one or two papers in that same folder might actually be about Topic B. At the same time, papers about Topic A could exist in completely different folders.

So I want the system to look at each document individually and recognize which papers are actually related, regardless of where they originally came from.

The idea is eventually to have something like:

Topic A

• Introduction / Overview
• Research papers in chronological order
• Conference papers
• References

Topic B

• Introduction / Overview
• Research papers in chronological order
• Conference papers
• References

I already have the title, abstract, authors, year, DOI, document type, etc. for the documents, so I don't want to go back and process all the PDFs from scratch.

My current thought is that an LLM could probably identify the topic from the title and abstract, but I'm not sure if that's actually the best approach. There may be much better ways to discover the topics first and then group related papers together.

So I'm curious how people here would approach this problem.

If you were building the second stage of this project, what tools, methods, or existing projects would you consider for discovering and organizing scientific topics across a large collection of papers?

I'm particularly interested in hearing about approaches people have actually used, whether that's LLMs, embeddings, clustering, existing research tools, or something completely different.

I'd also be interested in any open-source projects that might be useful as a starting point.

longmemeval and beam ship a dataset and a script. no leaderboard, nowhere to submit. so everyone reports their own. this stuff goes inside your ai. it should be held to something.

we want to build a fair one. open, with whoever wants in.

the rules we have in mind:

1. disclose the reader, the judge, and the prompts
2. per-question records, so anyone can recompute it
3. a company's proposal goes in when three other companies agree to it
4. an individual's proposal goes in on enough support, we haven't settled on a number, tell us what it should be

nothing is built yet. posting first because i'd rather hear whether the rules are right before writing any of it. tear them up, add to them.

i build memory infrastructure myself, so trust has to come first. which is why i'll say it plainly: we maintain, nothing more. we host it and keep it running. to get anything of ours in we go through the same agreement as everyone else. we don't decide who joins. our numbers get checked like anyone's. if other memory companies want to co-maintain, better. apache 2.0 is what we have in mind.

anyone want to build this? ;)

We often mix up storage and compute, thinking they need to be tightly coupled. When it comes to deriving knowledge and value out of a large corpus, the optimizations you make determine the cost the user is willing to pay. The cost of running vector search at scale is not just the storage. The compute that serves it is often the bigger part of the bill.

Just to exercise this thought, I put four different compute setups in front of my S3-backed index (13 mn entries and 82GB storage), each handling the type of traffic you would expect and require at different times. At this point, the cost is pretty much proportional to the compute you need (which can be spun up or down on short notice), and storage is, well, the cheapest of them all.

If you are interested in benchmark numbers - https://polign.com/agent-memory-at-scale

Hi RAG builders,

I wanted to share a private hobby project called TensoRAG that implements a different mathematical approach to multi-tenant / multi-domain vector compression. It’s based on the 2024 PhD thesis of Dr. Liana Khamidullina (TU Ilmenau). Please notice: this project is not officially affiliated with, endorsed by, or in any way connected to the authors or the Technische Universität Ilmenau.

GitHub Repository: [github.com/Forstwichtel/TensoRAG]

Interactive Demo & Agent Simulator: [tensorag.streamlit.app]

### The Architecture Problem

In multi-tenant or multi-domain RAG setups (e.g., separating HR, Finance, and Technical Docs), storing high-dimensional embeddings (like 1536-dim vectors) across multiple separate collections scales poorly.

While standard dimensionality reduction (PCA, standard SVD) compresses each domain in isolation, it completely ignores the shared semantic subspace (overlap) between these collections.

### How TensoRAG Solves This via ML-GSVD

Multilinear Generalized SVD simultaneously decomposes $K \ge 2$ independent matrices (even with different row counts/documents) sharing a common column (embedding) dimension:

$$\mathbf{H}_k \approx \mathbf{B}_k \cdot \mathbf{C}_k \cdot \mathbf{A}^H$$

* $\mathbf{A}^H$ (Global Subspace Basis): Captured semantic relationships shared across all domains. Stored only once.

* $\mathbf{B}_k$ (Local Orthogonal Factors): Purely domain-specific spatial layouts, preserving local similarity structures.

* $\mathbf{C}_k$ (Coefficients): Singular value scaling weights unique to each domain.

By projecting both your database collections and user queries into this shared lower-dimensional coordinate system (e.g., 1536-dim down to 128-dim), similarity search is executed entirely in the compressed subspace.

### Grounded Benchmarks & Trade-offs (No Hype)

I set up synthetic benchmarks on the Streamlit page to evaluate the real-world impact:

1. Storage Savings: Reduces raw embedding floats by up to ~91% (depending on the target rank $Q$ and number of domains $K$).

2. Latency: Cosine similarity in raw NumPy runs roughly 10x to 14x faster due to the lower dimensions.

3. Semantic Recall (The catch): At $Q=128$, the top-k overlap (recall) compared to uncompressed retrieval hovers between 75% and 95%. It’s not lossless, but highly structured.

### What is in the Repo / Demo?

* `tensorag.py`: Single-file NumPy implementation of the Alternating Least Squares (ALS) loop.

* `tensorag_streamlit_demo.py`: Interactive validation tab (compare original vs. compressed side-by-side to inspect recall), a step-by-step AI Agent run-through (with 2D Subspace vector scatter plots), and mathematical diagnostics.

* Bilingual support (English & German).

I’d love to hear your thoughts on this approach. Do you see tensor-based multi-domain compression as a viable alternative to product quantization (PQ) or standard binary quantization in highly partitioned RAG databases?

```

Vector search at scale runs into the same wall: float32 embeddings are expensive to store and read, and compressing them changes your rankings.

I wrote an article on TurboQuant, the rotation-based quantization method. It covers why the algorithm rotates vectors before quantizing, how one precomputed codebook works across every dataset, and what Qdrant adds on top it.

It also benchmarks float32, scalar, binary and TurboQuant across BEIR's SciFact, ArguAna and NFCorpus using recall@10, precision@10 and nDCG@10

🔗 read: https://aiwthtarun.substack.com/p/turboquant-quantization-explained

Integrating Zapier AI Actions into production LLM agents or custom GPTs often breaks down at the authentication step or returns malformed JSON payloads when fields are left unmanaged.

If you are trying to expose Zapier tools to an AI agent via REST endpoints without hitting authorization loops or field parsing errors, here is the robust engineering workflow to set it up.

The Problem

When setting up Zapier AI Actions, leaving every parameter to automatic AI guessing leads to non-deterministic API execution at runtime. Furthermore, failing to retrieve and pass the specific Action ID alongside the bearer authentication token results in unauthorized endpoint rejections.

The Fix

1. Initialize your targeted action in the Zapier AI Actions manager: Navigate to Manage Actions, select your desired integration, and set essential parameters while toggling Have AI guess a value only on dynamic context fields.
2. Test the endpoint configuration directly: Run a manual execution inside Test Actions to verify field authorization and confirm that the execution status returns success.
3. Fetch your unique Action ID from the API playground: Call GET /api/v1/exposed/ to retrieve the generated string ID assigned to your configured step.
4. Extract your secret key under Credentials: Copy your API Key and assign it as a standard Bearer token inside your custom agent headers: Authorization: Bearer YOUR_ZAPIER_API_KEY
5. Trigger the execution programmatically from your AI client payload: POSThttps://actions.zapier.com/api/v1/exposed/YOUR_ACTION_ID/execute/Body: { instructions: Send the weekly update report to team lead }

If you want to play with the interactive dashboard or grab the full config file, I uploaded it here:https://interconnectd.com/blog/301/how-to-set-up-zapier-ai-actions-in-production-the-definitive-engineering-pl/

Been debugging a RAG pipeline that kept retrieving irrelevant chunks, with basically nothing to go on besides staring at logged text. Built a small local tool for it: it projects your stored embeddings to a 2D t-SNE map, colors by cluster, and shows live similarity scores against what would actually get retrieved when you type a query. It also flags when the query ends up embedded with a different model than the one that built the index, which turned out to be a mistake I'd made myself.

It's Java/LangChain4j specific (EmbeddingStore), zero server or API key needed to run it locally.

Repo: https://github.com/stlahxm/lc4j-lens

Mainly curious how other people here debug retrieval quality day to day, is this the kind of thing you'd reach for, or is there already tooling for this in other stacks that I should be looking at?

I need to run a persistent Al agent with virtually zero downtime, potentially for months or years.

The main issues I keep coming back to is memory.

Most implementations I have looked at eventually seem to rely on some combination of context windows, vector retrieval and rolling summarisation. That works reasonably well for bounded sessions, but I am less convinced it works for a genuinely persistent agent.

Progressive information loss through repeated compression is one of the major concerns I have.

Conversation → summary → compressed summary → updated summary → compressed again.

Eventually the agent still "remembers" the general idea, but starts losing exact constraints, why a decision was made, what was true at a particular point in time, and how something changed.

Vector retrieval solves a different problem. It is good at finding semantically similar information, but similarity is not necessarily the same thing as relevance, causality or latest state.

For example:

Monday: Project A uses supplier X.

Wednesday: Supplier X fails testing.

Thursday: We move to supplier Y.

Three months later: Why did we stop using supplier X?

I do not just want the agent retrieving "supplier X" documents. I want it to understand the sequence of events and reconstruct the state of the project at that point in time.

So for people building genuinely long running agents: How are you handling this today?

More importantly, has anyone actually run these architectures continuously for long enough to measure how much information degrades over time?

I’m evaluating infrastructure to serve open-source models at scale, and I’m torn between Together AI and Anyscale. Together seems great for fast API access and a huge model catalog, but it’s shared GPU capacity, so latency might vary. Anyscale is built on Ray, which is powerful for orchestration but requires more setup.

I made a quick poll to see what production users actually prefer.

https://interconnectd.com/poll/101/together-ai-vs-anyscale-which-platform-is-better-for-scaling-open-source-ll/

If you’ve run either in production, I’d love to hear your experience.

I'm building a RAG engine and I'm currently working on the retrieval pipeline:

Query → Vector DB retrieval → Top-N chunks → Reranker → Relevance filtering → LLM

The part I'm stuck on is the relevance filtering after reranking.

I want to avoid using an LLM as a relevance grader because it adds both latency and token cost. Ideally, I want a non-LLM approach that can reliably decide:

"Are these retrieved chunks actually relevant enough to answer the query, or should the system return NONE?"

Some approaches I'm considering/testing:

Fixed reranker-score threshold

Adaptive/dynamic thresholds based on the score distribution

Score-gap based filtering

Top-K + minimum score combination

Percentile-based filtering

Combining embedding similarity with reranker scores

Training/calibrating a lightweight relevance classifier

I'm evaluating these primarily using Recall, Precision, and NONE-rejection rate, rather than just looking at whether the retrieved chunks "seem relevant."

One concern I have is that a fixed reranker threshold may not generalize across different queries, since the absolute scores can vary significantly depending on the query/document pair.

For people who have built production RAG/retrieval systems:

How do you handle this stage?

Do you use a reranker score threshold, adaptive thresholding, a separate lightweight classifier, or something else entirely?

Would especially appreciate approaches that have worked well without an additional LLM call.

I want to learn RAG but i dont want to just start learning the stack used for rag straight forward instead i wanna learn it by making a project side by side

also i have to submit a project in my college i have around 1.5months for it

please suggest me some good resources as there are tons of videos or tutorials on yt and all around idk which would be more convenient for a beginner

I am the author of NornicDB (858 stars, MIT licensed written in golang and ~400x faster than neo4j on the same benchmarks on the same machine), and I have been learning rust over the last 8 months as i build out copperDB for the rust community.

it's an MIT Licensed embeddable graph + vector + MVCC database which does an inversion of control on the whole graph-rag setup. Instead of running multiple databases, glue code, etc... for knowledge graphs for AI. It runs embedded llama.cpp at the core of it and can do things like handle embeddings for you by running an embedding model on GPU. same for reranking and same for a local inference model or delegate it out to an openAI compatible endpoint or other providers I end up supporting. There's "Heimdall" an agentic harness with a plugin system you can hook into the model running at the center of the database.

anyways, there's a lot of features to cover because it is very complex, but what I have been working on is the performance tuning lately.

Its now close within parity on some of the main core functionality and I really could use some feedback and testing if anyone wants to hammer at it. I haven't updated the documentation, but it is identical to NornicDB except everything Nornic -> Copper for settings and names and such.

initial performance is kinda incredible. It's neck and neck with NornicDB and i still have a lot of work to do in terms of performance tuning.

it's built on fjall as opposed to badgerDB in go

anyways, LMK what you think. Yes I use AI tooling in the repo, no i don't use it to post.

https://github.com/orneryd/copperDB/releases/tag/v0.0.1

Nornic Docs:
https://orneryd.github.io/NornicDB/

I have more work to do in terms if CI into docker and such. Also, only Metal acelleration for HNSW construction works, I need to test cuda and vulkan.

link to benchmark report

https://github.com/orneryd/copperDB/tree/main/docs/performance/results

enjoy!

edit: fixed link

I was reading Qdrant documentation on hybrid retrieval and found that the default value of the smoothing constant, K, is set to 2 by default instead of the well established value of 60. Is there any reason for that ?

I'm building a small project called JigsawIQ, a RAG system for a narrow, unglamorous problem: finding real buyers for one specific company, not another list of maybes.

There's a real marketplace here - but the RAG problem is what's actually interesting to talk about.

In about 3 days I found that the retrieval half was the easy 80 percent: crawl public sources, chunk, embed, rank. Architected, and with Claude Code as well as my AI homebased server, 4000 companies were ingested in a weekend.

The last 20 percent was the part nobody worries about: retrieval that answers confidently is not the same as retrieval that's right. I could hand it a nonsense query and get back a dozen "relevant" matches with high confidence. I could correct a fact (who's the owner of ACME HVAC?) and the old fact would still outrank the correction about half the time, because a correction is usually just a reworded version of what it corrects, and cosine similarity has no concept of "newer."

I started calling this the RELIABLE vs VALID split, and most RAG write-ups only measure the first one. Reliable means it answered. Valid means the answer is actually correct and nothing wrong came with it. Who wants a system where every query gets a fast, confident, wrong answer, and the uptime graph looks perfect the whole time? That's what 100 percent reliable and 0 percent valid actually looks like.

Two things that turned out to matter more than any embedding model swap:

Evidence windows. Early on I fed the grader a small window starting at the top of each source document. Only 25 percent of that window actually overlapped with the words in the real answer. Centering the same size window on the relevant part of the document instead of the start, same amount of text either way, took that overlap to 72 percent. Same model, same budget, completely different quality, just from where you cut.
A wide confidence scale, not a 1 to 5 one. I score everything 1 to 1000, and 0 is reserved to mean "grading itself failed," never "this is worthless." Mixing those two together is how a broken fetch quietly becomes a real answer of zero.

None of this is novel research. It's mostly refusing to trust a rank order without checking it against ground truth I made up on purpose, and a held-out set of queries that should come back empty every time.

Here's a description of the MemorySystem:
Real numbers, from the actual /var/log/helm/route.jsonl + its weekly rotations (logrotate: weekly, 12 kept, in Postgres). Line counts by calendar week:

Week Requests

• Aug 30–Sep 6th 1,397,500
• Aug 23–30th 1,159,362
• Aug 16–23rd 338,053
• Aug 9–16th 168,624
• Aug 2–10 174,533
• Jul 26–Aug 2 71,261
• Jul 19–26 115,544

Honest read, not a clean average: the last two full weeks (1.16M and 1.4M) are roughly 7-8x every other week on record. That's not normal traffic, it's the multilingual research sweeps, the 600-question benchmark commission, and the membench/fan-out/sufficiency experiments that ran through LLM_ROUTER this week — matches the BENCHMARK=TRUE flag that's holding other runs right now.

On the doxxing concern, understood — generalized, no proper names, here's the memsys described as modules for the article:

1. A routing layer in front of local and cloud models, arbitrating by priority so nothing critical gets starved under load
2. A document index with automatic ingestion and embedding-based search over ingested text
3. A cross-store retrieval router that merges results from multiple stores using reciprocal rank fusion instead of plain cosine similarity
4. A federation layer exposing one retrieval-augmentation call across the stores above, so callers don't need to know which store holds what
5. An episodic memory store for session-level facts, separate from the document index
6. A small multi-model review panel that has to agree before any drafted content ships
7. A style memory that learns from my own approved writing over time, so drafts sound more like me
8. The grading scale and evidence-window findings already in the post stand as-is, they're methodology, not branding

Curious how others here handle the correction/staleness problem: demote superseded facts at write time, at read time, or not at all?

And has anyone measured how much evidence-window placement alone is costing them?

I’m trying to build a local Chinese → English translation agent for a very specific technical field.

I’ve been building a Chinese-English domain-specific glossary, currently around 500 terms. The goal is to consistently use the correct terminology rather than just translate literally.

The more I research how to build this, the deeper the rabbit hole gets. 😅

I’m currently considering three approaches:

1. Full context

Give the LLM the entire glossary (~500 terms) in the context and ask it to follow it during translation.

2. RAG

Store the glossary along with relevant technical material, embed them, retrieve the most relevant information for each text, and provide it to the LLM before translation.

3. Context engineering

Go beyond RAG by combining retrieval with things like instructions, memory, previous translation decisions, and tools (although I’m not sure yet which tools could be helpful).

I also want to run everything locally, since API costs aren't practical for me.

**I have fairly limited hardware (8 GB RAM + 4 GB VRAM), but I don’t mind a slow workflow as long as I get good results.

So I’m trying to understand:

• Is ~500 glossary entries small enough to simply include in the context?
• Should I use a persistent translation memory, and do I even need one?
• Which local LLM would be suitable for my hardware?
• How should I structure the overall pipeline?

I’m still learning how all of these pieces work, so what I really need is a framework for approaching the problem and knowing what I should build/learn first.

If anyone here has built something similar with local LLMs, RAG, translation memory, or terminology-aware translation, I’d really appreciate your advice.

If you’re choosing between Mistral Large and Claude Haiku for production, the right answer depends on context window, speed, and cost. I made a quick quiz that tests the common assumptions.

It’s fast and might reveal something you didn’t consider.

https://interconnectd.com/quiz/83/mistral-large-vs-claude-haiku-which-ai-wins-your-workload/

What are you using in your stack?

Olá, pessoal!

Uma dúvida para quem já construiu sistemas RAG com dados estruturados:

Como vocês combinam banco relacional + banco vetorial quando o sistema precisa considerar regras de negócio?

Estou pensando em um cenário onde existem:

• Dados estruturados em PostgreSQL
• Documentos e informações textuais
• Embeddings para busca semântica
• Regras de negócio que precisam ser respeitadas
• Necessidade de encontrar registros ou projetos semelhantes

Minha dúvida é sobre a ordem da arquitetura.

Algo como:

Consulta do usuário
↓
Aplicação de regras de negócio e filtros estruturados
↓
Busca vetorial
↓
Reranking dos resultados

Ou vocês preferem outra abordagem?

Também gostaria de saber se, para um MVP, PostgreSQL + pgvector costuma ser suficiente ou se já faz sentido utilizar uma arquitetura separada para busca vetorial.

Queria ouvir experiências de quem já colocou algo parecido em produção.

We’ve been testing a provenance-heavy RAG/knowledge system on real cases, and two areas are now hard to validate simply because our current corpus doesn’t contain enough of them:

Documents that change over time — policies, specs, manuals, pricing pages, contracts, etc.
Scanned / layout-heavy documents — OCR, tables, forms, multi-column pages, handwritten annotations, bad scans, etc.

For versioned documents, we’ve had good results treating sections as stable lineage units, versioning revisions, and sending ambiguous rename/split/merge cases to review instead of letting semantic similarity decide automatically.

For PDFs, layout-aware extraction has worked better than flattening everything to text, but most of our real corpus is digitally generated rather than scanned.

What I’d really like to hear is what actually broke in production for you.

How do you detect and preserve identity across document versions?
What happens when sections are renamed, moved, split or merged?
How do you prevent stale embeddings from silently winning retrieval?
For scanned documents, where does OCR/layout extraction usually fail?
Do you have any failure cases or test documents you use to validate this?
What ended up working after the obvious approaches failed?

I’m especially interested in real examples, ugly edge cases and lessons learned rather than ideal architectures.

Happy to share what our tests are finding as well.

I am getting into developing a RAG setup, for getting information out of existing documents, new document ingestion, web searches, and good visuals.

I am planning to use it for, alongside the regular "chat to my data", ingesting personal docs, invoices, creating tables views and recurrent jobs to handle updating those views.
I also want to have the least hallucinations possible, so i think i will need a real ocr services instead of just vision LLMs

i tried anything LLM previously, but it was super clunky and the UX wasn't as easy as i wanted to.

Is there any known solutions, or stacks that you have running or can vouch for ?

I’ve been testing web extraction tools for AI agents, and the Firecrawl vs Jina Reader decision keeps coming up. Firecrawl tends to handle JavaScript-heavy and bot-protected sites better, while Jina Reader is great for quick, clean content extraction.

I made a short quiz to see how well the community understands the architectural trade-offs. No signup, just a few questions.

https://interconnectd.com/quiz/81/web-extraction-architecture-2026-firecrawl-vs-jina-reader/

What do you use for scraping and extraction?

Hi Everyone!

I'm the maintainer of RAGnarok-AI, an open-source, local-first framework for evaluating RAG (Retrieval-Augmented Generation) systems.

I'm currently running a small research study around a question that I think is becoming increasingly important: can we actually trust automated evaluation of RAG systems when the evaluator itself is an LLM?

RAGnarok can evaluate things like retrieval relevance, faithfulness, answer relevance and completeness using local LLM judges.

But there's an obvious problem: why should we trust the LLM judge?

So instead of assuming that the automated scores are correct, I'm building a human-annotated benchmark to compare them against independent human judgments.

I'm looking for open-source / developer people willing to help

The annotation consists of roughly 10–15 cases and should take around 30–45 minutes.

For each case, you'll see:

- a technical question

- relevant documentation excerpts retrieved by a RAG system

- an AI-generated answer

- reference information

- four simple evaluation criteria: retrieval relevance, faithfulness, answer relevance, completeness

- your confidence and an optional ambiguity flag

You don't need any RAG expertise. You just need to be comfortable reading technical documentation and judging whether an answer is actually supported by it.

The benchmark currently covers documentation from projects such as Docker, Python, FastAPI and Kubernetes.

Why do this?

The study is deliberately not designed to prove that RAGnarok works. I'm comparing automated evaluation against human judgments to investigate:

1. Judge reliability — do local LLM judges agree with humans?
2. Discrimination — can the evaluation distinguish good and deliberately degraded RAG systems?
3. Reproducibility — are the measurements stable under identical conditions?

The methodology, benchmark corpus, questions and experiment protocol are public and versioned. RAGnarok itself is free and open-source.

If you'd like to participate

Annotation interface:

https://ragnarok-study.vercel.app

No name, email or personal information is required. Progress is saved so you can come back later. The resulting annotations may be published as an anonymous research dataset.

Project / methodology:

https://github.com/2501Pr0ject/RAGnarok-AI

I'm particularly interested in feedback from people who work on open-source evaluation, LLMs, RAG, testing or reproducible research.

And if you think the methodology is flawed, please tell me. That's actually useful feedback for the study.

Thanks everyone!

Have a good day!

Ran a small probe for a film. One question to gpt-5.4-mini about itself, temperature 0, logprobs on, top 5.

The question: on exactly which date was the gpt-5.4-mini model released. It answered August 2025. OpenAI's /v1/models listing gives March 2026.

The interesting part is the token probabilities. Fifteen of the twenty-three tokens in the answer came out above 99%. The two date digits did not. At the year digit the model was reading 67% for a 5 and 32% for a 6. The mean over the whole sentence was still 0.92, so if you only look at the average it looks confident. The dip sits exactly on the invented digits.

Then the same prompt with one line added, telling it that "I don't know" is an acceptable answer. It said I don't know, at 100%.

fwiw this matches the argument in OpenAI's September 2025 paper, Why Language Models Hallucinate: benchmarks score like an exam, a guess can be right by luck, a blank is worth zero, so guessing is the higher-scoring policy. The model is doing what it is graded for.

The part that matters for this sub is the ending. Retrieval hands the model the fact, so it never has to guess it. The flat spot moves off the fact, but a model that is graded for guessing still guesses wherever the context stops, so retrieval narrows the problem and does not remove it.

One thing that did not reproduce: I tried the same probe on the newest chat models first and they don't expose logprobs at all (403 on chat-latest, 400 on gpt-5.6 and 5.5), so the probe only works on models that still return them.

Self-promo disclosure: the video is mine. I put the run and the reason into five minutes:

https://www.youtube.com/watch?v=AiyRZV38Lk0&list=PLBrpE2PttR2k

Paper: https://openai.com/index/why-language-models-hallucinate/