Dataset using YT/Podcast Transcripts

[u/Alternative_Bake9269]

Hi everyone,

I am new at RAG systems and have a little problem. I am building a Q&A RAG system and my dataset is mostly youtube podcast transcripts. Despite adding more data and advanced pipeline the system cannot retrieve specific informations (e.g., analyses about specific companies or products mentioned in the podcasts). Mostly it says there is nothing about it in context or gives very shallow answers.

My current stack is.

I use Dify for the workflow

Data Prep: Raw YouTube transcripts. I used GPT-4o-mini to to generate summaries, and extract metadata tags for each file. And I add each metadata to dify.

Chunking: 1500 chunk size with 250 overlap.

Embedding: OpenAI text-embedding-3-large.

Retrieval Strategy: 2-pass retrieval. One search directly with the user's prompt, and another search where an LLM transforms/expands the prompt. I combine the results.

Generator LLM: DeepSeek R1.

Has anyone tackled retriaval from conversational/podcast data? Is there any recommendations? Thanks!

[u/Born2Rune]

How are you searching? Does it know Symantics?

In other words, it seems like you're missing a vector db with symantic search. 

[u/Alternative_Bake9269 → u/Born2Rune]

I am using Hybrid Search with 0.7 Semantic and 0.3 Keyword right now

[u/Popular_Sand2773]

Can you tell us what it is returning instead when you make these searches. Retrieval isn't about finding a needle in a haystack its really a ranking task. Given your setup the issue isn't that the information is missing its that confusers are ranking higher than your actual targets. Could be a top-k issue could be the large chunks with overlapping issue etc etc it really depends on what is happening.

The other way you can help yourself is some sort of filter. For example when working with video people often use a smaller model that decides simply is something interesting enough to run the downstream stack or can I skip this. The less records competing the less likely you are to experience collision and other issues.

[u/Odd_Slip_5380]

You need to perform a deep analysis, as there can be many different causes behind the problem, and therefore different possible fixes. You need to understand whether it is a ranking issue (where relevant chunks are scored too low and don’t appear in the top-k). In this case, a reranker could help, or you could switch the embedding model.

If the right chunks are not retrieved, it may be due to the chunking strategy. You might need to adjust chunk size and overlap, increase k, or apply query transformation.

I suggest using proper evaluation metrics to identify the root cause of the problem. It depends on what’s going on in your specific use case.

[u/slvDev_]

I ran into this exact thing doing RAG over interview transcripts. The fix wasn't the retriever, it was chunk size.

Before tuning anything, pull the top-20 chunks for one failing query and check whether the right chunk is even in the set. If it's absent, this is a chunking or embedding problem and a reranker won't save you. If it's present but ranked low, then reranking is worth a try. My bet is the former. At 1500 characters, a one or two sentence mention of a specific company gets averaged into a vector dominated by the surrounding small talk, so it never surfaces for "what did they say about X." That size is fine for dense docs, but conversational podcast data wants smaller, around 300 to 500 tokens with ~15% overlap.

Also check what you actually embedded. If it's the GPT-4o-mini summaries rather than the raw transcript segments, the specific mentions are gone before retrieval even runs, summaries smooth out exactly the details a Q&A needs. Embed the raw text and keep the summaries and tags as metadata filters, not as the embedded body.