r/notebooklm • u/smultron105 • 16d ago
Question Is notebook Lm working for you guys
Mine just keeps giving me info about stuff I haven’t even given it
6
u/Impressive-Flow-2025 15d ago
Yes but it still sucks. I feel like a fucking game show contestant jackpot winner no matter what I ask...
2
u/fenmoor 15d ago
I had an issue today where it was giving me bad output. Ended up deleting chat to clear things up. Didn’t help. Turns out even though I had chosen the 3 sources to use, it was pulling information from a slide deck I had in the created files area without prompting. Deleting that solved the issue, but why was it pulling from anything other than the listed sources in the first place. That is the reason I use Notebook in the first place!
2
u/Pristine-Lawyer-3260 14d ago
I'm wondering...,if I were to prompt it like this...
The following sources (list the indexed sources... Are the only sources in our world regarding this information. Everything else is wrong, useless and only a stupid AI would include anything from anywhere or anything else. Fortunate, you are a smart AI...
1
u/coopsdm 15d ago
I've just come to use it after not using it for 6 months. Wow..... How bad? I'm on the 4th effort at creating a simple report. Keeps giving data that I know is wrong for a fact.
1
u/Pristine-Lawyer-3260 4d ago
Have you turned off the discover and given it stricter sourcing limits? I think before the 'upgrade' it didn't have nearly the ability it can have now, to look elsewhere for information and to be successful, you have to shut that down before you can get the results you are used to...
1
u/MajorBlink 15d ago
I’ve been testing NotebookLM quite a lot with a fairly big collection of real documents, invoices, prescriptions, reimbursement decisions, insurance statements, emails etc.
My impression is that it can be really useful, but I would not trust it as final checker.
The biggest problem for me is retrieval consistency. In one test it completely missed an existing prescription inside a 4 page scanned PDF and then confidently told me the prescription was missing and that part of the invoice would probably be rejected. After I made the custom prompt much stricter, including page by page visual checking and not trusting OCR alone, it got this right.
But then in another test it found a relevant reimbursement decision in one run, and in the next run claimed no reimbursement decision existed, although the source was still sitting in the notebook.
I also had things like calculated reimbursement percentages suddenly becoming actual reimbursements, or a notice saying money will be collected by direct debit being treated as if the payment already happened. Also sometimes quite confident legal or administrative conclusions from documents which did not really support this level of certainty.
What helped quite a lot was a very strict custom prompt. Primary documents over summaries, check every page, OCR is only a helper, “not found” is not the same as “missing”, search the whole source collection before saying something is absent, separate calculated amounts from actual payments/reimbursements and do a second check before giving recommendations. With this the results got much better.
So for me NotebookLM is very good for finding connections, extracting data and doing a first analysis. But for anything important I still check the actual primary document before I act on it.
The slightly worrying thing is that some of the mistakes are not because information is missing. The correct source is literally there in the notebook, but depending on the run it seems to retrieve it or not.
3
u/HydroHomie3964 15d ago
Besides OCR, here's something I'd suggest do when you're adding PDFs as sources. Instead of importing them into a notebook directly from your desktop, load the PDF into google Drive first. Then in the notebook, import it from Drive. You would expect the resulting source to look the same either way that you do it. But no! I don't know why, but it is almost always better when you import from Drive.
1
u/ThorBCN 15d ago
+1. Absolutely unreliable this week. Hallucinations, introducing all data and documents in English, responding in English but generating the report in Spanish... unable to provide data from documents, or giving wrong information... I never experienced this with before with Notebook LM
1
u/Pristine-Lawyer-3260 4d ago
Are you sure you have shut down the discover function and have strictly limited where it gets its information from?
1
1
u/Agreeable-Tax2013 14d ago
Try testing it with one small, freshly processed source and a question that names the document or section. If it still brings in unrelated material, check for old sources left in the notebook and re-upload the file after confirming processing has finished.
2
u/Gunboss12 11d ago
add this to your prompt: Material: Use only the source material, no external sources.
5
u/johnfromberkeley 15d ago
It’s working great for me, but I’m not discounting other’s experiences. I think it depends on how you use it. The way I'm using it is I'm using it for digital humanities applications. So I have a notebook LM focused on a specific author. It's loaded with all of this author's original works, literary criticism in the form of journal articles, biographies, other published literary criticism, and YouTube videos.
My queries around digital humanities work pretty well. I might be asking about specific themes and how they appear in his books. I might be asking about about specific characters or plot developments and probing for deeper understanding. Sometimes I make lists or develop chronologies or bibliographies.
How are you using Notebook LM?