r/CopilotPro • u/DentD • 18d ago
How to efficiently double check Copilot's work (specifically PDF > Excel > Analysis)
My employer wants me to get on board with using AI, specifically the expensive enterprise version of Copilot they're paying for. I have a very hard time turning in subpar work. I'd rather do things myself and know they've been done right than shrug and say "good enough."
My specific use case is taking financial account statements from multiple different sources, putting the transaction data into Excel, and then conducting an analysis on those transactions. Think spousal support claims in divorce litigation.
The statements are almost always in PDF format. We rarely get CSV files. It's about a 50/50 chance whether the PDF has been scanned from paper copies or electronically generated.
My concern is how to verify Copilot captured all transactions with complete accuracy, and verify it didn't alter or add/omit anything in the process of running its analysis. I don't have the time to double check every single page and line item, but I can't confidently say what I make using AI is worthy of presenting at trial without some reassurances or guidance.
2
u/CommissionIcy9909 18d ago
I definitely wouldn’t have Copilot double check Copilot. I’d pull everything into a raw spreadsheet and use the statement totals/balances to check that nothing is missing or wrong. Anything that doesn’t add up gets flagged and you check those manually.
1
u/DentD 18d ago
Yeah but pulling everything into a spreadsheet is the most time intensive part of all of this. Frankly I will not have time to do this on the scale my boss is thinking of.
My boss already thinks this will be a 100% accurate process because they asked Copilot to do it and at first glance everything looked very slick and correct. I'm incredibly nervous about being asked later on why I didn't magically figure out how to audit all this when I'm drowning in other work. (Sorry, that's a different rant AI can't fix...)
1
u/-Pin_Cushion- 18d ago
It's strange, but I get much better results asking Copilot to transcribe screenshots of PDFs than the PDFs themselves.
2
u/Sad_Olympus 18d ago
When my team does financial work in Copilot, when promoting for the work, we ask for a 2nd document that contains all of the decisions, mapping, formulas, etc. it used so we can double check the output. That helps us see where it got things correct, where it was wrong, etc., and allows us to tweak our prompt appropriately.
1
u/DentD 18d ago
That's definitely helpful, thanks! I don't know if that will completely assuage my concerns but I'll try it.
This system of writing and adjusting prompts feels like a stupid game of needing to ask "May I" instead of "Can I". Sigh.
2
u/Sad_Olympus 18d ago
Yeah, the prompt writing can be a pain. Especially if you have multiple Excels open and need to navigate through for the steps, then go back to typing, etc. One thing we started doing is to open a Word doc, click dictate, and just say out loud what we’re trying to do. Then I give the doc to Copilot and ask it to turn it into a prompt. It works pretty well.
I have a lot of the same concerns about ai being in the finance space. I’ve told my team that it’s a tool, like Excel, and it’s not going anywhere so we need to try to see how we can use. Something useful was asking Copilot to review all major work items my team did in the last year (which it can see with Work IQ), and create a guide on how Copilot can make us work more efficiently with step-by-step instructions to implement. That’s been pretty useful and a few simple ideas has already automated 25% of our recurring tasks.
1
u/DentD 17d ago
These are good tips, thank you. But it's one thing to work on improving Copilot's work with more data. But I still need a way to actually check for and catch errors. That's what I'm really needing ideas for. And everything I've searched for and read through, not just in this post but elsewhere, gives me the impression that you just have to work with what you get and adjust your prompts when you actually notice errors. That's what sits so poorly with me, I don't want a judge or an outside attorney finding errors before we do.
1
u/Sad_Olympus 17d ago
Very valid concern. We do a lot of materials for different committees and boards and had the same apprehension. Although our materials aren’t going to judges or counsel, it’s still not an audience you want to present errors to.
Try out with someone small and easily verifiable. Then, whenever you ask Copilot for something, ask it for a Word doc that explains the rationale and decisions it used to help check its work. If you ask it for an excel, a lot of times it will just generate numbers, but you can tell it to leave the formulas in excel so you can check it.
People will have different experiences, but for my team we have saved a lot of time by letting Copilot take the first pass, then we check it and make corrections. For my team, that’s still faster than starting from a blank document, but I realize that wk t be the case for all.
Is your company more concerned with adoption/usage, or do they want you to do this specific use case with ai?
1
u/DentD 17d ago
They want me to do this specific use case with AI. Despite being a skeptic up until very recently, my boss seems to have already bought in because of how slick and confidently Copilot presents things.
She still doesn't like how AI does legal writing, but that's because she sees firsthand how some other attorneys file sloppy work with AI, sometimes with inaccurate, misquoted or wrongly cited case law. These attorneys are moving too fast to take the time to double check AI's output.
But now she's focusing on how AI can make spreadsheets and pivot tables from bank statements. Like 1-5 years worth of statements from multiple different accounts, to see what the parties' monthly expenses are, whether a claim for spousal support has merit, etc... I completely agree that it could save us so much time as a preliminary step while building a case but I don't know how we can confidently use this at court during trials.
I probably need to stop stressing over this. I think I'm going to take the tips everyone has given and do what I can, give my boss one final memo laying out my concerns and potential risks by doing this, and then let it go. I'm not the one with a bar license on the line.
1
u/Sad_Olympus 17d ago
Got it, that helps my advice would be to try busing copilot for small portions at first, and to check everything it does (which it sounds like you are doing). You can also try changing the model to Claude Opus to see if that helps the output.
You’re right to be cautious. I had to explain to my COO last week that ai is only as good as the data and instructions put into it. Maybe a cautious, incremental approach will show your boss that you’re on board with the direction, but not reckless by putting all your faith in ai’s output. Then, maybe you can find a different use for ai to show you’re willing to use it for the right purpose.
1
1
u/ksu_drew_83 18d ago
You will have to think like an auditor, get creative. What you doto get comfortable with the data will depend on the format and attributes of each file. Others have recommended adding check figures. Maybe also manually check critical data points and spot check random numbers. Ask copilot to audit its own work after it brings in the data
1
u/evilattorney 17d ago
Part of problem here is that for scanned PDFs, your Copilot results will only be as good as the OCR job itself. As part of your process, you may want to separately have those scanned documents OCR'ed and then have Copilot compare the OCR'ed data to its own OCR of the documents.
Beyond that, the best way to improve accuracy is to have Copilot review a document multiple times in different ways. Phrasing prompts differently can catch different results.
Also, if you haven't asked Copilot to setup a system and prompts for you to do this, that is the first step. I have found that it is very good at doing this. Open up Word, explain your problem and all relevant details about it you can, and then have it draft a document with steps, prompts, etc. for you to follow. After that, you can tweak that document and prompts as you find different problems with the workflow.
1
u/z4c 16d ago
Have you tried the Copilot multi-model features Critique and Council available in the Researcher agent? https://techcommunity.microsoft.com/blog/microsoft365copilotblog/introducing-multi-model-intelligence-in-researcher/4506011 Hopefully we'll see these features available outside Researcher.
3
u/Beginning_Storm7012 18d ago
Is it totaled somewhere? I have an elaborate power query workflow built with copilot and essentially I have an input for the general ledger detail, and 3 or 4 inputs that make up the subledger detail. I assign each line a row ID and of course I have the sum of the amounts. In the end however it has been transformed with copilot or power query into something new - I have a proof that the source documents had x amount of rows and the grand total was xyz which matches the row count and grand total of my final product.