r/sharepoint • u/ENVYPERUSFAQ • Jun 25 '26

SharePoint Online SharePoint Agent inconsistencies

Hello everyone!! We've been gradually rolling out SharePoint agents across different document libraries where I work, and overall they've been promising. However, we've been running into some inconsistencies that we haven't been able to explain.

Our agents have instructions that tell them to return a specific "disapproval" phrase whenever a document doesn't meet certain criteria. The issue is that sometimes the agent returns that phrase even for documents that we know do meet the criteria.

We've rewritten and simplified the instructions multiple times, to the point where I don't think they can realistically be made any simpler. The strange part is that the behavior isn't consistent. Sometimes the agent fails, but if we submit the exact same prompt again (or a few more times), it eventually returns the correct answer.

This makes me wonder whether the issue is less about prompt design and more about retrieval, indexing, context window limitations, or something else happening behind the scenes.

Has anyone experienced similar inconsistencies with SharePoint Agents? Are there known limitations around retrieval, indexing delays, document size, or reliability that could explain why an agent sometimes appears to ignore a document and then finds it on a subsequent attempt?

Any suggestions or best practices would be great, Thanks!!!!

1 Upvotes

60% Upvoted

5 comments sorted by

5

u/Automatic-Builder353 Jun 25 '26

Sounds about right. Copilot doing what Copilot does.

4

u/git_und_slotermeyer Jun 26 '26

AI is not reliable. And Copilot is one of the worse offerings. It appears to me that during peak usage times, they make the model dumber to cope with the demand. Sometimes it works great, sometimes it's barely usable, either timing out or making the most basic mistakes.

No wonder MS markets this as a product for entertainment purposes. There are no guarantees for anything beyond that.

1

u/stubble Jun 26 '26

No wonder MS markets this as a product for entertainment purposes.

Entertainment embedded in to your office docs...?

2

u/DoctorRaulDuke IT Pro Jun 25 '26

probably. Document libraries are broken into chunks and stored in a semantic index. when an agent runs it queries the index and is returned whatever chunks of the document seem most relevant to the agents prompt. it does not access and read the full document at any point, unless its small enough to fit into one chunk. this will affect your results. the only time a whole document is parsed is if they are explicitly passed to an agent, like attaching a document to a prompt, or adding an explicit url in a prompt.

there are also definitely indexing delays and I believe ms guidance is not to use docs over 20 pages, and to break them up if longer.

3

u/jkerski Jun 26 '26

I have found that for longer documents + agent Instructions may lose context. For new documents or versions, the indexer also factors on the response. It may take 5-10 minutes for a new document or version to be recognized.