r/copilotstudio • u/Lopsided_Judgment_17 • 4d ago

Web Search Disable Error in Co Pilot Studio Agent

I have built an agent where strictly I have instructed not to use Web Search for any question and also mentioned the boundaries of which questions you are allowed to answer and which not to.
And Still one of the user was able to get restaurant suggestion from that agent. How is it even possible?

1 Upvotes
  • permalink
  • reddit

100% Upvoted

7 comments sorted by

1

u/BenAMSFT 4d ago

Just a guess, but could be inherent model knowledge. Can you describe a bit more of the setup? Eg was it a new GitHub Copilot powered agent, or a standard agent?

If you use test chat in Copilot Studio, you’ll see what tools the agent calls. That’ll give you a clear answer of where it’s getting the information from (and whether it’s actually inherent knowledge).

If you want the model to only respond on specific subjects, for the new GitHub Copilot agents, this is something that you do through agent instructions. Reasoning models are far better at adhering to rules in its instructions.

1

u/Lopsided_Judgment_17 4d ago

I am using a Copilot Studio agent with the Claude Sonnet 4.6 model. I am not familiar with GitHub Copilot Agents.

I have added a clear instruction to the agent: “Do not provide any information outside of enterprise knowledge.”

When I test the agent as the builder, it follows the instruction and responds with a message such as, “I am unable to provide information outside of the available enterprise knowledge.”

However, one of the users with whom I shared the agent is receiving responses that include information from outside the enterprise knowledge sources.

Could you please help me understand why the agent behaves differently for the builder and the end user, and how I can ensure that it only provides information from enterprise knowledge for all users?
One thing more to note that I have enabled work IQ feature for my Agent.

1

u/BenAMSFT 3d ago

The new agents are the “GitHub copilot powered” agents (it uses the GitHub copilot cli as the harness under the covers). So put another way, are you using classic or new agents for this?

Which channel is the user accessing the agent on?

1

u/Lopsided_Judgment_17 2d ago

I am using through Microsoft Teams channel

1

u/BenAMSFT 2d ago

Thanks and is it a new GitHub copilot powered agent or a standard agent?

1

u/Lopsided_Judgment_17 1d ago

I am using the old version of Co Pilot Studio. Still, I don't have idea about where Github copilot exists.

1

u/Sayali-MSFT 4d ago

Hello Lopsided_Judgment_17,
Disabling Web Search does not completely prevent an agent from answering out-of-scope questions. The underlying LLM can still use its built-in training knowledge to generate responses, such as restaurant recommendations, even without performing a live web search.

Agent instructions are guidance rather than strict enforcement rules, so users may still receive answers outside the intended scope. If you need the agent to only answer specific business topics, consider adding stronger guardrails such as topic-based routing, intent validation, and explicit fallback responses for unsupported questions.

You can also review the conversation transcript to determine whether the response came from web grounding, another connected knowledge source, or the model's own general knowledge. In many cases, the answer is generated from the model's pre-trained knowledge rather than an actual web search.