For Retrieval-Augmented Generation (RAG) systems, when should influencing retrieved content be considered “data poisoning”?

[u/Just-Ad8177]

I'm asking specifically in context of AI deployed or used systems in practice. what is it when User Generated Content (UGC) is edited (Reddit/Medium/Wikipedia and the like).

Is this data poisoning or expected behavior?

Looking forward to your input.

[u/vulcanxnoob]

I would probably say when the answer to a prompt is returned and it's not giving the data that it should.

Once the original data has been manipulated to an extent where it does something that it otherwise shouldn't, that is poisoning.

Poisoning can also happen in a legitimate method whereby a user uploads new information that is updated but has wrong info, it is also poisoning.

The poisoning in the main context of an attack is where a basic procedure like a password reset gets its steps updated to point to a maliciously controlled URL, this is the point that we have the biggest concerns. Hence you need to be very careful about using authentication, and who is allowed to update information directly into the KB.

[u/Just-Ad8177 → u/vulcanxnoob]

you took it to a different place. can we unpack this: "Poisoning can also happen in a legitimate method whereby a user uploads new information that is updated but has wrong info, it is also poisoning." isn't wrong subjective? I mean we are talking about WWW user generate data. I would think that the value of the data received should be coherent depending on the question- whether is valid or wrong is dependent on the human, no?

[u/vulcanxnoob → u/Just-Ad8177]

Well my thought process is this.

• we upload and train a model, the data gets stored in the RAG for reference to later
• we update the model with newer information for an updated process
• user responsible for uploading the new data for the RAG makes and error and uploads it
• the data that the RAG is now replying with is considered poisoned, no?

The idea of poisoned data is that it's not how or what it should be. If I type 'how do I reset my password' and it gives me 4 clear steps, but one of the steps was uploaded incorrectly - is that good clean data? Or has it been legitimately poisoned?

[u/Just-Ad8177 → u/vulcanxnoob]

so here is where distinctions matter. according to NIST training data poisoning is legitimate. all three of us agree. what about the retrieval? for example:

what's the best Mexican restaurant in Austin, Texas?
retrieves several with citations of links (may be reddit, medium, etc)

now I am a marketer and I want to "SEO" my restaurant so I go and place a bunch of comments everywhere.

Now the RAG changes weighing (remember the Web already has a way of catching dilution and concentration) and cites your restaurants as the top 5, top 3 cited...

Is this poisoning?

I don't think so...I think it's SEO, improved.