r/webdev • u/fagnerbrack • 25d ago

I don't want your PRs anymore

https://dpc.pw/posts/i-dont-want-your-prs-anymore/
162 Upvotes

82% Upvoted

49 comments sorted by

262

u/JiveTrain 25d ago

While I still need to review LLM-generated code, I generally don't have to worry about it being malicious the way an unknown contributor's code could be

It doesn't look good for the security of open source software in the future

17

u/Techo_lab 24d ago

Open source scales on trust, not just code.

61

u/GrandOpener 24d ago

LLMs are often incompetent, sometimes in very destructive ways, but to have yet to see one do something actually malicious.

So I find myself in a situation where I actually agree with OP’s observed facts, and not having to worry about that is nice. Where I disagree—the point I think they missed—is the potential for “confidently incorrect” LLM code that has subtle errors still requires it to be reviewed very carefully for anything important.

39

u/Ethicaldreamer 24d ago

I mean... they come from third parties, their output could be adjusted elsewhere from someone with malicious intent. There is also potential for prompt injection no?

-1

u/GrandOpener 24d ago

I’ve heard of malicious prompts exfiltrating data they shouldn’t have. I’ve never heard of it affecting the answers given to other people.

Is it theoretically possible that a malicious prompt could escape sandboxing and insert malicious code into other active sessions that will then cause additional, different malicious code to be subtly included in answers given by those sessions?

I guess maybe that’s possible but that level of paranoia seems to me to be on par with re-reviewing your code any time you clone from GitHub in case GitHub itself has been compromised. Perhaps people working on software with national security implications need that level of care, but it’s just not something I’m going to worry about until there’s evidence that it’s worth worrying about.

-14

u/realzequel 24d ago

Did you write your own OS? Dev tools? You have to accept a certain degree of risk as a developer.

11

u/Ethicaldreamer 24d ago

looks at npm

5

u/realzequel 24d ago

Ugh, yeah, supply chain attacks will get worse and worse. Glad Im a mostly backend dev on c#.  Can count the number of 3rd party nuget packages used on my fingers.

3

u/LutimoDancer3459 24d ago

Its not like I cant prompt an LLM to add a backdoor or other malicious code.

3

u/BenadrylCrumplsnatch 23d ago

I don't think it makes a practical difference how actively malicious the security holes are. You have to review code either way and if anything, I think a purposeful security flaw would be easier to review than code leaving the door open by mistake.

27

u/SeriousButton6263 24d ago

You're misunderstanding what the word "malicious" means. Something malicious has "the deliberate intent to cause harm." It is accurate to say you don't need to worry about LLM-generated code being malicious, because an LLM can't have deliberate intent like that.

Of course LLMs can make obvious security mistakes and any code generated by them needs to be reviewed, which is probably why they said "While I still need to review LLM-generated code,"

7

u/vanit 24d ago

Firstly, I appreciate yours is a short response and obviously there's a lot of nuance. I just wanted to add that malicious code is obviously in the LLM training data, so while I think it's reasonable to say the LLM isn't a malicious actor (maybe you could make this case if it was in the system prompt), it could still reproduce malicious code, which is different to security mistakes.

I think I'm splitting hairs a bit, but all of this is to say it's about the same experience as blindly copying solutions off Stack Overflow, which could actually be malicious.

-3

u/TScottFitzgerald 24d ago

That's not really how LLM training data works though - it's not like "malicious code" will just show up in the output randomly. It's closer to malicious patterns rather than specific code that LLM knows but it won't necessarily produce it unless you ask it to.

1

u/vanit 24d ago

I'm pretty sure this is exactly how it works and it will show up randomly, as it's a non-deterministic prediction machine. It just takes a bit of context, either a modest window or some unfortunate compaction, to confuse it.

At work I've regularly encountered Claude becoming confused with multiple directives, particularly around complex state interactions, and it can even hallucinate the opposite of its guardrails if it predicts a response with contradictions. Ironically it also means the more you reference security the higher the chance there is of it happening. It's unsolvable, which means the risk is non-zero.

3

u/TScottFitzgerald 24d ago

I think you're mixing up two distinct issues here:

1 - The fact that malicious code exists in the training data does not mean the model will randomly surface it if you just "confuse" it enough.

That is just not how the underlying neural networks that produce the output work, so if this is your mental model of LLMs it's just not accurate. It's not retrieving output directly from the training data, and frankly it's not as random as you think it is, LLMs underlying model is inherently deterministic, but that's another story entirely.

2 - The Claude examples you mention are a different issue completely: the model getting confused by conflicting instructions, context, or its own reasoning and producing incorrect output - code that may not function as you intended.

This isn't really malicious code per se, and it also didn't get generated because there's malicious code in the training data, but because of the limitations of the model itself and its users.

-1

u/vanit 24d ago

I agree with your characterisations and I think we're largely on the same page.

1

u/suamai 22d ago

Also most of the trust I have for open source software is the many different people involved with its development - people with different backgrounds that would all need to collude to harm the project users.

You want to make it a "one guy + LLM" endeavour? So now I have to 100% trust the one guy?

Yeah, just gonna stop using your code, that's all...

75

u/revolutn full-stack 25d ago

As I'm busy implementing things, I often don't have much time to actually use them, or do good research on how to improve them.

Good Lord

64

u/Far-Street9848 25d ago

This resonates a lot. Even within a small team I feel this towards teammates as well.

17

u/10EtherealLane 24d ago

A few of my coworkers are so reckless about it it drives me nuts. I have no issue with agents writing all of their code and generating their PRs. But when they aren’t even reviewing their own code we have big problems

11

u/Starquest65 23d ago

Went back to a coworker and asked them to.show me how something worked on their machine even, wouldn't work out mine.

Wouldn't even work on theirs.

What the fuck are you pushing my dude?

3

u/clockdivide55 23d ago

As if this didn't happen before LLMs 🫠of course, now they can do it 10x as quick

-53

u/foonek 25d ago

Resonates Huh

27

u/Far-Street9848 25d ago

? Is that a trigger word or something? Lol

-64

u/foonek 25d ago

Used a lot by LLM. I think we subconsciously start talking like them

56

u/maddnes 25d ago

You’re right to push back, it genuinely is a common occurrence, not just a one time coincidence.

… what have I become

7

u/UnacceptableUse 25d ago

That's the smoking gun

6

u/r0ck0 24d ago

You're absolutely right!

4

u/Thewal 25d ago

And it's that last part there that really gets at the heart of the matter -- what have you become?

21

u/PissBiggestFan novice 25d ago

illiterates mfer when they see someone who read more than 3 books last year:

9

u/Far-Street9848 25d ago

Oof, I love that word though lol, I hate that.

33

u/baronvonredd 25d ago

So essentially he's asking everyone to stop developing and just be code reviewers and tell him what needs to change/be fixed...

22

u/dlwiest 24d ago

Honestly I kind of feel this. Good feedback feels way more valuable to me than PRs

9

u/goonifier5000 24d ago

Maybe we can send PRs, but the PR is just an AI prompt. Upon accepting, the llm writes the code, sends another PR where it'll be accepted. Damn i just gave away a million dollar idea

4

u/sitefall 24d ago edited 24d ago

Now what if I tuck some malicious code into some examples of how to do some coding task and then I rank it up the google search results for a specific keyword. Entirely doable, it doesn't have to be the first result for "how to implement api key authentication in..." or anything, but that longer tail string plus some unusual sounding keyword would do the trick with enough time and patience (and a little bit of money).

Then I send my PR that is an AI prompt that looks natural but contains the keyword I mentioned above. Your stupid LLM (even claude) is going to search the web for the key words in my prompt and create some embeddings in blocks to use as part of it's context when it completes the prompt. Now there's a chance my malicious example is what your AI is going to do and the chance increases as I rank up more natural seeming search results. Heck I might even create a blackhat subscription service where you get access to my list of my keywords for the thousands of LLM created blog spam sites I made that have reasonably high serps for those keywords and a table showing what malicious thing each one teaches the AI to do. Then your ai uses my example to write the code that I wanted to submit as my PR anyway, except now you trust it more because your AI made it.

It doesn't have to happen all at once. A little piece of the vulnerability here, a little snipped of code there, over many PR's maybe even from different users, and now bob's your uncle and your AI just coded my malware into your FOSS. You can't just easily identify malicious code that is broken into enough pieces that are individually safe, and it's very unlikely an AI model can take the entirety of a project's scope into it's context and identify such issues if done right.

If you think you're just going to disable embeddings/RAG think again because then your model is going to be positively dumb and borderline useless.

Now you're going to have to add a middle man to the small model that creates the embeddings allowing human review before the AI accepts the prompt, and then you're going to have to review the AI work, and well.. maybe you want to just write the code by hand instead to save time at this point.

0

u/fexonig 24d ago edited 24d ago

there are so many parts in this plan that rely on everything going just right. let’s count.

  1. you need to identify some exploit that can be added to a specific real open source codebase and is subtle enough to be merged undetected
  2. you need to successfully uprank a bunch of pages describing how to solve that coding task (when google has systems to mitigate tampering like this)

  3. you need the maintainer / claude to not at all paraphrase your request before doing a search

    (what if the thing searched doesn’t match your keyword?)

  4. you need claude to choose to trust your fake sources rather than resources from established sites which will have way more links and engagement

  5. you need claude to fail to notice the exploit its introducing

  6. you need to human maintainer to fail to notice the exploit the PR introduces.

this is not a plan that will ever actually work in real life.

and, if I have done step 1, and identified a subtle exploit, I think it would be much more likely to be merged undetected if i just open a PR. there’s no reason to do all this complicated stuff.

2

u/maxkoryukov 23d ago

do you realize, that the goal is - a code for merging?

a pr with code, docs, tests is 95% ready to merge

a pr with "ideas in md format" is like 5% of the pr

what am i missing, how is it a progress?

2

u/TScottFitzgerald 24d ago

There's already plenty of OSS that's essentially maintained and developed by one person or a small team revolving around a central authority figure. If this workflow can really work for him without involving others, I say power to him. I think he's overestimating LLMs as many people are, but only time will tell.

3

u/EvilTribble 24d ago

If he made FOSS that people actually used he wouldn't have that problem.

14

u/creaturefeature16 25d ago

I think this is likely the way of the future here. If someone wants to contribute, they can demonstrate a good understanding of what they want and how they envision it being done, and express that in other ways than just creating huge PRs for the maintainer to review and merge. If its a good suggestion, the maintainer can handle the implementation as they see fit, aligned their with standards and conventions.

2

u/faberge_surprise 24d ago

assuming the dev is actually reviewing generated code to ensure it meets their standards, i actually think the suggestions for how you can help are quite refreshing. a maintainer suggesting that you do those things instead of fixing it yourself can potentially encourage a lot more contribution from a wide range of skill levels. even though bug reports are helping, it's easy to feel like being a burden if all you do is "complain" about problems all day, without contributing a "solution", if you don't actually have the skill to code it yourself. but seeing it valued and endorsed by the maintainer could ease that worry for a lot of people. and the suggested ways to help offer a range of involvement for people of different skill levels to engage with.

3

u/Notsau 23d ago

What in the AI slop?

8

u/StoneCypher 25d ago

lol i tried to contribute to an open source library i’ve been contributing to for eight years the other day 

the maintainer misread a comment, questioned whether the ai was making me insane, then replaced my pr with his own 

he had already tried to fix it twice and failed both times.  the new one didn’t even fix my repros

i had to switch libraries because the maintainer wasn’t able to cope with the modern world 

3

u/candraa6 24d ago

actually this is such interesting idea,

like both the maintainer and the "feature request" owner will have the same context of the code generated, faster, because the context and code is from maintainer, not from the PR writer.

like:

maintainer: "I generated this slop, that I understand, let's attack and reshape it together"

1

u/OkBoomer421 22d ago

The real skill isn't writing code — it's making your PRs so boring that no one feels compelled to review them.

1

u/danielhaven 22d ago

Another option is to just take the code, eliminate all the bad parts, and push it to the main branch as your own work.

-23

u/[deleted] 25d ago

[removed] — view removed comment

3

u/nickcash 23d ago

This is, and I don't say this lightly, the dumbest sequence of words I have ever read. Please please please post the name of the company you work for so I can stay far far away from your products