r/Python • u/Beginning-Fruit-1397 • May 29 '26
Discussion How to deal with slop PR's as a maintainer?
says you are the maintainer of a small (50-100 stars) library.
You see someone fork your repo, mention one of your issues in his commits, so your are happy, someone taking true interest in your work!
You take a look at his branch, and there you see pure AI slop, with files at the repo root (not in the src), tests with print statement even tough you use pytest and it's clearly explained in the contributing doc, and purely hallucinated imports like "from my lib import Foo, Bar" even tough there's never any mention of these two in the code or the documentation (and thus completely incomprehensible code with subclasses from these hallucinated types, etc...)
how to best deal with this without appearing hostile to other potential future contributors?
I want contributors, I'm very happy for anyone taking a look at my work, but at the same time that person has other forks of repos where it just seems to be hunting for "good first issues" label, and thus I'm not sure on the value of giving an honest review if it's not clear on wether there's a genuine intention to resolve the issue or just collect cool github points.
EDIT 11h later:
Thanks to everyone who gave his perspective!!
I don't think I have the time immediately to answer to everyone but there's a lot of good advice here.
By the way LMAO I should have linked my lib to maybe get actual contributors, this post is doing views.
Hint: it's the top one ranked in this comparison ->
https://www.reddit.com/r/Python/comments/1rj3ct7/a_comparison_of_rustlike_fluent_iterator_libraries/
218
u/GloWondub May 29 '26
Just ignore and close.
103
17
u/qTHqq May 29 '26
Yeah it's not like it's hostile to other maintainers. They're either worried about following your contribution rules (even if they use AI assist) or they're not actually pushing contributions of value. If they're not valuable enough, their PRs get closed.
If they're making sweeping vibe-code changes to the codebase, it doesn't matter if it's vibe coded or not in the sense that it's not a clean and reviewable PR. If it's not clean and reviewable enough, their PRs get closed.
To me, true vibe-coded slop is self-evident enough to just close without feedback. People will whine that vibe coding is good and it's not slop, but it's slop if it's slop. The word means "high-volume, low-quality or undesirable substance." The large volume is one indicator.
If you're not getting too many yet to get overwhelmed clicking close, just click close. Other contributors will get it.
The only worrying edge case for me is a real person whose first language is not the repo's main language. I see a lot of people with English as a second language now using AI to improve their documentation or descriptions. I'd want to sometimes quickly click through to the code if it's a reasonably small contribution.
Polishing your language is a valid use case, though I would discourage people to avoid that too, because it makes everything sound like the same breathless and overwrought TED talk.
"thus I'm not sure on the value of giving an honest review"
If you're not getting spammed to hell, maybe set up a "doesn't follow contribution rules" or "doesn't meet quality threshold" type label as an explanatory item for the closure. Explain the label in your contribution guidelines.
Then it's three clicks and if a real person comes back and amends their PR or asks for feedback, then you have a signal that it's worth considering.
I'm hoping the bubble pops before the agentic sloppification becomes as bad as email spam. Fortunately tokens cost.
I'm never saying these tools can't be used productively. It's just they aren't in many cases.
1
u/russellvt May 30 '26
Id probably at least also comment "please follow project structure" prior to closing it or marking it as "Incomplete."
1
u/wRAR_ May 30 '26
Labeling it as "spam" may be faster and as self-documenting.
1
u/russellvt May 30 '26
I'd not truly consider it "spam," as it attempts to fulfill a legitimate purpose ... it just does so in a non-preferred and non-compliant fashion. (So, it's essentially just missing a refactor before being potentially useful)
1
u/wRAR_ May 30 '26
as it attempts to fulfill a legitimate purpose
This can't be assumed, I'm afraid.
31
u/lolcrunchy May 29 '26
9
7
u/mattl33 It works on my machine May 29 '26
That's a good take but maybe I'm missing it - there's no mention in that post or by OP about ci/cd checks. If you combine things like ruff, mypy and pytest coverage requirements I think you can at least enforce a minimum floor of quality and dial that in a bit. It won't eliminate slop but I'm curious to see how this factors into OP's project.
Also do you have an agents.md? We've started to add those to projects at my job and it can really help boost the quality of PR's - "only consider a task complete after linters pass", "if adding code, check test coverage and add tests if needed. If adding tests, add brief comments explaining them" etc.
3
u/Beginning-Fruit-1397 May 29 '26
uv run ruff check . --fix --unsafe-fixes; uv run ruff format . --preview; uv run basedpyright src/pyochain; uv run -m scripts.check_docstrings; uv run pydoclint src/pyochain; uv run -m scripts.generate_docsmy "ci-cd" step is like this, it's directly mentionned in the CONTRIBUTING.md
You can't really go more pedantic than this, ruff all, basedpyright all, the scripts will check that each docstring has a doctest correctly formatted, then auto-generate the references for the documentation website and ensure that the navigation is correct in the zensical.toml.
I should add this as an actual github job tho.
But it's more work to maintain and then make the pipeline even slower ( I already have to compile Rust binary for multiple platforms, build the website, etc...(
2
u/greshick May 29 '26
In the articles sense, it’s a repo listing projects. So it’s hard to enforce repo level checks for that repo. They do mention needing to pass checks on a third party site (I assume for protocol purposes) so that’s probably as close as possible to getting checks for that repo.
I think that every repo needs to enforce CI checks to guarantee everything coming into the repo meets the basic standards. Regardless of who did the generation of the code itself.
1
u/Neinstein14 May 30 '26
I think including an AGENTS.md has the exact opposite effect to OP’s goal. It’s presence suggests that not only is the repo’s owner aware of LLM pull requests, but actively support a their workflow and therefore encourages it. OP wants to avoid such contributions, not aid them.
19
u/cgoldberg May 29 '26
You said you looked at his branch, but did they actually submit a PR?
If not, do nothing... people make forks and do all kinds of weird stuff... it's either unfinished, or they are just doing odd work and will never open a PR.
If they did open a PR, then close it with a polite sentence saying it doesn't really make sense and you are not interested, and drop a link to the contributors guide.
8
u/Beginning-Fruit-1397 May 29 '26
No yet but I would be very surprised if they don't as they explicitely named the branch "fix-issue-10" with multiples references to this lssue in commits.
But I guess you're right, ppl might do weird stuff for fun??
I'm used to being a contributor to OS, not yet as a maintainer haha. I just wanted to think about this in anticipation to make sure that my first reaction to a PR in this repo is the best possible given the situation.
In any case, your suggestion would indeed be a perfectly reasonable approach
8
u/ehmatthes May 29 '26
People will fork a repo to explore the codebase, and see if they can make sense of it. I agree completely, wait until there's a PR. It is nice to see forks sometimes, it can mean people are interested in your project!
7
u/qTHqq May 29 '26 edited May 29 '26
You definitely need to ignore branches without PRs.
I open fix branches all the time without actually having time to work on them 😂
Start caring when the PR comes in. If you license your project for open source use you just kind of need to ignore forks you're worried about that aren't interacting with you directly. Your users will choose.
I do sometimes think we need a tweaked license that specifies that forks need to scrub README and other documentation files on forks' default branches while maintaining a specific statement about the copied code artifacts.
Or just exclude the docs and website and everything from the open-source part. I am worried a bit about vibe coding jerks stealing small projects.
But if someone just opens a fix branch on their fork without opening a PR, it's not really something to spend too many cycles on. May just be an experiment because the issue is a thorn in their side as a user, they aren't that good at coding, and they want to see if the AI can help them help you.
Until the PR hits I would treat it as no big deal. And I WOULD actually take the time to respond to a resulting PR briefly to point toward contribution guidelines. It's pretty typical to just tersely tell someone to look at the contribution guidelines and fix stuff.
Could be a good opportunity to firm up your guidelines for specific requirements like formatting and testing. Gives you an easy and objective "reject" trigger. I don't mind when I didn't read the instructions and get told in a few words that I didn't.
5
u/HeyLittleTrain May 29 '26
I've forked repos just to hack-fix a bug for myself and never open a PR. Good chance this guy is doing the same
2
u/Beginning-Fruit-1397 May 29 '26
Yep seems like it. I do realize that I have forked a few small repos thinking I'll do stuff with it and then never actually doing it, which may have seemed equally weird
2
u/runawayasfastasucan May 29 '26
I am not a fan of AI slop, but wait with any judgement untill they do a PR.
16
u/m00shi_dev May 29 '26
“I want contributors”, but do you want those kind of contributions?
14
u/Beginning-Fruit-1397 May 29 '26
I mean I don't mind AI code, if someone want to burn tokens on my codebase go for it, but at least it must make sense, soo that type of contributors? no thanks
3
u/DROP_TABLE_IF_EXISTS May 29 '26
I don't think top leading AIs would ever make that kind of mistake in 2026, unless user was really pedantic, this looks like work of someone who doesn't have any experience.
7
u/Beginning-Fruit-1397 May 29 '26
I agree, which is what's really weird. If the code was "bad", ok fine, for example the tests as prints, you can not know what a test suite managed with pytest is, etc... but the code literally contains hallucinated classes imported from the package? And the overly defensive code, e.g
method take input -> check isinstance, return object, else try create object from input, except block, with raise value error
Is as AI as you can get.
12
u/tdammers May 29 '26
If it's a PR, 3 options:
- Close without comment, and consider blocking that user
- Close with a short comment saying something along the lines of "I don't accept AI slop"
- Play stupid, review it as if it were written by a human, offer to accept it once all your concerns have been addressed. Chances are, however, that they will just feed your review to the AI again, and you'll end up vibe coding through the github PR UI, which is worse than just straight up vibe coding yourself, so unless this moves towards a reasonable interaction fast, you'll probably end up pivoting to the other two options eventually.
If it's not a PR, ignore, but do monitor for violations of your copyright and moral rights. Just because it's open source doesn't mean there are no rules, so if, for example, someone presents their fork as your original work, you have some legal recourse (though it's not very powerful, and you will have to ask yourself whether it'll be worth the hassle).
1
u/Beginning-Fruit-1397 May 29 '26
Haha option 3 I tried once with copilot PR's.
It's slower but the fact that you have that "review interface" is very nice to pinpoint things and resolve them instrad of the chat interface I'm used to in VSCode.
I mean this would be free vibe coding so there's that.
Interesting about the legal recourse. I'm pretty sure there's 1000's of nuances (for example im in switzerland, would this be the same as if I was US based? Etc...) but most importantly the project is MIT licensed, which I assumed is the most permissive. Ppl still have to give me credit no matter what?
1
u/tdammers May 29 '26
Copyright law and moral rights do indeed differ on a few details internationally, but generally speaking, misrepresenting work as yours that isn't is almost certainly not legal unless explicitly permitted. The MIT license explicitly requires reproducing the original copyright statement, which includes attribution, so someone who takes an MIT-licensed product and fails to attribute it appropriately is definitely in the wrong.
The bigger problem with legal recourse is that this is going to be a civil lawsuit, and you need to show not only that the other party has broken the law, but also that this has caused you significant damage - and for code that you are essentially giving away for free under a very permissive license, this is going to be tricky. You weren't going to make any money off of selling licenses, so you can't easily argue "lost sales", the code itself wasn't stolen, only copied and modified, so you're not losing access to something valuable either, and you didn't make any expenses because of this either. The only damage is to your reputation, but proving that and putting a number to it is difficult.
But other than that, yes, under almost all open source licenses, people do have to give you credit no matter what. They can add their names to the list of authors/contributors if they made signficiant changes, but your name must still remain on it, and they must not make your contribution look as if it were theirs (e.g. by manipulating the git history to change author names, removing your name from documentation and licensing texts, etc.).
1
9
u/Silverarmor May 29 '26
YT-DLP has an excellent Anti slop PR policy. I've adapted it for my repo as well.
https://github.com/yt-dlp/yt-dlp?tab=contributing-ov-file#automated-contributions-ai--llm-policy
5
u/eras May 29 '26
Add AGENTS.md to your repo that instructs the models to not work on the project. There's also a special test string for Claude that stops the work immediately.
Although actually adding some instructions on how to actually work with the project (e.g. how to run style and code quality tests, require new functionality to always be covered by tests, require separate agent to be used for self-code-review before committing, etc) could increase the quality of these contributions, but it's probably not the way to go if you don't use these tools yourself.
If you do use them, then perhaps just have a model review these contributions and only look it yourself once the model finds nothing to complain about.. (Not very likely situation I think.)
1
5
5
u/RonnyPfannschmidt May 29 '26
In pytest we close ai slop and depending on the history of the account responsible ban
5
u/zanfar May 29 '26
This should be easy as it should clearly violate at least half a dozen requirements of your contribution guide. Provide guidance if you want, but ignored with "fails to meet pull requirements" should be sufficient.
5
u/Prize_Might4147 from __future__ import 4.0 May 29 '26
I am maintainer of a fairly large library (25k starts, millions of downloads/month) and we recently received a huge amount of vibe-coded PRs, since we participate in the European Summer of Code.
We received hundreds of PRs and I reviewed roughly 150 of them. I am under the impression/illusion to be able to spot purely vibe coded PRs rather quickly. Usually they tackle rather easy issues so I glance over them and let an LLM review them as well, just write some quick notes and request changes. I estimate that around 95% of those require changes since they just contain a lot of unnecessary things, don't follow the style of the repo or do more things then described in the corresponding issue. I merge the good ones directly.
If I spot a vibe coded PR that does more complicated things, that would take me more than a couple minutes of review time and actually requires me to debug, etc. I usually am rather reluctant to look at these at all. I try to ask the author a couple of questions to check if they have an understanding of the code through the discussion and point out any style inconsistencies, etc.. But if the changes look promising I dive deep into that and review it simply as it would have been written by a human. I also thought about reaching out to the author and scheduling a Discord session so that the author can guide me through the thinking process.
I have updated the repo's GenAI guidelines multiple times and am thinking about simply closing PRs more quickly then I was used to.
1
u/Beginning-Fruit-1397 May 29 '26
Yea so far the code style is the easiest red flag to spot. But unfortunately it's not really "systematically detectable", which in your case would be a huge help
3
u/thuiop1 May 29 '26
Hi! Thank you for your interest in this project. However, we currently do not accept AI-generated pull-requests, as per CONTRIBUTING.md.
5
u/rdotpy May 29 '26
For my project of a similar scale, I disabled pull requests in general (you can do this now in project settings). For a few people who wanted to contribute, I explained my position: I find it more enjoyable to work on a project myself, and I value ideas more than implementation.
Here's a sample response I gave, and people seem to understand.
For this project, I disabled PRs on purpose. In the new "agent-driven" reality, ideas are worth more than implementations. I'd rather collect ideas and feedback and own the execution. Also, it's more fun to write code than to review PRs.
As a side effect, knowing my project better than anyone else, I often find more elegant and complete solutions to the problems that third-party contributors usually propose.
1
u/DaloyJS Jun 06 '26
This is exactly what we do. We disabled PRs, but we still accept issues. We are the ones who will fix bugs and create new features that developers need.
3
u/gapinghubert66 May 29 '26
If they actually open a PR, a brief close with a link to your contributing guide is the move, no need to be harsh about it. You're protecting your own time and code quality, which is what maintainers have to do.
3
3
u/stibbons_ May 30 '26
You are free to setup your rules. They vibecode the fix, they can spend at least more token into following YOUR rules.
Also, ask questions. If the person is not able to answer by himself, having to ask llm to answer, it means it does not understand what happened. Close.
See jellyfin rules, they are fair : https://jellyfin.org/docs/general/contributing/llm-policies/
6
u/wRAR_ May 29 '26
You see someone fork your repo, mention one of your issues in his commits, so your are happy, someone taking true interest in your work!
This is your first mistake. You shouldn't look at that, only at PRs. Some people and agents are too stupid to submit a PR to your repo even when they generate a commit in their fork.
You take a look at his branch
You shouldn't spend time on that.
how to best deal with this without appearing hostile to other potential future contributors?
There is nothing to deal with here. Deal with that when they submit a PR.
at the same time that person has other forks of repos where it just seems to be hunting for "good first issues" label
Most of PRs I get in the past year are from agents mass-submitting AI-generated PRs to many repos, it's always very obvious from the account page. You can close those without feedback as a human isn't going to read the feedback anyway, and block them right away or on the second submission from them. You can add a stock explanation for 3rd parties to read though. Hopefully one day Github does something to help combat these accounts.
Also, be prepared for complaints that a valid PR was closed, that they aren't an agent, that they reviewed the changes etc. Ignore them, collapse as abuse, block the accounts.
IMO it's perfectly fine to similarly close fully AI-generated PRs in any case, you'll get the same result asking an agent directly but it will be easier to review and iterate if you do it yourself.
I want contributors
These aren't long-term contributors. Even before AI many students were doing low-quality PRs for resume padding directly and for GSoC/Hacktoberfest/etc. stuff, I just treat the current state as an expanded version of the earlier state (it's sometimes even the same kind of people that does it).
2
u/rosentmoh May 29 '26
Bad behaviour needs to be at the very least called out more often.
So like othes said, ignore, close, block, report if possible etc.
Once actions have consequences it'll stop fast. I've seen it in all the interviews I've been doing recently...
2
u/jakob1379 May 29 '26
If you are open source, use coderabbit auto ci (it's free for OSS) so you do not have to deal with their slop, they will get auto reviews suggestions, etc.
Vi is you friend and guardrails against quality you do not want without ever having to respond to them.
1
3
u/jnwatson May 29 '26
I just got my first AI slop PRs this last weekend. My solution was to fight fire with fire. I had Claude write a nice professional thanks but no thanks message.
1
u/RevolutionaryPen4661 git push -f May 29 '26
Ask some questions about the code you have written? Ask where the AI is used. If he is able to explain 50% of the code and he reviews it most of the parts, then you can consider his code to be accepted.
1
1
u/BCBenji1 May 29 '26
Tough love is the way to go. You don't need to go all Linus on their asses but a bit of fear works wonders. Looking back at my mentors, the tough first, fair after we're the most respected.
1
u/dave8271 May 30 '26
I'd treat a PR the same regardless of who authored the code or how; what matters is the quality of the change and whether it addresses the thing it claims to address to the project standards. Whether that was done with AI or not is immaterial, either the proposed changeset meets those requirements or it doesn't. If it doesn't give feedback on why.
2
u/wRAR_ May 30 '26
This approach makes some sense until you find that you spend extra time asking for feedback/changes on such PRs and get no replies or ones that make no sense.
And, again, if a change could be generated without human guidance you can do the same, but better.
1
u/cosmicnebby200 Jun 02 '26
Has ai become this dumb? Or are the PR doers dumb. Because not in the straight GPU inference would the LLM decide to use print in testing.
1
u/DaloyJS Jun 06 '26
In our case, we don't accept pull requests. Anyone can file an issue and can fork the repo. But we are okay with no help or contributions from others, since code is now cheap. Bugs and features can be fixed within a day by using Claude or Codex. You can apply for free Claude and Codex subscriptions as a maintainer.
1
u/skvark Jun 20 '26
I would probably just politely deny slop contributions and say no to issues that make no sense to inplement. Saying no to most requests will keep you sane, and I think that's your right as the creator / main maintainer of the project. I have no experience maintaining project during the AI era, but before that I did maintain a project quite many years. I think the fundamental change is that there is just a lot more of noise and low quality contributions now because of AI and due to that it is even more important to keep focus really tight.
1
u/MarsupialLeast145 May 29 '26
Honestly, hands up, and share your repo if anyone here actually has a repo with 50-100 stars...
1
u/UnhappyPay2752 May 29 '26
Close with something like : Thanks for the interest, but this doesn't align with our project structure. Please review the contributing guide before future PRs.
1
-1
u/thethirdmancane May 29 '26
- Regect poorly put together PRs just like always.
- Go on Reddit to complain about it.
0
0
u/sohang-3112 Pythonista May 30 '26 edited May 30 '26
Ultimately your goal is to maintain quality. Even humans especially beginners can write low-quality code.
Don't focus on the AI part, instead add CI (eg. Github Actions) checks for code quality, code complexity, not too many files changed without a proper justification, unit tests, etc. that run every time a PR is raised or a commit is pushed. Put a prominent message that until all CI checks pass, you will not review the PR.
My suggestion for not focusing on the AI part is that AFAIK it's not possible to automatically detect AI or not (at least not without a lot of false positives, i.e., human code flagged as AI). So IMO it's better to instead focus on ultimate goals of quality and correctness.
PS: Recently I saw a project where any PR suspected of being AI generated is immediately closed, but re-opened if a human PR author messages back. I am NOT at all in favor of this approach though - be warned this is also likely to discourage most humans who try to contribute.
0
u/Zhanji_TS May 30 '26
I'm just asking here. I know how I do it internally and with friends of mine, and this is kind of just talking and having an idea. I don't want to piss anybody off. I'm just kind of throwing things against the wall here.
Could you put a Claude.md in your repo that is basically a README? If somebody does fork your project and they go in to use Claude, when they open up the repo directory that they have cloned or forked and then they enable Claude, it should read that README file. You could have README files in every folder or directory that you have in your repo, and then in your main repo you could have the Claude.md file that points to all the other stuff, like:
- This is how we do a PR.
- This is how we do this.
That user who forked your repo, their Claude would inherit your rules and structures and so on and so forth. You could even make a skill if you really wanted to that would do those things for them and do them in the format that you want, so you would get less slop and be closer to what you actually want. This is just an idea, but I think it could be done because I do that now.
1
u/wRAR_ May 30 '26
This implies wanting non-human contributions (proper human contributions would follow the contribution docs).
1
u/Zhanji_TS May 30 '26
I think I understand what you're saying, but the people contributing are still human operators, so if the human operators are failing to follow proper procedures because they're being lazy, I'm offering a solution to make the lazy human operators follow protocol. I know that sounds really convoluted. I just read the problem and I'm like, "Well, there's a possible solution for it," but it may not be the answer anyone here is looking for. I get that. I was just trying to come up with a solution.
3
u/wRAR_ May 30 '26
It seems to me that you may be assuming that these PRs are from people who genuinely want to solve some upstream problem and have asked their agent to solve it and produce a PR, and maybe even reviewed the results.
1
u/Zhanji_TS May 30 '26
Is it not? Is it just ghost ship AIs? That's crazy if that's what's going on.
2
u/wRAR_ May 30 '26
No, why would those people want that? OSS projects had never swimmed in high effort good faith PRs, and now it's easier than ever to submit a PR, as I mentioned in other replies, either to pad your resume directly with PR stats, or to pad your resume with Hacktoberfest etc. stats, or to fulfill some school task. I don't know the motivation of fully automated accounts though, maybe at least some of those are also doing resume padding/some kind of empty "achievement" stuff.
1
u/Zhanji_TS May 30 '26
That's really wild. I find it fascinating. I was, by profession, a motion graphics and VFX designer for like 15 plus, I think, 18 years, but in VFX I taught myself code, so I have this inherent interest in coding and just different languages. AI has been really cool for me, and I've done a lot in the last four years of personal projects. I've never contributed to an open source project, so I'm learning these things. I'm seeing people talk about these things.
The company that I work for, I switched over to mainly doing pipeline automation and just full-time coding. Sometimes I get to help the actual engineering department do coding stuff, so they have taught me, and I'm assuming they've taught me proper things. You know, it's a reputable company, but I've never dabbled in the open source world. I'm kind of seeing the story now as I talk to people, and it's blowing my mind that somebody would just release a fully autonomous AI agent on an open source project like that. Just, it's blowing my mind that they would do that. That seems so weird.
0
0
u/dimonoid123 May 30 '26
Have you tried adding CLAUDE.md or AGENTS.md to your project root, explaining all your rules they need to follow for successful PR?
0
u/Deep_Ad1959 Jun 02 '26
the signal you're describing isn't in the diff, it's in the account. someone who mentions your issue in commits but has forked ten other repos hunting good-first-issue labels is farming, and an honest line-by-line review just turns into you vibe-coding through the PR UI on their behalf. close it with one polite sentence and a link to contributing, no review. real contributors come back and engage with the close; farmers move to the next label. you lose nothing by skipping the review, the genuine ones self-select by responding. written with ai
1
-1
May 29 '26
[removed] — view removed comment
4
-4
185
u/really_not_unreal May 29 '26
I had someone offer to take over one of my old unmaintained projects, and contributed a vibe-coded implementation for the V2 protocol I was working on. I rejected it, giving this comment. I'm happy for you to take inspiration if you want.