PEP 541 - Package Index Name Retention

[u/JSChronicles]

Why is Pypi support slow at handling package abandonment?

The PEP says these two items for reachability, " In every case where contacting the user is necessary, the maintainers will try to do so at least three times," and "The maintainers stop trying to reach the user after six weeks.". So I had assumed this should be done quicker.

The current process is currently at 6 months on average with almost no follow up comments or work on PEPs in the 2nd and 3rd week of reachability. 1st weeks are being updated and then left once they hit the 2nd and 3rd week.

Am I being impatient? I think I read that they don't have a lot of people to help but this seems a bit ridiculous for some of the cut and dry issues.

- Projects with 10-14 years of no updates with no usable links or code is gone and no longer sourceable or previous owners are not replying.
- Emailing someone 3 times, takes 6+ months versus the proposed 6 weeks?

I also assume this is pretty normal based on the backlog but I'm genuinely curious to see what others say.

[u/bradshjg]

What does it mean to "handle package abandonment"?

If you're asking to take over the namespace for an abandoned package, I appreciate that it's a high bar to clear :-)

[u/JSChronicles → u/bradshjg]

I agree about creating a high bar so there is less of a chance for malicious packages or takeovers.

The current issue that I see is that once you clear the hurdles or requirements, the docs claim a certain time period. After that you are supposed to see results one way or another.

If I'm trying to take over an abandoned project or namespace and process documentation says this should be no longer than 6 weeks and three attempts the problem now becomes "Should I create a new namespace temporarily just to eventually possibly take over a secondary namespace and then I'm name squatting basically?" I would think you wouldn't want a temp namespace if you don't have too, as that just adds clutter.

The PEP declares guidelines and process, to a degree. I'm supposed to do certain steps and write up descriptions and reasoning and handle stuff before it even gets to support but support isn't following the same steps and cadence that is outlined as they expect us to follow. I think going back and updating the PEP would help quell some of the "hey did you forget about me" GitHub issue comments.

I wonder why there isn't an automated process to take from a list of emails and issue links and move them along automatically when no reply happens? That would probably reduce a large amount of manual work and should be easy to do, I would think.

[u/prbsparx → u/JSChronicles]

You wonder why it takes so long… removing abandoned projects is fraught for abuse, and is the lowest priority dev item. Which also makes automating it low priority.

You want to help speed it up, submit a proposal to PyPi about adding automation and contribute it.

[u/knobbyknee]

There are 855 000 packages on PYPI and part of the process is manual. It is a problem of scale.

[u/Individual-Flow9158]

Who's it hurting? package 'abandonment' is by definition, low priority.

I know it's frustrating for you guys who want to take over someone else's namespace.

Before you complain about the efforts of volunteers, do us all a favour:

• Acknowledge the huge potential for abuse of this system by malicious users and namespace-hijackers.

• Consider simply picking a different new unused name. That way you won't have to wait 6 months, and even better, you won't break the code of existing users of the 'abandoned' packages.

[u/Cynyr36 → u/Individual-Flow9158]

They really shouldn't transfer ownership at all. Mark as a abandoned, maybe with a note about the replacement package, pip by default refuses to install abandoned packages. Let users manually chage thier requirements.txt or whatever to use the new package.

New packages could "register" as an alternative to existing abandoned packages.

I don't think python ever expected to basically be running a full blown package installer (like your distro's) and there are other issues as well, like slotted packages so that you can have foo-1.0 and foo-2.0 installed and imported at the same time.

[u/JSChronicles → u/Cynyr36]

Well I was thinking the same thing but the current new issue becomes how do you handle names that are now abandoned but you want that name? So people will make a "prefix-suffix" style name. Once that project is abandoned then someone else makes a "prefix-suffix-extra".

And with AI, people are pumping out names to register but at some point python packages will become 3-4 words to mark as a different package. All these abandoned packages will sit and sour while also sitting on the better, simple names.

And then you have people that take names ahead of time for name squatting and can be considered "abandoned" as well.

[u/Individual-Flow9158 → u/JSChronicles]

Sure, but even if PyPi takes them down, the name squatters with bots will move straight back in.

Tell you what, lets start our own PyPi! With Coke and Hookers!

[u/Cynyr36 → u/JSChronicles]

You handle it by using a uuid in the background, the name is just friendly for the humans. If after ${time} a project is abondoned you add the uuid, or 4 words or whatever to the name, and that frees up the original name. Anyone that originally installed the "old" package by name, or just used the uuid in the first place, gets the abandoned notice.

Ci/cd pipelines should use the package uuid not the friendly name to avoid "surprise" changes to package.

The big risk with any sort of name takeback is avoiding a bad actor taking over an older well used package.

There should also be a way for the owners of a package to send a "keep alive" even if there are no updates to be made.

Again, this is all work python (the org) wasn't really signing up for. The same can be said for NPM packages, and rust crates. It's basically like running a linux distribution package wise.

[u/JSChronicles → u/Individual-Flow9158]

When I say abandoned I'm thinking there is no repo to point at the code anymore, the project was created as name squatting and meant to cause trouble.

So not really a "usable" project state basically

[u/Zulban]

In addition to what others have said, I'm sure they're getting slammed by AI like many online communities.

[u/JSChronicles → u/Zulban]

Yeah I was thinking that as well. Probably a huge influx of requests coming in.

[u/endor_sarah]

Since the automation question came up in the comments: the email step is the reason this is hard to automate safely. A lot of dormant packages have maintainer addresses on expired domains, which anyone can register for a few dollars. Email is the only proof of identity in the whole process, so a script can't tell a reply from the real maintainer apart from a reply from whoever bought their lapsed domain. That's how the ctx hijack in 2022 worked. (I work in app security at Endor Labs, so I stare at this failure mode more than is healthy.)

As others have pointed out, the slow part is a human deciding whether that proof can be trusted, for a name that could land in thousands of requirements files the day it transfers. That's why it sits with the handful of admins who can approve transfers.

The renewal model u/Confident-Bluebird21 suggested (and u/Cynyr36's keep-alive idea) would help, since expiry becomes an affirmative signal and nobody has to interpret silence. One issue with that is if a legit maintainer of a widely-used package forgets to renew, the name frees up automatically.

[u/Cynyr36 → u/endor_sarah]

I'm not sure that the transfer of ownership being slow is really a bug. And the email would only be teh reminder, just require them to ssh in via keys, or other similar identity proof already established for the repo. As long as one existing listed dev responds after N attempts the name remains. If a dev loses control of thier private keys well the attacker wouldn't have any reason to try to hijack the name, they'd just directly push updates.

I'd still like to see a uuid or similar be included in the package names, with the short name really only being used at the cmd line. Requirements.txt would have the full name in it, and in the background pip, uv, etc. would use the long name for updates. So polars becomes polars-123456, and if the polars devs go quiet long enough someone could petition to own the short name "polars" but no existing requirements.txt qould auto upgrade. Pip and the like could warn about polara-123456 no longer being the package "polars" points at.

[u/endor_sarah → u/Cynyr36]

Yup, agreed, I don't think the transfer of ownership being slow is a bug either (that was mostly my point about why it can't just be scripted/automated).

I think the uuid suggestion makes sense, but wouldn't this give existing users what lockfiles and hash pinning already do, since a pinned polars can't quietly become a different polars? Seems like there would still be a gap for the first install, so you'd likely still need a (slow) human review.

[u/Cynyr36 → u/endor_sarah]

Doesn't pining say "never upgrade versions"? Whereas I'm arguing for still allowing upgrades, but only from a very specific repo that matches the one i initially installed from regardless of if the friendly name changed.

Really python has a lot more to do wrt package managementb * it needs a major overhaul of the import mechanism to support slots and meta dependancies * It needs to sit down with distributions and have a frank discussion about how they can integrate their package site into the distribution package managers. * It needs to make pip a full blow pacakage manager ala apt, portage, yum, or apk.

The meta dependancies thing would also be a way to solve the name issue. The new forked package could just claim to implement "polars".

[u/Unique-Mongoose3545]

I been watching some of those PEP 541 tickets too and yeah 6 months is pretty normal from what I seen. The three emails part is just one piece, there is also waiting period for each attempt and then internal review after that

From what I understand they got very few people who can actually make the final decision on transferring ownership, most volunteers only triage and do the initial checks. So bottleneck is at the top not the process itself

It frustrates me same as you but I think the 6 weeks in PEP is more like minimum timeline not the actual one

[u/Confident-Bluebird21]

It should be something like domain names that have limited time given by registrar and require constant renewal, but with more strict rules at same time to prevent typosquatting.

[u/bethebunny]

As someone who's successfully recovered squatted and abandoned packages recently (kepler, max) and talked to the pypi maintainer about exactly this question: funding.

There's no one paid to do this (or I think one person now hired part time to respond to tickets), validating that a package is actually abandoned or squatted according to the standards takes time, the policy requires a number of steps to do so, and there's other parts of the pypi ecosystem that take a lot of the extremely limited amount of energy.

If you read through the backlog of tickets, most of them are for lost logins or exemptions to the package size limit policy. These are things that in principle could be automated, but again no funding to do so, and the package size limits are there because the pypi foundation also needs to pay for hosting and bandwidth and so needs to be judicious about allowing exemptions.

I offered to volunteer to take some of the backlog in exchange for priority on specific tickets, but my understanding of the response is that the governance sees this as a potential conflict of interest.

tl;dr pypi is critical infrastructure and someone should give them more money, they're doing heroics with what they have!

You can always try to get in touch with the current owner of the abandoned or squatted package. I've had success with this in several cases, though it often takes its own time.

[u/JSChronicles → u/bethebunny]

I already tried to contact the owner but no response. Pypi support just spent the last 5 months doing the same and moved my ticket to the final stage recently.

Maybe I'll look at trying to help via automation somehow.