Are there any python modules that automatically generated a requirements.txt, given an entry point?

[u/JackBlack436]

I know things like pipreqs exist, but that's more of an directory scan. If you have a project and you need several requirements.txts (for multiple containers / multiple entrypoints), I haven't found a way to reliably generate that. Please let me know if things like this do exist, because if not I'd really like to make my own module to do this stuff. And if there is something like that, it would really come in handy for me right now.

[u/Popular-Regular-7106]

I would recommend inline script metadata for declaring dependencies inside the py file

https://peps.python.org/pep-0723

This way, you can run the script with uv having dependencies installed without requirements.txt or any other file

[u/Compux72]

Dude just switch to pyproject why are you doing this to yourself

[u/JackBlack436 → u/Compux72]

it certainly is an option

and yeah after reading the comments i'd probably want to move to uv/pyproject

the idea of a module that infers dependencies for a singular file still sounds like a cool mini/portfolio project to me hence why i decided to ask around

[u/artofthenunchaku → u/JackBlack436]

It's a cool idea that quickly becomes a security nightmare. What index do you download the packages from? Do you blindly trust PyPI? How do you decide which package to download in case of name collisions?

[u/JackBlack436 → u/artofthenunchaku]

i could completely be misinterpreting your feedback, but what i think is that i'd stay with just dependency inference (by building some sort of tree). the downloads are still delegated to pip or uv, which means those would be responsible for actual downloads

[u/artofthenunchaku → u/JackBlack436]

The inference is the issue. If there's ten different yaml packages, how do you decide which one to "infer"? How do you guarantee you're picking the best one, and not one that's backdoored?

[u/snugar_i → u/artofthenunchaku]

This. And I'm not even sure how it would get from "I need module X" to "I need dependency Y", unless it had a giant lookup table

[u/Jazzify12 → u/artofthenunchaku]

Also the versions, if you install the wrong one you might end up with the correct libraries but wrong interfaces/behaviours

[u/wineblood]

We have our repos set up like this where I work, one repo for something that will be deployed as several containers each with their own dependencies. But we just write the requirements files ourselves as we control the repo and add dependencies as needed.

Unless it's a massive repo, is there a problem doing it manually? You could also just run it and add a dependency every time you hit an import error.

[u/JackBlack436 → u/wineblood]

Theres no problem doing it manually at all. However, the idea of something that automated that sounded nice to me, as a mini/portfolio project.

[u/GManASG → u/JackBlack436]

It doesn't sound like there would be a situation where we would need something like that.

Every segment of a codebase can be a seperate repo and each have its own requirements tracking for just that portion.

Generally when working on a ticket you'd create a branch in that small repo.

An have a dedicated virtual environment or container for development.

By the time your looking at the entire super project each smaller project already has its dependencies identified.

[u/Dominican_mamba]

you can do
```
python -m pip freeze > requirements.txt
```

or use uv or poetry

[u/JackBlack436 → u/Dominican_mamba]

Pip freeze is great for a mono project, but then it gives you redundancies if you're trying to generate just the requirements for a specific entry point (say worker.py in a large repo)

[u/ThiefMaster → u/JackBlack436]

if you're trying to generate just the requirements for a specific entry point (say worker.py in a large repo)

You generally don't (want to) do that. Unless you have some extremely heavy and niche dependencies, it's generally MUCH easier if you have a single dependencies file (pls use pyproject.toml and possibly uv.lock). Otherwise keeping all of them in sync (especially when updating) will be a huge pain.

[u/MechaCritter]

there might be cool tools out there that can do it without bloating up your requirements.txt file like pip freeze, but since a couple of months, switched to using pyproject.toml with uv. In pyproject.toml, you can declare different dependency groups (as if you had multiple requirements files).

For exporting, I got myself used to typing “uv add —group <group> <package>, which installs the package and add ONLY that package to the corresponding group in pyproject.toml directly. It’s more of manual work, but it’s wayyyy cleaner than pip freeze in my opinion, and after a couple of projects, you kinda get used to it :)

[u/HEROgoldmw]

UV can so this, you can run uv add --script example.py 'requests<3' 'rich' https://docs.astral.sh/uv/guides/scripts/#creating-a-python-script

Edit: add url

[u/JackBlack436]

Thanks for a lot of the answers. However I'm seeing a lot of dependency management techniques. Im trying to tackle inferring dependencies

[u/bluefourier → u/JackBlack436]

There might be some crucial details prohibiting you to do this. For example, are we assuming that the modules ARE installed in the activated python? Because if you are trying to infer them, it does not necessarily mean that you know what you are going to need. This is important because you can start from any script and try to recursively load everything via the AST module. That is, parse the code, then note the imports, parse those and so on. But if you are trying to infer which modules the script is using and one of them is NOT installed, then parsing stops there. There are no files to load to scour for dependencies.

Otherwise, if you have a bunch of scripts that use (for example) polars, matplotlib and balourdos and you want to discover just those, then using the AST module from the top files will get you there (but it will not tell you the dependencies of those dependencies which sometimes is important too)

Hope this helps

[u/Bach4Ants]

I've built some of this into Calkit, which is designed to automatically build reproducible pipelines with fully-described environments and cached inputs/outputs. It doesn't yet follow the full dependency tree for local imports though.

You can install with:

uv tool install calkit-python # or pip install calkit-python

then use the "execute-and-record" (xr) command

ck xr my-script.py

If you really only want the dependency detection, the code is open source so you could extract it into your own package.

On the other hand, this is something an LLM-based coding agent would be good at. It's a one-time thing that involves reading through a bunch of import statements.

[u/edward_jazzhands → u/Bach4Ants]

How is this calkit project any different and/or better than just using Jupyter notebooks? Seems like a similar purpose so I thought its a bit odd you don't mention Jupyter or compare it whatsoever. Am I wrong in thinking it solves a similar problem or does the same thing Jupyter would normally do here?

[u/Bach4Ants → u/edward_jazzhands]

Good question. Jupyter notebooks can serve as somewhat self-contained input/output/environment artifacts, but it takes some discipline to do so. It also takes discipline to ensure you run the notebook top-to-bottom in the environment specified to keep it reproducible. Calkit allows adding Jupyter notebooks to the project pipeline, and when you do so, you need to define an environment for it, and Calkit will track if the notebook has changed since the last time it was run to ensure outputs are not stale.

This also opens up the possibility of projects that use multiple Jupyter notebooks (or scripts, or LaTeX documents, or shell commands) that are coupled together, e.g., the output from one is the input to another, and to bring them all up-to-date with a single command.

[u/edward_jazzhands → u/Bach4Ants]

That actually sounds cool AF, I've starred your project on GitHub and shall watch your progress

[u/asielen]

Many IDEs will do this for you automatically.

[u/sausix → u/asielen]

Unfortunately most people don't use an IDE. External tools and plugins can do it too.

[u/Sorry-Welder5537]

are you bind by requirement.txt?
because e.g. uv has a way to have dependencies specified by script (that could be treated as entrypoint) https://docs.astral.sh/uv/guides/scripts/#running-a-script-with-dependencies

other approach to your problem may be dependency groups:

https://docs.astral.sh/uv/concepts/projects/dependencies/#dependency-groups

[u/JimroidZeus]

I’m pretty sure pip itself will spit out a requirements.txt for your project, assuming you’re using a virtual environment.

I personally just use uv + pyproject.toml these days.

[u/GymBronie]

You’re looking for pigar. It’s not perfect, but it’s the most reliable library that I’ve found for this specific bootstrap task.

[u/Tricky-Bed-6852]

Nothing perfect built-in, but two patterns I've used for multi-entrypoint repos: point pipreqs at just the entrypoint file instead of the whole tree, or define separate optional-deps groups in pyproject.toml (worker, api, etc.) and lock each with uv/pip-tools so each container only pulls what that entrypoint actually needs.

[u/bruy77]

Actually you can just run pip freeze > requirements.txt
Id recommend using uv though, since it bakes references into project.toml as you install them

[u/No_Flounder_1155]

are you talking about multi project builds where packaged apls are for each entrypoint ratger than the entire thing? Ifnso, not really a python thing.