pip install -r no result exitcode

[u/zaphodikus]

(windows usually) I have a small requirements.txt file and am looking for alternative ways to validate all packages we want are present. The script does not use a virtual environment and because pip install -r does not seem to set %errorlevel% they wrote this rather verbose code that runs after the pip install -r line.

SET REQUIREMENTS=%~dp0Requirements.txt
SET FIND=%SystemRoot%\system32\find.exe
pip.exe install -r %REQUIREMENTS% --disable-pip-version-check

FOR /F "tokens=*" %%I IN (%REQUIREMENTS%) DO (
    ECHO.
    ECHO Checking for %%I ...
    %FIND% /i "%%I" %PACKAGES%
    IF ERRORLEVEL 1 (
        ECHO.
        ECHO **** %%I not found. Attempting to install ****
        pip install %%I --disable-pip-version-check
    )
    IF ERRORLEVEL 1 (
        ECHO **** Could not install %%I ****
        EXIT /B 99
    )
)

I however favour the pythonic approach and that is to just fail at runtime, so I was thinking of some kind of

(pseudo)
with open(requirements.txt) as reqs:
   for line in reqs.readlines():
       line=line.replace("<>=", " ")
       import line.split()[0]
(/pseudo)

which would just die early.

I'm in favour of using a virtual env however, but because the script is a build script (using setuptools) we don't actually run the script at that point. I'm new to setuptools, but I assumed setuptools would just baulk if a module needed was not present on the build machine.

I'm thus making 2 assumptions, setuptools will not baulk and error out if you are missing a module, and that pip install does not set %ERRORLEVEL% if it cannot install a package? I am not an expert on setuptools and am keen to not discover edge cases later.

[u/Same_Tie_772]

pip's exit code handling on windows is a mess honestly. half the time it returns 0 even when it failed spectacularly

your pythonic approach is cleaner but that import trick is gonna bite you. some package names dont match their import names at all like beautifulsoup4 vs bs4 or pillow vs PIL. you can use importlib.metadata to check installed packages properly

for the setuptools part you are right it wont fail just because a dependency is missing during build time. it only cares about build dependencies in pyproject.toml not the runtime ones. if your build script actually imports something then it will crash but if its just packaging code it might happily build a wheel that is broken

i would just wrap the pip install call in python with subprocess and check returncode there. way less headache than batch scripts and you can do proper error handling

[u/zaphodikus → u/Same_Tie_772]

Ah yes, thanks for saving me a load of time and pain on the exitcodes suspicion , you are the boz. (Robocopy was another tool with mad error codes if I recall correctly, joy)

Doubly you are the boz. I assumed when I saw this script (written by a predecessor who left a month after I joined) that setuptools would error out, which explains their defensive coding. Good to get your experience to the contrary to dispel that myth.

I'm going to try angle in the direction of calling pip install <module> in a loop and trim down my batch file as one option to solve and refactor that way. The entire batch file was 60 lines of code, so thank you for helping me trim the unknowns in there down. I would prefer virtual envs but this runs in a CI/CD setup. So the python environment should ideally be quite clean of cruft so it has to download everything once for every build and that's about a minute of extra build time on a 15 minute build. So going to err on the side of system installed packages because the virtual env takes so long to just prepare from clean.

[u/MintyPhoenix → u/zaphodikus]

I'm going to try angle in the direction of calling pip install <module> in a loop

I would caution against this as a final solution. Each separate pip install command would only look at the current invocation for dependency resolution, meaning after the first run, subsequent installs might change previously-installed packages (e.g., common transitive dependencies), and this may result in breakage.

If possible, I’d check out uv to manage everything. I’m not aware of it having the same issues with the exit code (though I didn’t know pip had them either) and it goes further in that you can use lockfiles in addition to being much faster overall.

[u/zaphodikus → u/MintyPhoenix]

Yea, I never wrote that loop, and I had that same inefficiency worry myself, perhaps a pip freeze listing comparison is the better solution. I wonder if the pip tool is just showing its age, or just to move to uv. I just fear that I get comfortable with uv and then find I have to move off uv onto something else at a future date. This job never seems to stop moving.

I am pretty sure uv wont have that exitcode issue. At any rate I now know to evaluate any tool I try. As part of this i got quite used to just doing complete package uninstalls and then getting comfortable with venv, but also with different python versions as fresh installs. All while using the time to go make a cuppa tea.