Install requirements from pyproject

[ja2142]

closes #11440 ?

This is more of a proof of concept, I can try improving it and adding tests if this approach is deemed to be ok.

Since actual --only-deps option opens up a lot of rabbit holes (when installing anything other than .) and installing only dependencies of some remote package for example doesn't seem to be useful to me, this PR just handles dependencies from pyproject.toml in -r, e.g. pip install -r pyproject.toml.

I guess this is technically a breaking change, but I don't know what kind of person keeps their requirements in non-toml format in a file named pyproject.toml. -r pyproject.toml is also apparently what uv already does.

Only main dependencies (the ones in project.dependencies) are handled (for now?). Trying to do install -r pyproject.toml on pyproject with dynamic dependencies results in an error.

As is, I believe this PR is enough to address at least my use case for this: it allows caching dependency layers in docker easily, without additional tools.

[notatallshaw]

Hey @ja2142 thanks for opening a PR against pip.

There are some important design questions to consider on if this is the right choice of design from both a UX perspective (same flag or a different flag as -r?) and a technical perspective (should pip parse the dependencies or request them from the build backend?).

Please be aware all maintainers are currently volunteers, so we don't tend to have a lot of time to spend on pip, and it may be quite some time before any of weighs in on this.

[ja2142]

Thanks for a response.

I lean slightly towards -r pyproject.toml, it's short, seems fairly self explanatory to me, and it's what uv uses, some consistency wouldn't hurt. But if there are reasons against it, something like --requirements-from-pyproject is fine by me (maybe it could be a bit shorter).

As far as I understand from discussion on #11440, there is no way to get dependencies from the build system with a guarantee that build (or part of build) won't be invoked, which would likely result in a failure when used in a container with just pyproject.toml. Reading comments on #11440, it seems that reading pyproject.toml directly is a reasonable solution for static dependencies.

[rexzhang]

Could you support optional-dependencies? like this:

pip install -r pyproject.toml[test,mypy]

[uranusjr]

With not only optional-dependencies but also dependency-groups, I feel this needs a separate flag than to overload -r.

[ja2142]

With not only optional-dependencies but also dependency-groups, I feel this needs a separate flag than to overload -r.

I don't see why?

Optional dependencies can probably be easily and consistently supported like they are elsewhere, like in @rexzhang's example.

I'm not that familiar with dependency groups, but it seems like installing just a group is already possible, for example:

pip install --group somegroup

only installs somegroup, not the package itself. So it's unclear to me what would need to be done here.

One minor issue I see is that:

pip install -r dir/pyproject.toml --group somegroup

right now would install deps from dir/pyproject.toml, and somegroup dependency group from ./pyproject.toml, which might seem confusing. But this behaviour makes sense when -r and --group are considered independent of each other and can be easily fixed by specifying pyproject.toml path in --group. In any case, changing -r to something else doesn't help here (pip install --deps-from dir/pyproject.toml --group somegroup could also be misinterpreted as installing somegroup from dir/pyproject.toml).

#11440 also mentioned installing just build dependencies, but it sounds like it should just be a separate flag that works independently of -r (like --group), maybe something like:

pip install --build-deps-from pyproject.toml

Could you support optional-dependencies? like this:

pip install -r pyproject.toml[test,mypy]

By the way, judging by the names (test, mypy) these sound like development dependencies. If they are, these should probably be dependency groups instead of optional dependencies. Doesn't mean optional dependencies shouldn't be supported, I'm just nitpicking that a better example would be something like:

pip install -r pyproject.toml[gui,cli]

[uranusjr]

I don’t think your minor issue is minor at all. Also note that you can supply both -r and positional arguments together, which would make the argument even more confusing.

One possible design is to allow specifying paths to pyproject.toml as positional arguments instead, which is unambigious. That leaves an awkward issue to --group when you pass multiple positionals, but that’s fine since the same issue aleady exists when the positionals are paths to local project roots anyway. I’m not saying it’s the best idea, but IMO better than overloading -r. Adding a new flag (and disallowing positionals when it is used) should be explored too.

[ja2142]

One possible design is to allow specifying paths to pyproject.toml as positional arguments instead, which is unambigious.

You mean going back to original proposal, something like pip install --only-deps pyproject.toml? My understanding of discussion on this is that it would only be an acceptable solution if --only-deps supported all kinds of arguments pip install already supports (sdists, wheels, pypi packages, I don't even know what else pip can install). Which is reasonable (pip install <package> working, pip install --only-deps <package> failing could raise some eyebrows), but it seems like massive overkill to me - as far as I can tell, people in the issue (including me) just want to be able to install (static) dependencies from pyproject.toml, not from anything pip can install.

I don’t think your minor issue is minor at all. Also note that you can supply both -r and positional arguments together, which would make the argument even more confusing.

Isn't that --group being confusing just by itself? Unless I'm doing something wrong even pip install ./dir_with_pyproject --group g tries to install ./dir/pyproject.toml and group g from ./pyproject.toml. If nothing else, keeping this behaviour when overloading -r would at least be consistent with --group being a completely self contained argument.

Adding a new flag (and disallowing positionals when it is used) should be explored too.

I don't see any issue with positionals, it's only with --group. Did you mean --group shouldn't be allowed with this new flag (or overloaded -r)? That seems like a rather heavy handed approach, if we want to disallow things that can be confusing together then --group also shouldn't be allowed with positionals.

[pfmoore]

Unless I'm doing something wrong even pip install ./dir_with_pyproject --group g tries to install ./dir/pyproject.toml and group g from ./pyproject.toml.

Your statement here is confusing, and that might reflect a lack of understanding on your part. Pip doesn't install ./dir/pyproject.toml. That statement doesn't even make sense - pip doesn't install files, it installs packages. So pip is installing the package contained in ./dir, which is in the form of a source tree containing a pyproject.toml. That will (unless --no-deps is specified) also install the dependencies of the package in ./dir, which are usually listed statically in pyproject.toml. But they don't have to be. This is fundamentally different than -r requirements.txt, which simply reads a list of requirements from a file in requirements format.

If you want to read a list of requirements from a file in a different format, you should use a new option. That's what the --group option did, for reading dependency groups from a pyproject.toml file. To read the project.dependencies table from the pyproject.toml file, needs another option. So something like --project-dependencies ./pyproject.toml.

I'm a strong -1 for trying to cram this functionality into the -r option.

I don't actually find the arguments for this feature very compelling in any case. In nearly every realistic case I've seen, the list of requirements would be better recorded as a dependency group - certainly that's the case for those that use extras. And for the case of installing just the project's dependencies without the project itself, doing that properly requires calling the build backend - reading pyproject.toml is good enough in many cases, but it's not correct, and when it isn't, it's likely to be the cause of subtle and frustrating bugs¹.

Basically, this PR doesn't feel like a simpler way of addressing the use case --only-deps is intended to handle. Instead, it feels like a hack to do part of the job, while avoiding the important design considerations that make --only-deps hard in the first place.

Footnotes

1. Also, PEP 808 introduces additional cases where naively reading pyproject.toml would be wrong, so the design here would need updating when that PEP gets approved. ↩

[notatallshaw]

I agree that overloading -r is problematic because of the confusion about what a requirement file is vs. what a pyproject.toml file is.

I think a unambiguous flag for what this PR does would need to be something like --static-project-dependencies. This would make it clear specifically what it is getting.

However, IMO, it also makes clear it's probably too specific to for pip flags, and rather what would be more general would be a --project-dependencies flag. And it could short circuit to static dependencies when dependencies is not in the dynamic list, and call the build backend to find the full project dependencies when it is in the dynamic list.

Thanks for your PR @ja2142 but I don't think this design is going to be accepted so I'm going to close the PR to make that clear. I'm not stopping discussion, but I think if a new design is pursued it should be a new PR.