We have a common pattern here in nixpkgs for Python applications: when a
Python package ships with either a requirements.txt or setup.py file, we
generally end up having to modify its version restriction, otherwise we have
build failures since we package only one specific version of each package
normally.
However, this end up being done in a completely ad-hoc way: some people
use substituteInPlace, some others use sed, others uses patches, etc.
In many cases, the code ends up being buggy, so it may work in one
version and breaks on the next one. We can instead implement one
standard way of doing this, and trying to be a correct as possible.
So this is what this commit does: it implements a new build hook, that
when called will automatically patch the wheel file. This is one of the
most generic ways to patch Python dependencies, and should work in
multiple cases.
Noticed option globbing when tried to enable parallelism by default
locally for most packages by default. python3Packages.yt-dlp failed as:
python3.9-yt-dlp> /nix/store/5mywvxdjkk1q6srwwwgdkzc37ibla801-python3.9-setuptools-61.2.0/lib/python3.9/site-packages/setuptools/dist.py:516: UserWarning: Normalizing '2022.04.08' to '2022.4.8'
python3.9-yt-dlp> warnings.warn(tmpl.format(**locals()))
python3.9-yt-dlp> invalid command name 'build_lazy_extractors--parallel'
The change adds leading whitespace everywhere where options might
already be present.
Pip had the option --build to build in a custom or temporary directory.
Nowadays, pip just listens to TMPDIR, which we already set.
This option was deprecated and is removed in pip 20.3.
When a PEP 517 project file is present, pip will not install
prerequisites in `site-packages`:
https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support
For the shell hook, this has the consequence that the generated
temporary directory that is added to PYTHONPATH does not contain
`site.py`. As a result, Python does not discover the Python
module. Thus when a user executes nix-shell in a project, they cannot
import the project's Python module.
This change adds the `--no-build-isolation` option to pip when
creating the editable environment, to correctly generate `site.py`,
even when a `pyproject.toml` is present.
When a PEP 517 project file is present, pip will not install
prerequisites in `site-packages`:
https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support
For the shell hook, this has the consequence that the generated
temporary directory that is added to PYTHONPATH does not contain
`site.py`. As a result, Python does not discover the Python
module. Thus when a user executes nix-shell in a project, they cannot
import the project's Python module.
This change adds the `--no-build-isolation` option to pip when
creating the editable environment, to correctly generate `site.py`,
even when a `pyproject.toml` is present.
Go beyond the obvious setup hooks now, with a bit of sed, with a skipped case:
- cc-wrapper's `dontlink`, because it already is handled.
Also, in nix files escaping was manually added.
EMP
This commit splits the `buildPythonPackage` into multiple setup hooks.
Generally, Python packages are built from source to wheels using `setuptools`.
The wheels are then installed with `pip`. Tests were often called with
`python setup.py test` but this is less common nowadays. Most projects
now use a different entry point for running tests, typically `pytest`
or `nosetests`.
Since the wheel format was introduced more tools were built to generate these,
e.g. `flit`. Since PEP 517 is provisionally accepted, defining a build-system
independent format (`pyproject.toml`), `pip` can now use that format to
execute the correct build-system.
In the past I've added support for PEP 517 (`pyproject`) to the Python
builder, resulting in a now rather large builder. Furthermore, it was not possible
to reuse components elsewhere. Therefore, the builder is now split into multiple
setup hooks.
The `setuptoolsCheckHook` is included now by default but in time it should
be removed from `buildPythonPackage` to make it easier to use another hook
(curently one has to pass in `dontUseSetuptoolsCheck`).
