1
0
Fork 1
mirror of https://github.com/NixOS/nixpkgs.git synced 2024-12-11 15:15:36 +00:00
Commit graph

863 commits

Author SHA1 Message Date
Greg Price 52c04b0347 cpython: Use autoreconfHook to rebuild configure script.
In particular this will let us use patches that apply to configure.ac.
2020-05-13 21:23:48 -07:00
Greg Price f8a8243bd3 cpython: Use --enable-optimizations, for a 16% speedup.
Without this flag, the configure script prints a warning at the end,
like this (reformatted):

  If you want a release build with all stable optimizations active
  (PGO, etc), please run ./configure --enable-optimizations

We're doing a build to distribute to people for day-to-day use,
doing things other than developing the Python interpreter.  So
that's certainly a release build -- we're the target audience for
this recommendation.

---

And, trying it out, upstream isn't kidding!  I ran the standard
benchmark suite that the CPython developers use for performance
work, "pyperformance".  Following its usage instructions:
  https://pyperformance.readthedocs.io/usage.html
I ran the whole suite, like so:

  $ nix-shell -p ./result."$variant" --run '
      cd $(mktemp -d); python -m venv venv; . venv/bin/activate
      pip install pyperformance
      pyperformance run -o ~/tmp/result.'"$variant"'.json
    '

and then examined the results with commands like:

  $ python -m pyperf compare_to --table -G \
      ~/tmp/result.{$before,$after}.json

Across all the benchmarks in the suite, the median speedup was 16%.
(Meaning 1.16x faster; 14% less time).

The middle half of them ranged from a 13% to a 22% speedup.

Each of the 60 benchmarks in the suite got faster, by speedups
ranging from 3% to 53%.

---

One reason this isn't just the default to begin with is that, until
recently, it made the build a lot slower.  What it does is turn on
profile-guided optimization, which means first build for profiling,
then run some task to get a profile, then build again using the
profile.  And, short of further customization, the task it would use
would be nearly the full test suite, which includes a lot of
expensive and slow tests, and can easily take half an hour to run.

Happily, in 2019 an upstream developer did the work to carefully
select a more appropriate set of tests to use for the profile:
  https://github.com/python/cpython/commit/4e16a4a31
  https://bugs.python.org/issue36044
This suite takes just 2 minutes to run.  And the resulting final
build is actually slightly faster than with the much longer suite,
at least as measured by those standard "pyperformance" benchmarks.
That work went into the 3.8 release, but the same list works great
if used on older releases too.

So, start passing that --enable-optimizations flag; and backport
that good-for-PGO set of tests, so that we use it on all releases.
2020-05-11 23:37:04 -07:00
Jonathan Ringer 884436b254 pythonPackages.pytestCheckHook: disable setuptoolsCheckPhase 2020-05-11 22:12:08 +02:00
Josef Kemetmüller 3818cd9049 python: Fix creating RPMs from Python packages
This should enable (manual) building of RPMs from python projects using
the `python setup.py bdist_rpm` command on systems where `rpmbuild` is
not located in `/usr/bin/`. (e.g. NixOS)
The discovery of the rpmbuild command was fixed upstream in Python 3.8,
so this commit backports the relevant patch to our currently supported
Python 3 versions.

Fixes: #85204
2020-05-09 11:15:28 +02:00
Pavol Rusnak 420124adf8 python: remove isPy33, isPy34 2020-05-04 18:49:45 -07:00
Frederik Rietdijk 6f873e98f4 Python integration tests: disable for older python 3 versions
because the package that is used as part of the test does not support
older versions.
2020-04-25 07:59:37 +02:00
Frederik Rietdijk 71171b3225 Python tests: test venv from a nix env with Python 3.8
This test was disabled because it did not function yet, however,
apparently it does with 3.8.
2020-04-25 07:59:37 +02:00
Frederik Rietdijk cf1a68360e python2: 2.7.17 -> 2.7.18 2020-04-21 11:21:39 +02:00
Michael Reilly 84cf00f980
treewide: Per RFC45, remove all unquoted URLs 2020-04-10 17:54:53 +01:00
Jan Tojnar 3e0f4e202f
Merge branch 'master' into staging-next 2020-03-31 21:32:15 +02:00
Greg Price 9d8831c8fe cpython: Drop unrecognized --with-threads configure flag.
The ./configure script prints a warning when passed this flag,
starting with 3.7:

  configure: WARNING: unrecognized options: --with-threads

The reason is that there's no longer such a thing as a build
without threads.

Eliminate the warning, by only passing the flag on the older releases
that accept it.

Upstream change and discussion:
  https://github.com/python/cpython/commit/a6a4dc816
  https://bugs.python.org/issue31370
2020-03-30 22:56:48 -07:00
Frederik Rietdijk c392d70518 pkgsStatic.python3: fix build 2020-03-30 17:06:38 +02:00
Frederik Rietdijk a36be028f5 Merge staging-next into staging 2020-03-28 21:15:15 +01:00
Jonathan Ringer 279438e7f8 python: add use-pkgs-prefix option to update script 2020-03-27 01:47:30 -07:00
Jonathan Ringer a9c994ad0b python: add pythonNamespacesHook 2020-03-18 12:20:51 -07:00
Frederik Rietdijk 7447fff95a
Fix sys.prefix in case of a Nix env
The prefix will now be correct in case of Nix env.

Note, however, that creating a venv from a Nix env still does not function. This does not seem to be possible
with the current approach either, because venv will copy or symlink our Python wrapper. In case it symlinks
(the default) it won't see a pyvenv.cfg. If it is copied I think it should function but it does not...
2020-03-14 21:39:32 +00:00
adisbladis 753122388d
Python: Add integration test verifying NIX_PYTHONPATH with Mypy 2020-03-14 21:39:32 +00:00
adisbladis 05571d3059
Python Add test for NIX_PYTHONPREFIX 2020-03-14 21:39:31 +00:00
adisbladis d88a7735d2
Python: introduce NIX_PYTHONPREFIX in order to set site.PREFIXES
This is needed in case of `python.buildEnv` to make sure site.PREFIXES
does not only point to the unwrapped executable prefix.

--------------------------------------------------------------------------------

This PR is a story where your valiant hero sets out on a very simple adventure but ends up having to slay dragons, starts questioning his own sanity and finally manages to gain enough knowledge to slay the evil dragon and finally win the proverbial price.

It all started out on sunny spring day with trying to tackle the Nixops plugin infrastructure and make that nice enough to work with.

Our story begins in the shanty town of [NixOps-AWS](https://github.com/nixos/nixops-aws) where [mypy](http://mypy-lang.org/) type checking has not yet been seen.

As our deuteragonist (@grahamc) has made great strides in the capital city of [NixOps](https://github.com/nixos/nixops) our hero wanted to bring this out into the land and let the people rejoice in reliability and a wonderful development experience.

The plugin work itself was straight forward and our hero quickly slayed the first small dragon, at this point things felt good and our hero thought he was going to reach the town of NixOps-AWS very quickly.

But alas! Mypy did not want to go, it said:
`Cannot find implementation or library stub for module named 'nixops'`

Our hero felt a small sliver of life escape from his body. Things were not going to be so easy.

After some frustration our hero discovered there was a [rule of the land of Python](https://www.python.org/dev/peps/pep-0561/) that governed the import of types into the kingdom, more specificaly a very special document (file) called `py.typed`.
Things were looking good.

But no, what the law said did not seem to match reality. How could things be so?

After some frustrating debugging our valiant hero thought to himself "Hmm, I wonder if this is simply a Nix idiosyncrasy", and it turns out indeed it was.
Things that were working in the blessed way of the land of Python (inside a `virtualenv`) were not working the way they were from his home town of Nix (`nix-shell` + `python.withPackages`).

After even more frustrating attempts at reading the mypy documentation and trying to understand how things were supposed to work our hero started questioning his sanity.
This is where things started to get truly interesting.

Our hero started to use a number of powerful weapons, both forged in the land of Python (pdb) & by the mages of UNIX (printf-style-debugging & strace).

After first trying to slay the dragon simply by `strace` and a keen eye our hero did not spot any weak points.
Time to break out a more powerful sword (`pdb`) which also did not divulge any secrets about what was wrong.

Our hero went back to the `strace` output and after a fair bit of thought and analysis a pattern started to emerge. Mypy was looking in the wrong place (i.e. not in in the environment created by `python.withPackages` but in the interpreter store path) and our princess was in another castle!

Our hero went to the pub full of old grumpy men giving out the inner workings of the open source universe (Github) and acquired a copy of Mypy.
He littered the code with print statements & break points.
After a fierce battle full of blood, sweat & tears he ended up in 20f7f2dd71/mypy/sitepkgs.py and realised that everything came down to the Python `site` module and more specifically https://docs.python.org/3.7/library/site.html#site.getsitepackages which in turn relies on https://docs.python.org/3.7/library/site.html#site.PREFIXES .

Our hero created a copy of the environment created by `python.withPackages` and manually modified it to confirm his findings, and it turned out it was indeed the case.
Our hero had damaged the dragon and it was time for a celebration.

He went out and acquired some mead which he ingested while he typed up his story and waited for the dragon to finally die (the commit caused a mass-rebuild, I had to wait for my repro).

In the end all was good in [NixOps-AWS](https://github.com/nixos/nixops-aws)-town and type checks could run. (PR for that incoming tomorrow).
2020-03-14 21:39:31 +00:00
Frederik Rietdijk 13e7a3e112 Python: introduce tests for interpreters
This adds tests to the passthru of all Python interpreters.
2020-03-14 15:05:37 +01:00
Frederik Rietdijk dbf125d286 Python: introduce tests for interpreters
This adds tests to the passthru of all Python interpreters.
2020-03-14 15:01:30 +01:00
Graham Christensen 39aac70d74 pythonMinimal: don't include site-customise
Experimenting with patching the site-customize file causes mass
rebuilds for no reason.
2020-03-14 15:02:51 +01:00
Frederik Rietdijk 31855d74a3 python3: 3.7.6 -> 3.7.7 2020-03-12 12:00:26 +01:00
Jonathan Ringer 3973a3c79c python: add pythonRemoveTestsDirHook
This will automatically remove a top-level "tests"
directory from being installed.
2020-03-03 20:01:00 +01:00
Jonathan Ringer fe4580359e python39: 3.9.0.a3 -> 3.9.0.a4 2020-03-03 07:33:39 +01:00
Jonathan Ringer 2a019cc48c python38: 3.8.1 -> 3.8.2 2020-03-03 07:33:39 +01:00
Frederik Rietdijk 79217339d2 Merge master into staging-next 2020-02-29 15:29:11 +01:00
Emily 6d3fc35620 pypy{,3}: use openssl_1_1
"We now support building PyPy with OpenSSL 1.1 in our built-in _ssl
module, as well as maintaining support for previous versions."
-- https://pypy.readthedocs.io/en/latest/release-pypy2.7-v5.6.0.html
2020-02-28 16:06:20 +00:00
Jonathan Ringer cd97c055a0 python: execute pythonImportsCheckPhase in subshell
Execute in subshell so that the cwd returns back
to the original directory. This is important as
it enables pythonImportsCheck to work with checkPhase
2020-02-19 22:30:50 -08:00
Frederik Rietdijk 38cf6eac19 Merge master into staging-next 2020-02-06 19:43:36 +01:00
Frederik Rietdijk 9827e4994b python39: 3.9.0a2 -> 3.9.0a3 2020-02-06 10:25:18 +01:00
Maximilian Bosch 6b0cd9ad47
Merge branch 'staging' into glibc230
Conflicts:
	pkgs/applications/misc/vit/default.nix
2020-01-28 14:54:51 +01:00
Frederik Rietdijk dce0ca29d9 Merge master into staging-next 2020-01-28 10:46:13 +01:00
Maximilian Bosch eddfcc32b4
Merge branch 'staging' into glibc230 2020-01-23 11:31:13 +01:00
Frederik Rietdijk bc18cc72dd
Merge pull request #77610 from LnL7/darwin-stdenv-python3
darwin-stdenv: bootstrap with python3
2020-01-15 09:24:57 +01:00
Frederik Rietdijk eba1f79418 pythonPackages.venvShellHook: init
This is a hook that loads a virtualenv from the specified `venvDir`
location. If the virtualenv does not exist, it is created.
2020-01-14 22:36:21 +01:00
Luka Blaskovic 76d90cf1f5 cpython: fetch darwin-libutil.patch to nixpkgs
python3 is now required buildInput for glibc>=2.29.
Remove fetchpatch to solve infinite recursion in
glibc bootstrap process.
2020-01-14 08:26:58 +00:00
Daiderd Jordan 6328518e98
stdenv: bootstrap darwin with python3
- Replaced python override from the final stdenv, instead we
  propagate our bootstrap python to stage4 and override both
  CF and xnu to use it.

- Removed CF argument from python interpreters, this is redundant
  since it's not overidden anymore.

- Inherit CF from stage4, making it the same as the stdenv.
2020-01-13 11:34:36 +01:00
adisbladis 2d6f1ff4dd
python: Add support for installing Python eggs 2020-01-08 13:59:04 +00:00
Jan Tojnar f4e74edd8c python.pkgs.wrapPython: get rid of warning
When `makeWrapperArgs` variable is not set, `declare -p makeWrapperArgs`
will return with 1 and print an error message to stderr.

I did not handle the non-existence case in b0633406cb
because I thought `mk-python-derivation` will always define `makeWrapperArgs`
but `wrapProgram` can be called independently. And even with `mk-python-derivation`,
`makeWrappers` will not be set unless explicitly declared in the derivation
because of https://github.com/NixOS/nix/issues/1461.

I was lead to believe that because the builds were succeeding and I confirmed
that the mechanism fails when the variable is not defined and `-o nounset` is enabled.
It appears that `wrapPython` setup hook is not running under `-o nounset`, though,
invaldating the assumption.

Now we are checking that the variable exists before checking its type, which
will get rid of the warning and also prevent future error when `-o nounset`
is enabled in the setup hook.

For more information, see the discussion at
https://github.com/NixOS/nixpkgs/commit/a6bb2ede232940a96150da7207a3ecd15eb6328
2019-12-30 16:46:55 +01:00
Jan Tojnar a6bb2ede23
python.pkgs.wrapPython: fix string makeWrapperArgs
Bash takes an assignment of a string to an array variable:

local -a user_args
user_args="(foo bar)"

to mean appending the string to the array, not parsing the string into
an array as is the case when on the same line as the declaration:

local -a user_args="(foo bar)"

b0633406cb extracted the declaration before
the newly branched code block, causing string makeWrapperArgs being added
to the array verbatim.

Since local is function scoped, it does not matter if we move it inside
each of the branches so we fix it this way.
2019-12-28 22:23:36 +01:00
worldofpeace 4a2621da53
Merge pull request #76283 from jtojnar/python-mwa
python.pkgs.wrapPython: fix makeWrapperArgs
2019-12-26 19:21:37 -05:00
Jan Tojnar 4bbc6cc66f
Merge branch 'staging-next' into staging 2019-12-25 05:18:52 +01:00
Jan Tojnar ca39dd3a8a
Merge branch 'master' into staging-next 2019-12-25 05:15:06 +01:00
Jan Tojnar b0633406cb
python.pkgs.wrapPython: fix makeWrapperArgs
When `makeWrapperArgs` is a Bash array, we only passed the first
item to `wrapProgram`. We need to use `"${makeWrapperArgs[@]}"`
to extract all the items. But that breaks the common string case so
we need to handle that case separately.
2019-12-23 18:02:44 +01:00
Frederik Rietdijk ad733b5505 python37: 3.7.5 -> 3.7.6 2019-12-19 17:37:02 +01:00
Frederik Rietdijk c0c65fe83c python39: 3.9.0a1 -> 3.9.0a2 2019-12-19 17:36:21 +01:00
Frederik Rietdijk 2012dd5734 python38: 3.8.0 -> 3.8.1 2019-12-19 17:36:21 +01:00
Frederik Rietdijk 5796029c5d python36: 3.6.9 -> 3.6.10 2019-12-19 17:36:20 +01:00
Andreas Rammhold e9f522eee1
python: remove _manylinux.py
This will turn manylinux support back on by default.

PIP will now do runtime checks against the compatible glibc version to
determine if the current interpreter is compatible with a given
manylinux specification. However it will not check if any of the
required libraries are present.

The motivation here is that we want to support building python packages
with wheels that require manylinux support. There is no real change for
users of source builds as they are still buildings packages from source.

The real noticeable(?) change is that impure usages (e.g. running `pip
install package`) will install manylinux packages that previously
refused to install.
Previously we did claim that we were not compatible with manylinux and
thus they wouldn't be installed at all.

Now impure users will have basically the same situation as before: If
you require some wheel only package it didn't work before and will not
properly work now. Now the program will fail during runtime vs during
installation time.

I think it is a reasonable trade-off since it allows us to install
manylinux packages with nix expressions and enables tools like
poetry2nix.

This should be a net win for users as it allows wheels, that we
previously couldn't really support, to be used.
2019-12-16 16:37:16 +01:00
Tobias Pflug 418ad571c3 Fix manylinux packages
Make sure lib outputs are used where applicable.
2019-12-13 11:40:44 +01:00
Frederik Rietdijk 6530535b20
manylinux packages for Python
This adds three lists with manylinux dependencies as well as three
packages that include all the manylinux dependencies.
2019-12-05 09:56:20 +00:00
Frederik Rietdijk 92d2153e6c pythonInterpreters: remove unnecessary rec 2019-12-02 20:10:51 +01:00
Frederik Rietdijk 1d18c5a0fe Merge staging-next into staging 2019-11-24 10:13:31 +01:00
Frederik Rietdijk 182571cdc3 update-python-libraries: ignore pyproject 2019-11-22 08:37:03 +01:00
Frederik Rietdijk 1939a97811 python3: add pythonForBuild as parameter, fixes python3Minimal
`pythonForBuild` exists for cross-compilation. When one overrides
python, one needs to ensure pythonForBuild matches.
2019-11-21 22:00:23 +01:00
Frederik Rietdijk ad3ef645b0 python3Minimal: 3.7.4 -> 3.7.5
Base it on python37 so we ensure the package remains up to date.
2019-11-21 15:34:37 +01:00
Frederik Rietdijk 65edeb8633 Merge master into staging-next 2019-11-20 10:01:49 +01:00
Frederik Rietdijk 648152fdbb python39: init at 3.9.0a1
It's a year until the final release but this will give a chance to test
out certain features and how it integrates with other packages.
https://www.python.org/dev/peps/pep-0596/
2019-11-20 09:42:27 +01:00
Jan Tojnar ae465621ff
pythonPackages.pipBuildHook: fix unbound variable
for compatibility with set -u
2019-11-20 05:06:51 +01:00
Frederik Rietdijk c4e30cf98c Merge staging-next into staging 2019-11-05 14:18:08 +01:00
Frederik Rietdijk 03a9822405 Merge master into staging-next 2019-11-05 14:17:37 +01:00
John Ericson acd2d19484
Merge pull request #72347 from NixOS/bash-no-undef-vars
treewide: `set -u` everywhere
2019-11-04 19:52:33 -05:00
Jonathan Ringer deb201b311 update-python-libraries: update usage comments 2019-11-03 12:03:27 -08:00
Frederik Rietdijk 9d59d57d45 Merge staging-next into staging 2019-11-03 14:01:28 +01:00
Frederik Rietdijk 7827d3f449 python35: 3.5.8 -> 3.5.9
There were no new changes in version 3.5.9; 3.5.9 was released only because of a CDN caching problem,
which resulted in some users downloading a prerelease version of the 3.5.8 .xz source tarball.
Apart from the version number, 3.5.9 is identical to the proper 3.5.8 release.
2019-11-03 11:21:05 +01:00
Dmitry Kalinkin 3466faf9d8 pythonPackages.setuptoolsBuildHook: fix debug message 2019-11-03 10:34:44 +01:00
John Ericson b7f4bda282 treewide: *Phase(s)? variables are optional
If these aren't defined, the stdenv defaults are used in the `*Phase`
case, or no extra phases are done, in the `*Phases` case.
2019-11-01 14:44:44 -04:00
John Ericson 2811b032d6 treewide: Make still dont* Variables are optional in most cases
Go beyond the obvious setup hooks now, with a bit of sed, with a skipped case:

 - cc-wrapper's `dontlink`, because it already is handled.

Also, in nix files escaping was manually added.

EMP
2019-11-01 14:44:44 -04:00
Frederik Rietdijk ccf514a61d python35: 3.5.7 -> 3.5.8 2019-10-29 10:39:50 +01:00
Mario Rodas 0f8dc3cc2e
pypy: fix build on darwin 2019-10-26 04:20:00 -05:00
Frederik Rietdijk 43bbecea5e pythonInterpreters.graalpython37: init
Add an interpreter with package set for graalpython 3.7.
2019-10-22 09:16:18 +02:00
Frederik Rietdijk 5b55013aa2 python2: 2.7.16 -> 2.7.17
Co-authored-by: Dmitry Kalinkin <dmitry.kalinkin@gmail.com>
2019-10-20 19:48:32 +02:00
Frederik Rietdijk aedf4fc8fc Revert "update-python-libraries: use version key for latest release (#68857)"
This broke support for --target.

This reverts commit e8c29fa77c.
2019-10-18 09:53:55 +02:00
Frederik Rietdijk 7181aca6d9 update-python-libraries: handle other format 2019-10-18 09:53:55 +02:00
Frederik Rietdijk 12441bdf9f Merge staging-next into staging 2019-10-16 11:17:12 +02:00
Frederik Rietdijk 41abe12bb3 python37: 3.7.4 -> 3.7.5 2019-10-16 09:14:48 +02:00
Matthew Glazar 81d15948cc python38: fix build on macOS
Python 3.8 fails to build on macOS for two reasons:

* python-3.x-distutils-C++.patch fails to apply cleanly.
* An #include for <util.h> is missing, causing a build failure:

    ./Modules/posixmodule.c:6586:9: error: implicit declaration of function 'openpty' is invalid in C99
        if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) != 0)
            ^

Use the correct version of python-3.x-distutils-C++.patch, and add a
patch to #include <util.h>.
2019-10-15 13:12:13 +02:00
Frederik Rietdijk 911b053802 python38: 3.8.0rc1 -> 3.8.0 2019-10-15 11:56:01 +02:00
Jan Tojnar 6c8aed6391
Merge branch 'master' into staging-next 2019-10-12 00:50:21 +02:00
Frederik Rietdijk 1b8f8a65ba python38: 3.8.0b3 -> 3.8.0rc1 2019-10-11 13:57:18 +02:00
Frederik Rietdijk 503081fa5b Merge staging into staging-next 2019-09-29 11:05:22 +02:00
Jon 28af6ac647 python3Packages.flit: fix tests and packaging (#69546)
* python3Packages.flit: fix tests

* python: fix flit setup hook
2019-09-26 21:09:03 +00:00
adisbladis 2b2c6e0bc4
pythonPackages.pipBuildHook: Add wheel dependency
And remove missing "options"
2019-09-26 17:17:16 +01:00
Sebastian Jordan 5505d2f036 python: Fix invalid pip call in setuptoolsShellHook 2019-09-22 09:55:13 +02:00
Mario Rodas e8c29fa77c
update-python-libraries: use version key for latest release (#68857)
From warehouse API reference [1]:

     GET /pypi/<project_name>/json

     Returns metadata (info) about an individual project at the latest
     version […]

[1] https://warehouse.pypa.io/api-reference/json/#project
2019-09-16 17:34:15 -05:00
Frederik Rietdijk c99529a4b6 python.pkgs.wheelUnpackHook: propagate wheel
This was accidentally removed when buildPython* was rewritten as hooks.
2019-09-10 22:26:20 +02:00
volth 7b8fb5c06c treewide: remove redundant quotes 2019-09-08 23:38:31 +00:00
Frederik Rietdijk 9894a70299 Merge staging into staging-next 2019-09-06 22:47:40 +02:00
Frederik Rietdijk 66bc7fc1b3 Merge master into staging-next 2019-09-06 22:46:05 +02:00
Frederik Rietdijk b75f9a5f6d pythonInterpreters.pypy*prebuilt: use openssl 1.0 2019-09-06 21:45:10 +02:00
Frederik Rietdijk f7e28bf5d8 Split buildPythonPackage into setup hooks
This commit splits the `buildPythonPackage` into multiple setup hooks.

Generally, Python packages are built from source to wheels using `setuptools`.
The wheels are then installed with `pip`. Tests were often called with
`python setup.py test` but this is less common nowadays. Most projects
now use a different entry point for running tests, typically `pytest`
or `nosetests`.

Since the wheel format was introduced more tools were built to generate these,
e.g. `flit`. Since PEP 517 is provisionally accepted, defining a build-system
independent format (`pyproject.toml`), `pip` can now use that format to
execute the correct build-system.

In the past I've added support for PEP 517 (`pyproject`) to the Python
builder, resulting in a now rather large builder. Furthermore, it was not possible
to reuse components elsewhere. Therefore, the builder is now split into multiple
setup hooks.

The `setuptoolsCheckHook` is included now by default but in time it should
be removed from `buildPythonPackage` to make it easier to use another hook
(curently one has to pass in `dontUseSetuptoolsCheck`).
2019-09-06 15:18:45 +02:00
Matthew Bauer 98662eba0e python: enable loadable sqlite extensions (#67472)
Our sqlite supports it, so might as well make use of it.

Fixes #66526
2019-08-31 07:59:00 -04:00
volth 08f68313a4 treewide: remove redundant rec 2019-08-28 11:07:32 +00:00
Frederik Rietdijk 98640fd482 Merge master into staging-next 2019-08-27 16:36:47 +02:00
Marek Mahut ee62ec62ec python2: CVE-2018-20852
Fixes #67200

(cherry picked from commit 302cac35f5)
2019-08-27 10:04:55 +02:00
Matthew Bauer 48d0e9401e cpython: skip macOS system frameworks
We don’t want cpython picking up /Library/Frameworks and
/System/Library/Frameworks which contains Tcl.framework. Instead it
should use the one provided by Nix. this would not be an issue if
sandboxing was enabled, but unfortunately that has its own issues.

Fixes #66647
2019-08-25 20:35:19 -04:00
Vladimír Čunát 2e6bf42a22
Merge branch 'master' into staging-next
There ver very many conflicts, basically all due to
name -> pname+version.  Fortunately, almost everything was auto-resolved
by kdiff3, and for now I just fixed up a couple evaluation problems,
as verified by the tarball job.  There might be some fallback to these
conflicts, but I believe it should be minimal.

Hydra nixpkgs: ?compare=1538299
2019-08-24 08:55:37 +02:00
Robin Gloster c7e75df21f
Merge remote-tracking branch 'upstream/master' into openssl-1.1 2019-08-23 02:00:31 +02:00
adisbladis 99e6ca2bdc python3Minimal: Add top-level for a minimal Python 3 build
This builds Python without optional dependencies.

We can't just use python3.override, as things like
python3Minimal.withPackages would pass the wrong python derivation into
these modules.
2019-08-21 14:52:49 +02:00
Robin Gloster f4fc845e5b
Merge remote-tracking branch 'upstream/master' into openssl-1.1 2019-08-21 14:25:13 +02:00
Florian Klink 100eae3b77 python: allow stripping tkinter 2019-08-19 00:11:17 +02:00
Florian Klink db087f226d python: allow stripping bytecode 2019-08-18 23:47:44 +02:00
Florian Klink 52fcbbff44 python: make rebuilding bytecode optional 2019-08-18 23:47:44 +02:00
Florian Klink 8335f3dd5c python: allow stripping config 2019-08-18 23:47:43 +02:00
Florian Klink d03283c029 python: allow stripping tests 2019-08-18 23:47:30 +02:00
Florian Klink 414cde4df4 python: allow stripping IDLE 2019-08-18 23:47:16 +02:00
Frederik Rietdijk f65aa21bb2 Merge master into staging-next 2019-08-18 12:53:44 +02:00
adisbladis 5c0d493df4
python: Enable building without openssl 2019-08-18 08:18:58 +01:00
volth 46420bbaa3 treewide: name -> pname (easy cases) (#66585)
treewide replacement of

stdenv.mkDerivation rec {
  name = "*-${version}";
  version = "*";

to pname
2019-08-15 13:41:18 +01:00
Emily ec07b882b3 pypy{2,27,3,35 -> 36}: 7.0.0 -> 7.1.1 2019-08-13 16:49:27 -07:00
Emily 5638dd55e3 pythonInterpreters.pypy{27,35 -> 36}_prebuilt: 6.0.0 -> 7.1.1 2019-08-13 16:49:27 -07:00
Frederik Rietdijk 658c0f0ce9 python38: 3.8.0b2 -> 3.8.0b3 2019-08-03 08:14:25 +02:00
Robin Gloster 30969073f0
Merge remote-tracking branch 'upstream/master' into openssl-1.1 2019-08-02 03:01:30 +02:00
Michael Weiss 8760c695d4
swiften, pypy: Mark as broken
swiften is broken since 2018-10-18:
x86_64-linux:  https://hydra.nixos.org/build/97180305
aarch64-linux: https://hydra.nixos.org/build/97193043

pypy is broken since 2019-05-29:
x86_64-linux (pypy):  https://hydra.nixos.org/build/97214777
x86_64-linux (pypy3): https://hydra.nixos.org/build/97221287

The builds for swiften and reposurgeon (depends on pypy) regularly fail
during rebuilds for SCons updates (e.g. [0]).

[0]: https://github.com/NixOS/nixpkgs/pull/65392
2019-07-30 19:25:43 +02:00
Andreas Rammhold d80649d2a3
pypy: use openssl 1.0.2
While upstream claims support of OpenSSL 1.1 the test suite fails.
2019-07-29 21:25:13 +02:00
Frederik Rietdijk bd47c5721f Python: introduce NIX_PYTHONEXECUTABLE in order to set sys.executable
This is needed in case of `python.buildEnv` to make sure sys.executable
does not point to the unwrapped executable.
2019-07-27 11:52:49 +02:00
Nikolay Amiantov da295a1206 python2: backport fix for pyc race condition, part 2
Turns out fixing this only in importlib is not sufficient and we
need to backport CPython part of the fix too.

This patch is based on https://hg.python.org/cpython/rev/c16063765d3a
but because the code around is different there are some changes (C-strings
instead of Python objects etc.)

With this patch Tensorflow builds successfully on many-core machine.
2019-07-17 10:22:11 +02:00
Frederik Rietdijk 0d832585b7 Merge staging-next into staging 2019-07-17 10:14:29 +02:00
Frederik Rietdijk 7da15d9b36 buildPythonPackage: add support for setupPyGlobalFlags (2) 2019-07-15 17:47:57 +02:00
Mario Rodas 7094a1af0c
python37: fix distutils C++ patch (PR #64758)
Update distutils patch to the upstream changes.
See: 6c0e0d141a
2019-07-15 09:37:43 +02:00
Frederik Rietdijk 271b57c219 Revert "buildPythonPackage: add support for setupPyDistFlags"
I merged this a bit too quick. We need to have a closer look at it.
See the points brought up in https://github.com/NixOS/nixpkgs/pull/64682.

This reverts commit 1e0ebdb8a4.
2019-07-13 13:25:32 +02:00
Kirill Boltaev 1e0ebdb8a4 buildPythonPackage: add support for setupPyDistFlags
Flags passed to the "python setup.py" command.
2019-07-13 09:47:00 +02:00
Frederik Rietdijk 4aee94629d Merge staging-next into staging 2019-07-13 09:46:03 +02:00
Frederik Rietdijk 54065ae20d Merge master into staging-next 2019-07-13 09:45:40 +02:00
Frederik Rietdijk ec7f569211 python.buildEnv: use NIX_PYTHONPATH 2019-07-13 09:37:33 +02:00
Frederik Rietdijk 46409b5c32 Python: add sitecustomize.py, listen to NIX_PYTHONPATH
This commit adds a Nix-specific module that recursively adds paths that
are on `NIX_PYTHONPATH` to `sys.path`. In order to process possible
`.pth` files `site.addsitedir` is used.

The paths listed in `PYTHONPATH` are added to `sys.path` afterwards, but
they will be added before the entries we add here and thus take
precedence.

The reason for adding support for this environment variable is that we
can set it in a wrapper without breaking support for `PYTHONPATH`.
2019-07-13 09:37:33 +02:00
Frederik Rietdijk 8074133844 pythonPackages.fetchPypi: support SRI hashes 2019-07-11 18:01:04 +02:00
Frederik Rietdijk 30f299027a pythonPackages.fetchPypi: separate url computation from fetchurl 2019-07-11 18:01:04 +02:00
Frederik Rietdijk f2a8117492 pythonPackages.fetchPypi: put expression in own file
Also fixes an evaluation bug.
2019-07-11 18:01:04 +02:00
Charlie Dyson 0426ae63f8 fix: #64340 building python wheels in non-standard store 2019-07-11 10:27:28 +02:00
Frederik Rietdijk 79a03641d5 Merge staging-next into staging 2019-07-09 15:46:26 +02:00
Frederik Rietdijk 74c24385cb Merge master into staging-next 2019-07-09 15:46:00 +02:00
Frederik Rietdijk 421a87d94c python38: init at 3.8.0b2 2019-07-09 15:31:37 +02:00
Frederik Rietdijk 46d91410af python37: 3.7.3 -> 3.7.4 2019-07-09 10:04:25 +02:00
Frederik Rietdijk a88e319591 python36: 3.6.8 -> 3.6.9 2019-07-09 10:04:25 +02:00
Timo Kaufmann 9db3a5869e python2: backport fix for pyc race condition
This is python bug https://bugs.python.org/issue13146. Fixed since
python 3.4. It makes pyc creation atomic, preventing a race condition.
The patch has been rebased on our deterministic build patch.

It wasn't backported to python 2.7 because there was a complaint about
changed semantics. Since files are now created in a temporary directory
and then moved, symlinks will be overridden. See
https://bugs.python.org/issue17222.

That is an edge-case however. Ubuntu and debian have backported the fix
in 2013 already, making it mainstream enough for us to adopt.
2019-07-03 08:40:51 +02:00
volth f3282c8d1e treewide: remove unused variables (#63177)
* treewide: remove unused variables

* making ofborg happy
2019-06-16 19:59:05 +00:00
Matthew Bauer 9f7bb1f512 python27: add override to build statically 2019-06-03 12:28:25 -04:00
Frederik Rietdijk d801204fdd
Merge pull request #51641 from codedownio/nousersite-master
Allow skipping the use of the PYTHONNOUSERSITE variable
2019-05-13 12:39:25 +02:00
Tom McLaughlin a3f24daa7b Add flag to disable PYTHONNOUSERSITE for wrapped binaries in python environments 2019-05-13 02:56:56 -07:00
Tom McLaughlin 0045164b15 Add flag to disable PYTHONNOUSERSITE for wrapped python binaries 2019-05-13 02:56:32 -07:00
Frederik Rietdijk 3462fe27d9 Merge staging-next into staging 2019-04-30 15:52:19 +02:00
Ihar Hrachyshka 43f9d5ccaa Generate python3 docs 2019-04-28 09:24:01 +02:00
Daiderd Jordan a76e11a8c5 python: make it possible to disable the python setup-hook
Without this it's impossible to use a python binary that depends on a
different python version than what's used during the current build.
2019-04-28 09:20:41 +02:00
Dmitry Kalinkin 8fa36fc8a1 python: provide hasCxxDistutils attribute for pythonPackages.numpy
Patching numpy.distutils used to be required for pythonPackages.cython
to build on darwin. It was later accidentally disabled during one of the
refactorings, but that did not break cython. This change reinstantiates
the patch. It still applies, so it should be low maintenance and it can
still be useful.
2019-04-28 09:17:59 +02:00
Jan Tojnar 83914f2c56
buildPython*: allow overriding strictDeps 2019-04-24 22:27:06 +02:00
Frederik Rietdijk 9d87ccabce Merge master into staging-next 2019-04-18 08:25:25 +02:00
bb010g 28ac309309 python3{5,6}: Reintroduce bytecode determinism
Originally introduced in 8970a9c and 1531b5e, these patches were lost in
efbe87f when the CPython version files were merged, likely due to
CPython 3.7 not needing them anymore. These patches should remain
in-tree until CPython 3.5 and 3.6 support is dropped completely.

Found with the diffoscope ( https://diffoscope.org/ ).
2019-04-18 08:12:29 +02:00
Frederik Rietdijk bae32a9f5c Merge staging-next into staging 2019-04-16 18:54:15 +02:00
Matthew Glazar fdbcf5ba36 python37: fix build on macOS
The upstream patch for distutils does not apply cleanly to Python
3.7.3's sources. (The patch applies cleanly to Python 3.7.2's sources,
but nixpkgs commit 0ddae82e6a upgraded
Python to 3.7.3.) Fix the patch to make python37 build on macOS.
2019-04-15 10:49:34 -07:00
Corbin edd5555029 pypy: Disable broken test on 3.x. 2019-04-09 20:41:08 +02:00
Corbin 9c04b75cf9 pypy: 6.0.0 -> 7.0.0
Note that this bumps both the 2.7 and 3.5 flavors of PyPy.
2019-04-09 20:41:08 +02:00
Corbin 5719f892e1 python: Make .isPyPy flag more accurate.
nix-repl> map (s: s.isPyPy) [ python python3 pypy pypy3 ]
[ false false true true ]
2019-04-09 20:41:08 +02:00
Frederik Rietdijk 0ddae82e6a python3: 3.7.2 -> 3.7.3 2019-03-26 16:24:04 +01:00
Frederik Rietdijk 23e431387b Merge staging-next into staging 2019-03-23 09:20:09 +01:00
Frederik Rietdijk 2cc6444113 python35 -> 3.5.6 -> 3.5.7 2019-03-23 09:17:35 +01:00
Wael M. Nasreddine 5af0780492
Merge remote-tracking branch 'origin/master' into staging
* origin/master: (693 commits)
  buildGoModule: use go_1_12 instead of go_1_11 (#58103)
  gitAndTools.lab: 0.15.2 -> 0.15.3 (#58091)
  signal-desktop: 1.22.0 -> 1.23.0
  added missing semicolon to documentation
  terminus_font_ttf: 4.46.0 -> 4.47.0
  buildGoModule: remove SSL env vars in favor of cacert in buildInputs (#58071)
  dav1d: init at 0.2.1
  dropbox-cli: 2018.11.28 -> 2019.02.14
  atlassian-confluence: 6.14.1 -> 6.14.2
  maintainers: update email for dywedir
  python.pkgs.hglib: use patch to specify hg path (#57926)
  chkrootkit: 0.52 -> 0.53
  radare2-cutter: 1.7.2 -> 1.8.0
  autorandr: 1.7 -> 1.8
  pythonPackages.pyhepmc: fix build
  llvm-polly/clang-polly: use latest llvm
  apulse: 0.1.11.1 -> 0.1.12, cleanup
  factorio: experimental 0.17.14 → 0.17.16 (#58000)
  sequeler: 0.6.7 -> 0.6.8
  nasc: 0.5.1 -> 0.5.2
  ...
2019-03-21 21:01:25 -07:00
Matthew Glazar 0bf2d63366 python35: fix build on macOS
The upstream patch for distutils does not apply cleanly to Python 3.5's
sources. Fix the patch to make python35 build on macOS.
2019-03-10 13:38:43 +01:00
Vladimír Čunát 8d502fd425
Merge branch 'staging-next' into staging 2019-03-10 08:05:27 +01:00
Frederik Rietdijk a2736d27d1 buildPythonPackage: fix building with flit 2019-03-06 13:42:46 +01:00
Frederik Rietdijk 347680df96 python: 2.7.15 -> 2.7.16 2019-03-05 08:30:04 +01:00
Frederik Rietdijk c2eac6741b Merge master into staging-next 2019-02-24 09:19:12 +01:00
Frederik Rietdijk 1fccd25595 buildPythonPackage: always export LANG=C.UTF-8 2019-02-23 20:08:26 +01:00
Frederik Rietdijk e7b4608d02 buildPythonPackage: initial support for PEP 517 2019-02-23 20:05:19 +01:00
Florian Friesdorf fbef5ab82f Remove myself as maintainer from packages
I'm currently not maintaining any packages.
2019-02-22 16:14:13 +01:00
Frederik Rietdijk b4acd97729 buildPython*: enable strictDeps 2019-02-17 14:40:48 +01:00
Averell Dalton 91aeda1432 python: sysconfig: fix paths 2019-02-07 18:52:08 +01:00
Frederik Rietdijk 9cfae76fc5 Merge staging-next into staging 2019-01-17 09:21:29 +01:00
Anders Kaseorg 004b908ae6 pypy, pypy3: Remove wrapper
The wrapper is not needed because the runpath is already set correctly,
and LD_LIBRARY_PATH was breaking child processes linked against
different libc versions.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-16 08:36:47 +01:00
Anders Kaseorg 6282071229 pypy: Correct Python version in meta description
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-16 08:15:50 +01:00
Frederik Rietdijk c5d99308de buildPython*: fix nativeBuildInputs 2019-01-15 10:16:11 +01:00
Lengyel Balazs f4a53ff3bc treewide/xorg: replace *proto with xorgproto 2019-01-04 14:38:57 +01:00
Frederik Rietdijk 63bcd07888 python: add isPy2 attribute to passthru
Some packages relied on it. For consistency, also introduce isPy3 (which
is the same as isPy3k).
2019-01-04 12:31:42 +01:00
Frederik Rietdijk d91b496eac pythonInterpreters.pypy{27,35}_prebuilt: init at 6.0.0
These interpreters are prebuilt by upstream and patched using patchelf.
They are primarily added for testing purposes and development on the
non-prebuilt PyPy interpreters as it can speed up translation
significantly.
2019-01-04 10:45:22 +01:00
Frederik Rietdijk 0a2caa41fe Python: drop python.majorVersion
Drop `python.majorVersion`. For Python language version, use `python.pythonVersion`.
For implementation version, use `python.sourceVersion`.

Some expressions were broken. Those that were identified were fixed.

fixup major
2019-01-04 10:45:22 +01:00
Frederik Rietdijk f665828fa3 Python: improve cross-compilation
This changeset allows for cross-compilation of Python packages. Packages
built with buildPythonPackage are not allowed to refer to the build
machine. Executables that have shebangs will refer to the host.
2019-01-04 10:45:22 +01:00
Frederik Rietdijk 613498af97 pypy: merge 2.7 and 3.5 into a single expression
This commit merges the two expressions in a single one, using
the passthru function that is shared with CPython.
2019-01-04 10:45:22 +01:00
Frederik Rietdijk efbe87f3ef CPython: merge expressions of interpreters
Each time a new major/minor version of CPython was released, a new
expression would be written, typically copied from the previous release.
Often fixes are only made in the current/latest release. By merging the
expressions it's more likely that modifications end up in all versions,
as is likely intended.

This commit introduces one expression for Python 3, and another for 2.7.
These two may also be merged, but it will result in a lot of extra
conditionals making the expression harder to follow.

A common passthru is introduced for CPython and PyPy.

python 2.7: use common passthru
2019-01-04 10:45:22 +01:00
Franz Pletz 47587f5d31
python37: fix cross build 2019-01-01 23:00:56 +01:00
Frederik Rietdijk 10afccf145 Merge staging-next into staging 2018-12-27 18:11:34 +01:00
Mario Rodas 1764870813 python37: fix darwin build
Remove issue24658 patch because is already fixed and backported to
Python 3.7 178d1c0777
2018-12-27 09:27:59 +01:00
Mario Rodas 31680c54e1 python36: fix darwin build
Remove issue24658 patch because is already fixed and backported to
Python 3.6 a5ebc205be
2018-12-27 09:27:59 +01:00
Frederik Rietdijk 2526f9f157 python37: remove references to openssl.dev 2018-12-26 10:37:55 +01:00
Frederik Rietdijk e45ca47f14 Merge staging-next into staging 2018-12-26 09:30:32 +01:00
Frederik Rietdijk 552d8a70d4 python37: 3.7.1 -> 3.7.2 2018-12-25 10:05:49 +01:00
Frederik Rietdijk 741e00fd10 python36: 3.6.7 -> 3.6.8 2018-12-25 10:05:38 +01:00
Jan Tojnar ef935fa101
Merge branch 'master' into staging 2018-12-24 15:02:29 +01:00
Frederik Rietdijk 3d5025ba79 buildPython*: don't override custom updateScript 2018-12-23 09:59:56 +01:00
Jörg Thalheim 1b146a8c6f
treewide: remove paxutils from stdenv
More then one year ago we removed grsecurity kernels from nixpkgs:
https://github.com/NixOS/nixpkgs/pull/25277

This removes now also paxutils from stdenv.
2018-12-22 12:55:05 +01:00
Jörg Thalheim 855777ad22 wrap-python: fix pypy site-packages path 2018-12-20 14:18:25 +00:00
Jörg Thalheim cc30e48624
Merge pull request #52481 from Mic92/python37-cross
python37: fix cross-compilation
2018-12-20 15:06:22 +01:00
Jan Tojnar e69ee9b53e
buildPython*: simplify updateScript
updateScript accepts a list, in which case, it will execute the head with the
tail as arguments. Switching to this style will allow us to get rid of the need
to create extra script doing just that.
2018-12-20 00:28:49 +01:00
Anders Kaseorg 85379a29ec pypy3: init at 6.0.0
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-19 09:53:31 +01:00
Jörg Thalheim ab9c602fd4
Merge pull request #40459 from andersk/pypy
pypy: 5.10.0 -> 6.0.0
2018-12-18 20:35:38 +00:00
Jörg Thalheim 14b2dcea8f python37: fix cross-compilation 2018-12-18 12:39:24 +00:00
Frederik Rietdijk 74d0b82f29 buildPython*: add updateScript to passthru
All Python packages now have an updateScript. The script calls
`update-python-libraries` and passes it the position of the derivation
expression obtained using `meta.position`. This works fine in case a Nix
expression represents only a single derivation. If there are more in it,
`update-python-libraries` will fail.
2018-12-07 15:48:54 +01:00
Anders Kaseorg 6d80927a43 pypy: Set isPy2, isPy27
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-20 01:59:08 -08:00
Anders Kaseorg 8b80dec522 pypy: Reenable tests fixed upstream
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-20 01:59:08 -08:00
Anders Kaseorg 91333c1740 pypy: Remove redundant --withmod-* flags
--allworkingmodules is already the default.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-20 01:59:08 -08:00