diff --git a/doc/default.nix b/doc/default.nix index 196b9e445390..7583813c94f1 100644 --- a/doc/default.nix +++ b/doc/default.nix @@ -45,6 +45,10 @@ stdenv.mkDerivation { + toDocbook { inputFile = ./introduction.md; outputFile = "introduction.xml"; + } + + toDocbook { + inputFile = ./languages-frameworks/python.md; + outputFile = "./languages-frameworks/python.xml"; useChapters = true; } + toDocbook { diff --git a/doc/languages-frameworks/python.md b/doc/languages-frameworks/python.md new file mode 100644 index 000000000000..fc0a0ba987ae --- /dev/null +++ b/doc/languages-frameworks/python.md @@ -0,0 +1,714 @@ +# Python + +## User Guide + +Several versions of Python are available on Nix as well as a high amount of +packages. The default interpreter is CPython 2.7. + +### Using Python + +#### Installing Python and packages + +It is important to make a distinction between Python packages that are +used as libraries, and applications that are written in Python. + +Applications on Nix are installed typically into your user +profile imperatively using `nix-env -i`, and on NixOS declaratively by adding the +package name to `environment.systemPackages` in `/etc/nixos/configuration.nix`. +Dependencies such as libraries are automatically installed and should not be +installed explicitly. + +The same goes for Python applications and libraries. Python applications can be +installed in your profile, but Python libraries you would like to use to develop +cannot. If you do install libraries in your profile, then you will end up with +import errors. + +#### Python environments using `nix-shell` + +The recommended method for creating Python environments for development is with +`nix-shell`. Executing + +```sh +$ nix-shell -p python35Packages.numpy python35Packages.toolz +``` + +opens a Nix shell which has available the requested packages and dependencies. +Now you can launch the Python interpreter (which is itself a dependency) + +```sh +[nix-shell:~] python3 +``` + +If the packages were not available yet in the Nix store, Nix would download or +build them automatically. A convenient option with `nix-shell` is the `--run` +option, with which you can execute a command in the `nix-shell`. Let's say we +want the above environment and directly run the Python interpreter + +```sh +$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3" +``` + +This way you can use the `--run` option also to directly run a script + +```sh +$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3 myscript.py" +``` + +In fact, for this specific use case there is a more convenient method. You can +add a [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) to your script +specifying which dependencies Nix shell needs. With the following shebang, you +can use `nix-shell myscript.py` and it will make available all dependencies and +run the script in the `python3` shell. + +```py +#! /usr/bin/env nix-shell +#! nix-shell -i python3 -p python3Packages.numpy + +import numpy + +print(numpy.__version__) +``` + +Likely you do not want to type your dependencies each and every time. What you +can do is write a simple Nix expression which sets up an environment for you, +requiring you only to type `nix-shell`. Say we want to have Python 3.5, `numpy` +and `toolz`, like before, in an environment. With a `shell.nix` file +containing + +```nix +with import {}; + +(pkgs.python35.buildEnv.override { + extraLibs = with pkgs.python35Packages; [ numpy toolz ]; +}).env +``` +executing `nix-shell` gives you again a Nix shell from which you can run Python. + +What's happening here? + +1. We begin with importing the Nix Packages collections. `import ` import the `` function, `{}` calls it and the `with` statement brings all attributes of `nixpkgs` in the local scope. Therefore we can now use `pkgs`. +2. Then we create a Python 3.5 environment with `pkgs.buildEnv`. Because we want to use it with a custom set of Python packages, we override it. +3. The `extraLibs` argument of the original `buildEnv` function can be used to specify which packages should be included. We want `numpy` and `toolz`. Again, we use the `with` statement to bring a set of attributes into the local scope. +4. And finally, for in interactive use we return the environment. + +### Developing with Python + + +Now that you know how to get a working Python environment on Nix, it is time to go forward and start actually developing with Python. +We will first have a look at how Python packages are packaged on Nix. Then, we will look how you can use development mode with your code. + +#### Python packaging on Nix + +On Nix all packages are built by functions. The main function in Nix for building Python packages is [`buildPythonPackage`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/generic/default.nix). +Let's see how we would build the `toolz` package. According to [`python-packages.nix`](https://raw.githubusercontent.com/NixOS/nixpkgs/master/pkgs/top-level/python-packages.nix) `toolz` is build using + +```nix +toolz = buildPythonPackage rec{ + name = "toolz-${version}"; + version = "0.7.4"; + + src = pkgs.fetchurl{ + url = "https://pypi.python.org/packages/source/t/toolz/toolz-${version}.tar.gz"; + sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd"; + }; + + meta = { + homepage = "http://github.com/pytoolz/toolz/"; + description = "List processing tools and functional utilities"; + license = licenses.bsd3; + maintainers = with maintainers; [ fridh ]; + }; +}; +``` + +What happens here? The function `buildPythonPackage` is called and as argument +it accepts a set. In this case the set is a recursive set ([`rec`](http://nixos.org/nix/manual/#sec-constructs)). +One of the arguments is the name of the package, which consists of a basename +(generally following the name on PyPi) and a version. Another argument, `src` +specifies the source, which in this case is fetched from an url. `fetchurl` not +only downloads the target file, but also validates its hash. Furthermore, we +specify some (optional) [meta information](http://nixos.org/nixpkgs/manual/#chap-meta). + +The output of the function is a derivation, which is an attribute with the name +`toolz` of the set `pythonPackages`. Actually, sets are created for all interpreter versions, +so `python27Packages`, `python34Packages`, `python35Packages` and `pypyPackages`. + +The above example works when you're directly working on +`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though, +you will want to test a Nix expression outside of the Nixpkgs tree. If you +create a `shell.nix` file with the following contents + +```nix +with import {}; + +pkgs.python35Packages.buildPythonPackage rec { + name = "toolz-${version}"; + version = "0.7.4"; + + src = pkgs.fetchurl{ + url = "https://pypi.python.org/packages/source/t/toolz/toolz-${version}.tar.gz"; + sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd"; + }; + + meta = { + homepage = "http://github.com/pytoolz/toolz/"; + description = "List processing tools and functional utilities"; + license = licenses.bsd3; + maintainers = with maintainers; [ fridh ]; + }; +} +``` + +and then execute `nix-shell` will result in an environment in which you can use +Python 3.5 and the `toolz` package. As you can see we had to explicitly mention +for which Python version we want to build a package. + +The above example considered only a single package. Generally you will want to use multiple packages. +If we create a `shell.nix` file with the following contents + +```nix +with import {}; + +( let + toolz = pkgs.python35Packages.buildPythonPackage rec { + name = "toolz-${version}"; + version = "0.7.4"; + + src = pkgs.fetchurl{ + url = "https://pypi.python.org/packages/source/t/toolz/toolz-${version}.tar.gz"; + sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd"; + }; + + meta = { + homepage = "http://github.com/pytoolz/toolz/"; + description = "List processing tools and functional utilities"; + license = licenses.bsd3; + maintainers = with maintainers; [ fridh ]; + }; + }; + + in pkgs.python35.buildEnv.override rec { + + extraLibs = [ pkgs.python35Packages.numpy toolz ]; +} +).env +``` + +and again execute `nix-shell`, then we get a Python 3.5 environment with our +locally defined package as well as `numpy` which is build according to the +definition in Nixpkgs. What did we do here? Well, we took the Nix expression +that we used earlier to build a Python environment, and said that we wanted to +include our own version of `toolz`. To introduce our own package in the scope of +`buildEnv.override` we used a +[`let`](http://nixos.org/nix/manual/#sec-constructs) expression. + +### Handling dependencies + +Our example, `toolz`, doesn't have any dependencies on other Python +packages or system libraries. According to the manual, `buildPythonPackage` +uses the arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If something is +exclusively a build-time dependency, then the dependency should be included as a +`buildInput`, but if it is (also) a runtime dependency, then it should be added +to `propagatedBuildInputs`. Test dependencies are considered build-time dependencies. + +The following example shows which arguments are given to `buildPythonPackage` in +order to build [`datashape`](https://github.com/blaze/datashape). + +```nix +datashape = buildPythonPackage rec { + name = "datashape-${version}"; + version = "0.4.7"; + + src = pkgs.fetchurl { + url = "https://pypi.python.org/packages/source/D/DataShape/${name}.tar.gz"; + sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278"; + }; + + buildInputs = with self; [ pytest ]; + propagatedBuildInputs = with self; [ numpy multipledispatch dateutil ]; + + meta = { + homepage = https://github.com/ContinuumIO/datashape; + description = "A data description language"; + license = licenses.bsd2; + maintainers = with maintainers; [ fridh ]; + }; +}; +``` + +We can see several runtime dependencies, `numpy`, `multipledispatch`, and +`dateutil`. Furthermore, we have one `buildInput`, i.e. `pytest`. `pytest` is a +test runner and is only used during the `checkPhase` and is therefore not added +to `propagatedBuildInputs`. + +In the previous case we had only dependencies on other Python packages to consider. +Occasionally you have also system libraries to consider. E.g., `lxml` provides +Python bindings to `libxml2` and `libxslt`. These libraries are only required +when building the bindings and are therefore added as `buildInputs`. + +```nix +lxml = buildPythonPackage rec { + name = "lxml-3.4.4"; + + src = pkgs.fetchurl { + url = "http://pypi.python.org/packages/source/l/lxml/${name}.tar.gz"; + sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk"; + }; + + buildInputs = with self; [ pkgs.libxml2 pkgs.libxslt ]; + + meta = { + description = "Pythonic binding for the libxml2 and libxslt libraries"; + homepage = http://lxml.de; + license = licenses.bsd3; + maintainers = with maintainers; [ sjourdois ]; + }; +}; +``` + +In this example `lxml` and Nix are able to work out exactly where the relevant +files of the dependencies are. This is not always the case. + +The example below shows bindings to The Fastest Fourier Transform in the West, commonly known as +FFTW. On Nix we have separate packages of FFTW for the different types of floats +(`"single"`, `"double"`, `"long-double"`). The bindings need all three types, +and therefore we add all three as `buildInputs`. The bindings don't expect to +find each of them in a different folder, and therefore we have to set `LDFLAGS` +and `CFLAGS`. + +```nix +pyfftw = buildPythonPackage rec { + name = "pyfftw-${version}"; + version = "0.9.2"; + + src = pkgs.fetchurl { + url = "https://pypi.python.org/packages/source/p/pyFFTW/pyFFTW-${version}.tar.gz"; + sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074"; + }; + + buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble]; + + propagatedBuildInputs = with self; [ numpy scipy ]; + + # Tests cannot import pyfftw. pyfftw works fine though. + doCheck = false; + + LDFLAGS="-L${pkgs.fftw}/lib -L${pkgs.fftwFloat}/lib -L${pkgs.fftwLongDouble}/lib" + CFLAGS="-I${pkgs.fftw}/include -I${pkgs.fftwFloat}/include -I${pkgs.fftwLongDouble}/include" + ''; + + meta = { + description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms"; + homepage = http://hgomersall.github.com/pyFFTW/; + license = with licenses; [ bsd2 bsd3 ]; + maintainer = with maintainers; [ fridh ]; + }; +}; +``` +Note also the line `doCheck = false;`, we explicitly disabled running the test-suite. + + +#### Develop local package + +As a Python developer you're likely aware of [development mode](http://pythonhosted.org/setuptools/setuptools.html#development-mode) (`python setup.py develop`); +instead of installing the package this command creates a special link to the project code. +That way, you can run updated code without having to reinstall after each and every change you make. +Development mode is also available on Nix as [explained](http://nixos.org/nixpkgs/manual/#ssec-python-development) in the Nixpkgs manual. +Let's see how you can use it. + +In the previous Nix expression the source was fetched from an url. We can also refer to a local source instead using + +```nix +src = ./path/to/source/tree; +``` + +If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src` +is a local source, and if the local source has a `setup.py`, then development +mode is activated. + +In the following example we create a simple environment that +has a Python 3.5 version of our package in it, as well as its dependencies and +other packages we like to have in the environment, all specified with `propagatedBuildInputs`. +Indeed, we can just add any package we like to have in our environment to `propagatedBuildInputs`. + +```nix +with import ; +with pkgs.python35Packages; + +buildPythonPackage rec { + name = "mypackage"; + src = ./path/to/package/source; + propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ]; +}; +``` + +It is important to note that due to how development mode is implemented on Nix it is not possible to have multiple packages simultaneously in development mode. + + +### Organising your packages + +So far we discussed how you can use Python on Nix, and how you can develop with +it. We've looked at how you write expressions to package Python packages, and we +looked at how you can create environments in which specified packages are +available. + +At some point you'll likely have multiple packages which you would +like to be able to use in different projects. In order to minimise unnecessary +duplication we now look at how you can maintain yourself a repository with your +own packages. The important functions here are `import` and `callPackage`. + +### Including a derivation using `callPackage` + +Earlier we created a Python environment using `buildEnv`, and included the +`toolz` package via a `let` expression. +Let's split the package definition from the environment definition. + +We first create a function that builds `toolz` in `~/path/to/toolz/release.nix` + +```nix +{ pkgs, buildPythonPackage }: + +buildPythonPackage rec { + name = "toolz-${version}"; + version = "0.7.4"; + + src = pkgs.fetchurl{ + url = "https://pypi.python.org/packages/source/t/toolz/toolz-${version}.tar.gz"; + sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd"; + }; + + meta = { + homepage = "http://github.com/pytoolz/toolz/"; + description = "List processing tools and functional utilities"; + license = licenses.bsd3; + maintainers = with maintainers; [ fridh ]; + }; +}; +``` + +It takes two arguments, `pkgs` and `buildPythonPackage`. +We now call this function using `callPackage` in the definition of our environment + +```nix +with import {}; + +( let + toolz = pkgs.callPackage ~/path/to/toolz/release.nix { pkgs=pkgs; buildPythonPackage=pkgs.python35Packages.buildPythonPackage; }; + in pkgs.python35.buildEnv.override rec { + extraLibs = [ pkgs.python35Packages.numpy toolz ]; +} +).env +``` + +Important to remember is that the Python version for which the package is made +depends on the `python` derivation that is passed to `buildPythonPackage`. Nix +tries to automatically pass arguments when possible, which is why generally you +don't explicitly define which `python` derivation should be used. In the above +example we use `buildPythonPackage` that is part of the set `python35Packages`, +and in this case the `python35` interpreter is automatically used. + + + +## Reference + +### Interpreters + +Versions 2.6, 2.7, 3.3, 3.4 and 3.5 of the CPython interpreter are available on +Nix and are available as `python26`, `python27`, `python33`, `python34` and +`python35`. The PyPy interpreter is also available as `pypy`. Currently, the +aliases `python` and `python3` correspond to respectively `python27` and +`python35`. The Nix expressions for the interpreters can be found in +`pkgs/development/interpreters/python`. + + +#### Missing modules standard library + +The interpreters `python26` and `python27` do not include modules that +require external dependencies. This is done in order to reduce the closure size. +The following modules need to be added as `buildInput` explicitly: + +* `python.modules.bsddb` +* `python.modules.curses` +* `python.modules.curses_panel` +* `python.modules.crypt` +* `python.modules.gdbm` +* `python.modules.sqlite3` +* `python.modules.tkinter` +* `python.modules.readline` + +For convenience `python27Full` and `python26Full` are provided with all +modules included. + +All packages depending on any Python interpreter get appended +`out/{python.sitePackages}` to `$PYTHONPATH` if such directory +exists. + +#### Attributes on interpreters packages + +Each interpreter has the following attributes: + +- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter. +- `interpreter`. Alias for `${python}/bin/${executable}`. +- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation. +- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`. +- `executable`. Name of the interpreter executable, ie `python3.4`. + +### Building packages and applications + +Python packages (libraries) and applications that use `setuptools` or +`distutils` are typically built with respectively the `buildPythonPackage` and +`buildPythonApplication` functions. + +All Python packages reside in `pkgs/top-level/python-packages.nix` and all +applications elsewhere. Some packages are also defined in +`pkgs/development/python-modules`. It is important that these packages are +called in `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee +the right version of the package is built. + +Based on the packages defined in `pkgs/top-level/python-packages.nix` an +attribute set is created for each available Python interpreter. The available +sets are + +* `pkgs.python26Packages` +* `pkgs.python27Packages` +* `pkgs.python33Packages` +* `pkgs.python34Packages` +* `pkgs.python35Packages` +* `pkgs.pypyPackages` + +and the aliases + +* `pkgs.pythonPackages` pointing to `pkgs.python27Packages` +* `pkgs.python3Packages` pointing to `pkgs.python35Packages` + +#### `buildPythonPackage` function + +The `buildPythonPackage` function is implemented in +`pkgs/development/python-modules/generic/default.nix` + +and can be used as: + + twisted = buildPythonPackage { + name = "twisted-8.1.0"; + + src = pkgs.fetchurl { + url = http://tmrc.mit.edu/mirror/twisted/Twisted/8.1/Twisted-8.1.0.tar.bz2; + sha256 = "0q25zbr4xzknaghha72mq57kh53qw1bf8csgp63pm9sfi72qhirl"; + }; + + propagatedBuildInputs = [ self.ZopeInterface ]; + + meta = { + homepage = http://twistedmatrix.com/; + description = "Twisted, an event-driven networking engine written in Python"; + license = stdenv.lib.licenses.mit; }; + }; + +The `buildPythonPackage` mainly does four things: + +* In the `buildPhase`, it calls `${python.interpreter} setup.py bdist_wheel` to build a wheel binary zipfile. +* In the `installPhase`, it installs the wheel file using `pip install *.whl`. +* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to wrap all programs in the `$out/bin/*` directory to include `$PYTHONPATH` and `$PATH` environment variables. +* In the `installCheck` phase, `${python.interpreter} setup.py test` is ran. + +As in Perl, dependencies on other Python packages can be specified in the +`buildInputs` and `propagatedBuildInputs` attributes. If something is +exclusively a build-time dependency, use `buildInputs`; if it’s (also) a runtime +dependency, use `propagatedBuildInputs`. + +By default tests are run because `doCheck = true`. Test dependencies, like +e.g. the test runner, should be added to `buildInputs`. + +By default `meta.platforms` is set to the same value +as the interpreter unless overriden otherwise. + +##### `buildPythonPackage` parameters + +All parameters from `mkDerivation` function are still supported. + +* `namePrefix`: Prepended text to `${name}` parameter. Defaults to `"python3.3-"` for Python 3.3, etc. Set it to `""` if you're packaging an application or a command line tool. +* `disabled`: If `true`, package is not build for particular python interpreter version. Grep around `pkgs/top-level/python-packages.nix` for examples. +* `setupPyBuildFlags`: List of flags passed to `setup.py build_ext` command. +* `pythonPath`: List of packages to be added into `$PYTHONPATH`. Packages in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`). +* `preShellHook`: Hook to execute commands before `shellHook`. +* `postShellHook`: Hook to execute commands after `shellHook`. +* `makeWrapperArgs`: A list of strings. Arguments to be passed to `makeWrapper`, which wraps generated binaries. By default, the arguments to `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling the binary. Additional arguments here can allow a developer to set environment variables which will be available when the binary is run. For example, `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`. +* `installFlags`: A list of strings. Arguments to be passed to `pip install`. To pass options to `python setup.py install`, use `--install-option`. E.g., `installFlags=["--install-option='--cpp_implementation'"]. + +#### `buildPythonApplication` function + +The `buildPythonApplication` function is practically the same as `buildPythonPackage`. +The difference is that `buildPythonPackage` by default prefixes the names of the packages with the version of the interpreter. +Because with an application we're not interested in multiple version the prefix is dropped. + +#### python.buildEnv function + +Python environments can be created using the low-level `pkgs.buildEnv` function. +This example shows how to create an environment that has the Pyramid Web Framework. +Saving the following as `default.nix` + + with import {}; + + python.buildEnv.override { + extraLibs = [ pkgs.pythonPackages.pyramid ]; + ignoreCollisions = true; + } + +and running `nix-build` will create + + /nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env + +with wrapped binaries in `bin/`. + +You can also use the `env` attribute to create local environments with needed +packages installed. This is somewhat comparable to `virtualenv`. For example, +running `nix-shell` with the following `shell.nix` + + with import {}; + + (python3.buildEnv.override { + extraLibs = with python3Packages; [ numpy requests ]; + }).env + +will drop you into a shell where Python will have the +specified packages in its path. + + +##### `python.buildEnv` arguments + +* `extraLibs`: List of packages installed inside the environment. +* `postBuild`: Shell command executed after the build of environment. +* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`). + +### Development mode + +Development or editable mode is supported. To develop Python packages +`buildPythonPackage` has additional logic inside `shellPhase` to run `pip +install -e . --prefix $TMPDIR/`for the package. + +Warning: `shellPhase` is executed only if `setup.py` exists. + +Given a `default.nix`: + + with import {}; + + buildPythonPackage { name = "myproject"; + + buildInputs = with pkgs.pythonPackages; [ pyramid ]; + + src = ./.; } + +Running `nix-shell` with no arguments should give you +the environment in which the package would be build with +`nix-build`. + +Shortcut to setup environments with C headers/libraries and python packages: + + $ nix-shell -p pythonPackages.pyramid zlib libjpeg git + +Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked. + +### Tools + +Packages inside nixpkgs are written by hand. However many tools exist in +community to help save time. No tool is preferred at the moment. + +- [python2nix](https://github.com/proger/python2nix) by Vladimir Kirillov +- [pypi2nix](https://github.com/garbas/pypi2nix) by Rok Garbas +- [pypi2nix](https://github.com/offlinehacker/pypi2nix) by Jaka Hudoklin + +## FAQ + +### How to solve circular dependencies? + +Consider the packages `A` and `B` that depend on each other. When packaging `B`, +a solution is to override package `A` not to depend on `B` as an input. The same +should also be done when packaging `A`. + +### How to override a Python package? + +Recursively updating a package can be done with `pkgs.overridePackages` as explained in the Nixpkgs manual. +Python attribute sets are created for each interpreter version. We will therefore override the attribute set for the interpreter version we're interested. +In the following example we change the name of the package `pandas` to `foo`. +``` +newpkgs = pkgs.overridePackages(self: super: rec { + python35Packages = super.python35Packages.override { + self = python35Packages // { pandas = python35Packages.pandas.override{name="foo";};}; + }; +}); +``` +This can be tested with +``` +with import {}; + +(let + +newpkgs = pkgs.overridePackages(self: super: rec { + python35Packages = super.python35Packages.override { + self = python35Packages // { pandas = python35Packages.pandas.override{name="foo";};}; + }; +}); +in newpkgs.python35.buildEnv.override{ + extraLibs = [newpkgs.python35Packages.blaze ]; +}).env +``` +A typical use case is to switch to another version of a certain package. For example, in the Nixpkgs repository we have multiple versions of `django` and `scipy`. +In the following example we use a different version of `scipy`. All packages in `newpkgs` will now use the updated `scipy` version. +``` +with import {}; + +(let + +newpkgs = pkgs.overridePackages(self: super: rec { + python35Packages = super.python35Packages.override { + self = python35Packages // { scipy = python35Packages.scipy_0_16;}; + }; +}); +in pkgs.python35.buildEnv.override{ + extraLibs = [newpkgs.python35Packages.blaze ]; +}).env +``` +The requested package `blaze` depends upon `pandas` which itself depends on `scipy`. + + +### `install_data` / `data_files` problems + +If you get the following error: + + could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc': + Permission denied + +This is a [known bug](https://bitbucket.org/pypa/setuptools/issue/130/install_data-doesnt-respect-prefix) in setuptools. +Setuptools `install_data` does not respect `--prefix`. An example of such package using the feature is `pkgs/tools/X11/xpra/default.nix`. +As workaround install it as an extra `preInstall` step: + + ${python.interpreter} setup.py install_data --install-dir=$out --root=$out + sed -i '/ = data\_files/d' setup.py + + +### Rationale of non-existent global site-packages + +On most operating systems a global `site-packages` is maintained. This however +becomes problematic if you want to run multiple Python versions or have multiple +versions of certain libraries for your projects. Generally, you would solve such +issues by creating virtual environments using `virtualenv`. + +On Nix each package has an isolated dependency tree which, in the case of +Python, guarantees the right versions of the interpreter and libraries or +packages are available. There is therefore no need to maintain a global `site-packages`. + +If you want to create a Python environment for development, then the recommended +method is to use `nix-shell`, either with or without the `python.buildEnv` +function. + + +## Contributing + +### Contributing guidelines + +Following rules are desired to be respected: + +* Make sure package builds for all python interpreters. Use `disabled` argument to `buildPythonPackage` to set unsupported interpreters. +* If tests need to be disabled for a package, make sure you leave a comment about reasoning. +* Packages in `pkgs/top-level/python-packages.nix` are sorted quasi-alphabetically to avoid merge conflicts. +* Python libraries are supposed to be in `python-packages.nix` and packaged with `buildPythonPackage`. Python applications live outside of `python-packages.nix` and are packaged with `buildPythonApplication`. diff --git a/doc/languages-frameworks/python.xml b/doc/languages-frameworks/python.xml deleted file mode 100644 index 57aceeb48685..000000000000 --- a/doc/languages-frameworks/python.xml +++ /dev/null @@ -1,447 +0,0 @@ -
- -Python - - - Currently supported interpreters are python26, python27, - python33, python34, python35 - and pypy. - - - - python is an alias to python27 and python3 is an alias to python34. - - - - python26 and python27 do not include modules that require - external dependencies (to reduce dependency bloat). Following modules need to be added as - buildInput explicitly: - - - - python.modules.bsddb - python.modules.curses - python.modules.curses_panel - python.modules.crypt - python.modules.gdbm - python.modules.sqlite3 - python.modules.tkinter - python.modules.readline - - -For convenience python27Full and python26Full -are provided with all modules included. - - - Python packages that - use setuptools or distutils, - can be built using the buildPythonPackage function as documented below. - - - - All packages depending on any Python interpreter get appended $out/${python.sitePackages} - to $PYTHONPATH if such directory exists. - - - - - Useful attributes on interpreters packages: - - - - libPrefix - - Name of the folder in ${python}/lib/ for corresponding interpreter. - - - - - interpreter - - Alias for ${python}/bin/${executable}. - - - - - buildEnv - - Function to build python interpreter environments with extra packages bundled together. - See for usage and documentation. - - - - - sitePackages - - Alias for lib/${libPrefix}/site-packages. - - - - - executable - - Name of the interpreter executable, ie python3.4. - - - - -
<varname>buildPythonPackage</varname> function - - - The function is implemented in - pkgs/development/python-modules/generic/default.nix. - Example usage: - - -twisted = buildPythonPackage { - name = "twisted-8.1.0"; - - src = pkgs.fetchurl { - url = http://tmrc.mit.edu/mirror/twisted/Twisted/8.1/Twisted-8.1.0.tar.bz2; - sha256 = "0q25zbr4xzknaghha72mq57kh53qw1bf8csgp63pm9sfi72qhirl"; - }; - - propagatedBuildInputs = [ self.ZopeInterface ]; - - meta = { - homepage = http://twistedmatrix.com/; - description = "Twisted, an event-driven networking engine written in Python"; - license = stdenv.lib.licenses.mit; - }; -}; - - - Most of Python packages that use buildPythonPackage are defined - in pkgs/top-level/python-packages.nix - and generated for each python interpreter separately into attribute sets python26Packages, - python27Packages, python35Packages, python33Packages, - python34Packages and pypyPackages. - - - - buildPythonPackage mainly does four things: - - - - In the buildPhase, it calls - ${python.interpreter} setup.py bdist_wheel to build a wheel binary zipfile. - - - - In the installPhase, it installs the wheel file using - pip install *.whl. - - - - In the postFixup phase, wrapPythonPrograms - bash function is called to wrap all programs in $out/bin/* - directory to include $PYTHONPATH and $PATH - environment variables. - - - - In the installCheck phase, ${python.interpreter} setup.py test - is ran. - - - - - By default doCheck = true is set - - - As in Perl, dependencies on other Python packages can be specified in the - buildInputs and - propagatedBuildInputs attributes. If something is - exclusively a build-time dependency, use - buildInputs; if it’s (also) a runtime dependency, - use propagatedBuildInputs. - - - - By default meta.platforms is set to the same value - as the interpreter unless overriden otherwise. - - - - - <varname>buildPythonPackage</varname> parameters - (all parameters from <varname>mkDerivation</varname> function are still supported) - - - - namePrefix - - Prepended text to ${name} parameter. - Defaults to "python3.3-" for Python 3.3, etc. Set it to - "" - if you're packaging an application or a command line tool. - - - - - disabled - - If true, package is not build for - particular python interpreter version. Grep around - pkgs/top-level/python-packages.nix - for examples. - - - - - setupPyBuildFlags - - List of flags passed to setup.py build_ext command. - - - - - pythonPath - - List of packages to be added into $PYTHONPATH. - Packages in pythonPath are not propagated - (contrary to propagatedBuildInputs). - - - - - preShellHook - - Hook to execute commands before shellHook. - - - - - postShellHook - - Hook to execute commands after shellHook. - - - - - makeWrapperArgs - - A list of strings. Arguments to be passed to - makeWrapper, which wraps generated binaries. By - default, the arguments to makeWrapper set - PATH and PYTHONPATH environment - variables before calling the binary. Additional arguments here can - allow a developer to set environment variables which will be - available when the binary is run. For example, - makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]. - - - - - -
- -
<function>python.buildEnv</function> function - - Create Python environments using low-level pkgs.buildEnv function. Example default.nix: - - - {}; - -python.buildEnv.override { - extraLibs = [ pkgs.pythonPackages.pyramid ]; - ignoreCollisions = true; -}]]> - - - Running nix-build will create - /nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env - with wrapped binaries in bin/. - - - - You can also use env attribute to create local - environments with needed packages installed (somewhat comparable to - virtualenv). For example, with the following - shell.nix: - - - {}; - -(python3.buildEnv.override { - extraLibs = with python3Packages; - [ numpy - requests - ]; -}).env]]> - - - Running nix-shell will drop you into a shell where - python will have specified packages in its path. - - - - - <function>python.buildEnv</function> arguments - - - - extraLibs - - List of packages installed inside the environment. - - - - - postBuild - - Shell command executed after the build of environment. - - - - - ignoreCollisions - - Ignore file collisions inside the environment (default is false). - - - -
- -
Tools - -Packages inside nixpkgs are written by hand. However many tools -exist in community to help save time. No tool is preferred at the moment. - - - - - - python2nix - by Vladimir Kirillov - - - - pypi2nix - by Rok Garbas - - - - pypi2nix - by Jaka Hudoklin - - - - -
- -
Development - - - To develop Python packages buildPythonPackage has - additional logic inside shellPhase to run - pip install -e . --prefix $TMPDIR/ for the package. - - - shellPhase is executed only if setup.py - exists. - - - Given a default.nix: - - - {}; - -buildPythonPackage { - name = "myproject"; - - buildInputs = with pkgs.pythonPackages; [ pyramid ]; - - src = ./.; -}]]> - - - Running nix-shell with no arguments should give you - the environment in which the package would be build with - nix-build. - - - - Shortcut to setup environments with C headers/libraries and python packages: - - $ nix-shell -p pythonPackages.pyramid zlib libjpeg git - - - - There is a boolean value lib.inNixShell set to - true if nix-shell is invoked. - - -
- -
FAQ - - - - - How to solve circular dependencies? - - If you have packages A and B that - depend on each other, when packaging B override package - A not to depend on B as input - (and also the other way around). - - - - - install_data / data_files problems resulting into error: could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc': Permission denied - - - Known bug in setuptools install_data does not respect --prefix. Example of - such package using the feature is pkgs/tools/X11/xpra/default.nix. As workaround - install it as an extra preInstall step: - - ${python.interpreter} setup.py install_data --install-dir=$out --root=$out -sed -i '/ = data_files/d' setup.py - - - - - Rationale of non-existent global site-packages - - There is no need to have global site-packages in Nix. Each package has isolated - dependency tree and installing any python package will only populate $PATH - inside user environment. See to create self-contained - interpreter with a set of packages. - - - - - -
- - -
Contributing guidelines - - Following rules are desired to be respected: - - - - - - Make sure package builds for all python interpreters. Use disabled argument to - buildPythonPackage to set unsupported interpreters. - - - - If tests need to be disabled for a package, make sure you leave a comment about reasoning. - - - - Packages in pkgs/top-level/python-packages.nix - are sorted quasi-alphabetically to avoid merge conflicts. - - - - -
- -
-