mirror of
https://github.com/NixOS/nixpkgs.git
synced 2024-12-11 23:24:35 +00:00
bbc9af1f0a
Each invocation of pkgs.extends adds 130MB of allocation to the hydra evaluator. We are already struggling with the amount of memory nixpkgs requires. `pkgs.extend` is a useful escape-hatch, but should be not be used inside of nixpkgs directly.
296 lines
13 KiB
XML
296 lines
13 KiB
XML
<chapter xmlns="http://docbook.org/ns/docbook"
|
||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||
xml:id="chap-overlays">
|
||
<title>Overlays</title>
|
||
<para>
|
||
This chapter describes how to extend and change Nixpkgs using overlays. Overlays are used to add layers in the fixed-point used by Nixpkgs to compose the set of all packages.
|
||
</para>
|
||
<para>
|
||
Nixpkgs can be configured with a list of overlays, which are applied in order. This means that the order of the overlays can be significant if multiple layers override the same package.
|
||
</para>
|
||
<!--============================================================-->
|
||
<section xml:id="sec-overlays-install">
|
||
<title>Installing overlays</title>
|
||
|
||
<para>
|
||
The list of overlays can be set either explicitly in a Nix expression, or through <literal><nixpkgs-overlays></literal> or user configuration files.
|
||
</para>
|
||
|
||
<section xml:id="sec-overlays-argument">
|
||
<title>Set overlays in NixOS or Nix expressions</title>
|
||
|
||
<para>
|
||
On a NixOS system the value of the <literal>nixpkgs.overlays</literal> option, if present, is passed to the system Nixpkgs directly as an argument. Note that this does not affect the overlays for non-NixOS operations (e.g. <literal>nix-env</literal>), which are <link xlink:href="#sec-overlays-lookup">looked</link> up independently.
|
||
</para>
|
||
|
||
<para>
|
||
The list of overlays can be passed explicitly when importing nixpkgs, for example <literal>import <nixpkgs> { overlays = [ overlay1 overlay2 ]; }</literal>.
|
||
</para>
|
||
|
||
<para>
|
||
NOTE: DO NOT USE THIS in nixpkgs.
|
||
Further overlays can be added by calling the <literal>pkgs.extend</literal> or <literal>pkgs.appendOverlays</literal>, although it is often preferable to avoid these functions, because they recompute the Nixpkgs fixpoint, which is somewhat expensive to do.
|
||
</para>
|
||
</section>
|
||
|
||
<section xml:id="sec-overlays-lookup">
|
||
<title>Install overlays via configuration lookup</title>
|
||
|
||
<para>
|
||
The list of overlays is determined as follows.
|
||
</para>
|
||
|
||
<para>
|
||
<orderedlist>
|
||
<listitem>
|
||
<para>
|
||
First, if an <link xlink:href="#sec-overlays-argument"><varname>overlays</varname> argument</link> to the Nixpkgs function itself is given, then that is used and no path lookup will be performed.
|
||
</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>
|
||
Otherwise, if the Nix path entry <literal><nixpkgs-overlays></literal> exists, we look for overlays at that path, as described below.
|
||
</para>
|
||
<para>
|
||
See the section on <literal>NIX_PATH</literal> in the Nix manual for more details on how to set a value for <literal><nixpkgs-overlays>.</literal>
|
||
</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>
|
||
If one of <filename>~/.config/nixpkgs/overlays.nix</filename> and <filename>~/.config/nixpkgs/overlays/</filename> exists, then we look for overlays at that path, as described below. It is an error if both exist.
|
||
</para>
|
||
</listitem>
|
||
</orderedlist>
|
||
</para>
|
||
|
||
<para>
|
||
If we are looking for overlays at a path, then there are two cases:
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para>
|
||
If the path is a file, then the file is imported as a Nix expression and used as the list of overlays.
|
||
</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>
|
||
If the path is a directory, then we take the content of the directory, order it lexicographically, and attempt to interpret each as an overlay by:
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para>
|
||
Importing the file, if it is a <literal>.nix</literal> file.
|
||
</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>
|
||
Importing a top-level <filename>default.nix</filename> file, if it is a directory.
|
||
</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
</para>
|
||
|
||
<para>
|
||
Because overlays that are set in NixOS configuration do not affect non-NixOS operations such as <literal>nix-env</literal>, the <filename>overlays.nix</filename> option provides a convenient way to use the same overlays for a NixOS system configuration and user configuration: the same file can be used as <filename>overlays.nix</filename> and imported as the value of <literal>nixpkgs.overlays</literal>.
|
||
</para>
|
||
|
||
<!-- TODO: Example of sharing overlays between NixOS configuration
|
||
and configuration lookup. Also reference the example
|
||
from the sec-overlays-argument paragraph about NixOS.
|
||
-->
|
||
</section>
|
||
</section>
|
||
<!--============================================================-->
|
||
<section xml:id="sec-overlays-definition">
|
||
<title>Defining overlays</title>
|
||
|
||
<para>
|
||
Overlays are Nix functions which accept two arguments, conventionally called <varname>self</varname> and <varname>super</varname>, and return a set of packages. For example, the following is a valid overlay.
|
||
</para>
|
||
|
||
<programlisting>
|
||
self: super:
|
||
|
||
{
|
||
boost = super.boost.override {
|
||
python = self.python3;
|
||
};
|
||
rr = super.callPackage ./pkgs/rr {
|
||
stdenv = self.stdenv_32bit;
|
||
};
|
||
}
|
||
</programlisting>
|
||
|
||
<para>
|
||
The first argument (<varname>self</varname>) corresponds to the final package set. You should use this set for the dependencies of all packages specified in your overlay. For example, all the dependencies of <varname>rr</varname> in the example above come from <varname>self</varname>, as well as the overridden dependencies used in the <varname>boost</varname> override.
|
||
</para>
|
||
|
||
<para>
|
||
The second argument (<varname>super</varname>) corresponds to the result of the evaluation of the previous stages of Nixpkgs. It does not contain any of the packages added by the current overlay, nor any of the following overlays. This set should be used either to refer to packages you wish to override, or to access functions defined in Nixpkgs. For example, the original recipe of <varname>boost</varname> in the above example, comes from <varname>super</varname>, as well as the <varname>callPackage</varname> function.
|
||
</para>
|
||
|
||
<para>
|
||
The value returned by this function should be a set similar to <filename>pkgs/top-level/all-packages.nix</filename>, containing overridden and/or new packages.
|
||
</para>
|
||
|
||
<para>
|
||
Overlays are similar to other methods for customizing Nixpkgs, in particular the <literal>packageOverrides</literal> attribute described in <xref linkend="sec-modify-via-packageOverrides"/>. Indeed, <literal>packageOverrides</literal> acts as an overlay with only the <varname>super</varname> argument. It is therefore appropriate for basic use, but overlays are more powerful and easier to distribute.
|
||
</para>
|
||
</section>
|
||
<section xml:id="sec-overlays-alternatives">
|
||
<title>Using overlays to configure alternatives</title>
|
||
<para>
|
||
Certain software packages have different implementations of the
|
||
same interface. Other distributions have functionality to switch
|
||
between these. For example, Debian provides <link
|
||
xlink:href="https://wiki.debian.org/DebianAlternatives">DebianAlternatives</link>.
|
||
Nixpkgs has what we call <literal>alternatives</literal>, which
|
||
are configured through overlays.
|
||
</para>
|
||
<section xml:id="sec-overlays-alternatives-blas-lapack">
|
||
<title>BLAS/LAPACK</title>
|
||
<para>
|
||
In Nixpkgs, we have multiple implementations of the BLAS/LAPACK
|
||
numerical linear algebra interfaces. They are:
|
||
</para>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para>
|
||
<link xlink:href="https://www.openblas.net/">OpenBLAS</link>
|
||
</para>
|
||
<para>
|
||
The Nixpkgs attribute is <literal>openblas</literal> for
|
||
ILP64 (integer width = 64 bits) and
|
||
<literal>openblasCompat</literal> for LP64 (integer width =
|
||
32 bits). <literal>openblasCompat</literal> is the default.
|
||
</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>
|
||
<link xlink:href="http://www.netlib.org/lapack/">LAPACK
|
||
reference</link> (also provides BLAS)
|
||
</para>
|
||
<para>
|
||
The Nixpkgs attribute is <literal>lapack-reference</literal>.
|
||
</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>
|
||
<link
|
||
xlink:href="https://software.intel.com/en-us/mkl">Intel
|
||
MKL</link> (only works on the x86_64 architecture, unfree)
|
||
</para>
|
||
<para>
|
||
The Nixpkgs attribute is <literal>mkl</literal>.
|
||
</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>
|
||
<link
|
||
xlink:href="https://github.com/flame/blis">BLIS</link>
|
||
</para>
|
||
<para>
|
||
BLIS, available through the attribute
|
||
<literal>blis</literal>, is a framework for linear algebra kernels. In
|
||
addition, it implements the BLAS interface.
|
||
</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>
|
||
<link
|
||
xlink:href="https://developer.amd.com/amd-aocl/blas-library/">AMD
|
||
BLIS/LIBFLAME</link> (optimized for modern AMD x86_64 CPUs)
|
||
</para>
|
||
<para>
|
||
The AMD fork of the BLIS library, with attribute
|
||
<literal>amd-blis</literal>, extends BLIS with optimizations for
|
||
modern AMD CPUs. The changes are usually submitted to
|
||
the upstream BLIS project after some time. However, AMD BLIS
|
||
typically provides some performance improvements on AMD Zen CPUs.
|
||
The complementary AMD LIBFLAME library, with attribute
|
||
<literal>amd-libflame</literal>, provides a LAPACK implementation.
|
||
</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<para>
|
||
Introduced in <link
|
||
xlink:href="https://github.com/NixOS/nixpkgs/pull/83888">PR
|
||
#83888</link>, we are able to override the <literal>blas</literal>
|
||
and <literal>lapack</literal> packages to use different implementations,
|
||
through the <literal>blasProvider</literal> and
|
||
<literal>lapackProvider</literal> argument. This can be used
|
||
to select a different provider. BLAS providers will have
|
||
symlinks in <literal>$out/lib/libblas.so.3</literal> and
|
||
<literal>$out/lib/libcblas.so.3</literal> to their respective
|
||
BLAS libraries. Likewise, LAPACK providers will have symlinks
|
||
in <literal>$out/lib/liblapack.so.3</literal> and
|
||
<literal>$out/lib/liblapacke.so.3</literal> to their respective
|
||
LAPACK libraries. For example, Intel MKL is both a BLAS and
|
||
LAPACK provider. An overlay can be created to use Intel MKL
|
||
that looks like:
|
||
</para>
|
||
<programlisting>
|
||
self: super:
|
||
|
||
{
|
||
blas = super.blas.override {
|
||
blasProvider = self.mkl;
|
||
}
|
||
lapack = super.lapack.override {
|
||
lapackProvider = self.mkl;
|
||
}
|
||
}
|
||
</programlisting>
|
||
<para>
|
||
This overlay uses Intel’s MKL library for both BLAS and LAPACK
|
||
interfaces. Note that the same can be accomplished at runtime
|
||
using <literal>LD_LIBRARY_PATH</literal> of
|
||
<literal>libblas.so.3</literal> and
|
||
<literal>liblapack.so.3</literal>. For instance:
|
||
</para>
|
||
<screen>
|
||
<prompt>$ </prompt>LD_LIBRARY_PATH=$(nix-build -A mkl)/lib:$LD_LIBRARY_PATH nix-shell -p octave --run octave
|
||
</screen>
|
||
<para>
|
||
Intel MKL requires an <literal>openmp</literal> implementation
|
||
when running with multiple processors. By default,
|
||
<literal>mkl</literal> will use Intel’s <literal>iomp</literal>
|
||
implementation if no other is specified, but this is a
|
||
runtime-only dependency and binary compatible with the LLVM
|
||
implementation. To use that one instead, Intel recommends users
|
||
set it with <literal>LD_PRELOAD</literal>. Note that
|
||
<literal>mkl</literal> is only available on
|
||
<literal>x86_64-linux</literal> and
|
||
<literal>x86_64-darwin</literal>. Moreover, Hydra is not
|
||
building and distributing pre-compiled binaries using it.
|
||
</para>
|
||
<para>
|
||
For BLAS/LAPACK switching to work correctly, all packages must
|
||
depend on <literal>blas</literal> or <literal>lapack</literal>.
|
||
This ensures that only one BLAS/LAPACK library is used at one
|
||
time. There are two versions versions of BLAS/LAPACK currently
|
||
in the wild, <literal>LP64</literal> (integer size = 32 bits)
|
||
and <literal>ILP64</literal> (integer size = 64 bits). Some
|
||
software needs special flags or patches to work with
|
||
<literal>ILP64</literal>. You can check if
|
||
<literal>ILP64</literal> is used in Nixpkgs with
|
||
<varname>blas.isILP64</varname> and
|
||
<varname>lapack.isILP64</varname>. Some software does NOT work
|
||
with <literal>ILP64</literal>, and derivations need to specify
|
||
an assertion to prevent this. You can prevent
|
||
<literal>ILP64</literal> from being used with the following:
|
||
</para>
|
||
<programlisting>
|
||
{ stdenv, blas, lapack, ... }:
|
||
|
||
assert (!blas.isILP64) && (!lapack.isILP64);
|
||
|
||
stdenv.mkDerivation {
|
||
...
|
||
}
|
||
</programlisting>
|
||
</section>
|
||
</section>
|
||
</chapter>
|