nixpkgs/doc/manual/development.xml

<chapter xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink">

<title>Development</title>

<para>This chapter has some random notes on hacking on
NixOS.</para>


<!--===============================================================-->

<section>

<title>Hacking on NixOS</title>

<para>By default, NixOS’s <command>nixos-rebuild</command> command
uses the NixOS and Nixpkgs sources provided by the
<literal>nixos-unstable</literal> channel (kept in
<filename>/nix/var/nix/profiles/per-user/root/channels/nixos</filename>).
To modify NixOS, however, you should check out the latest sources from
Git.  This is done using the following command:

<screen>
$ nixos-checkout <replaceable>/my/sources</replaceable>
</screen>

or

<screen>
$ mkdir -p <replaceable>/my/sources</replaceable>
$ cd <replaceable>/my/sources</replaceable>
$ nix-env -i git
$ git clone git://github.com/NixOS/nixos.git
$ git clone git://github.com/NixOS/nixpkgs.git
</screen>

This will check out the latest NixOS sources to
<filename><replaceable>/my/sources</replaceable>/nixos</filename> and
the Nixpkgs sources to
<filename><replaceable>/my/sources</replaceable>/nixpkgs</filename>.
If you want to rebuild your system using your (modified) sources, you
need to tell <command>nixos-rebuild</command> about them using the
<option>-I</option> flag:

<screen>
$ nixos-rebuild switch -I <replaceable>/my/sources</replaceable>
</screen>

</para>

<para><command>nixos-rebuild</command> affects only the system profile.
To install packages to your user profile from expressions in
<replaceable>/my/sources</replaceable>, use
<command>nix-env -f <replaceable>/my/sources</replaceable>/nixpkgs</command>,
or change the default by replacing the symlink in
<filename>~/.nix-defexpr</filename>:

<screen>
$ rm -f ~/.nix-defexpr/channels
$ ln -s <replaceable>/my/sources</replaceable>/nixpkgs ~/.nix-defexpr/nixpkgs
</screen>

</para>

<para>You should not pass the base directory
<filename><replaceable>/my/sources</replaceable></filename>
to <command>nix-env</command>, as it will break after interpreting expressions
in <filename>nixos/</filename> as packages.</para>

</section>


<!--===============================================================-->

<section>

<title>Extending NixOS</title>

<para>NixOS is based on a modular system for declarative configuration.
  This system combines multiple <emphasis>modules</emphasis> to produce one
  configuration.  One of the module which compose your computer
  configuration is <filename>/etc/nixos/configuration.nix</filename>.  Other
  modules are available under NixOS <filename>modules</filename>
  directory</para>

<para>A module is a file which handles one specific part of the
  configuration.  This part of the configuration could correspond to
  hardware, a service, network settings, or preferences.  A module
  configuration does not have to handle everything from scratch, it can base
  its configuration on other configurations provided by other modules.  Thus
  a module can <emphasis>define</emphasis> options to setup its
  configuration, and it can also <emphasis>declare</emphasis> options to be
  fed by other modules.</para>

<!-- module syntax -->

<para xml:id="para-module-syn">A module is a file which contains a Nix
  expression.  This expression should be either an expression which gets
  evaluated into an attribute set or a function which returns an attribute
  set.</para>

<para>When the expression is a function, it should expect only one argument
  which is an attribute set containing an attribute
  named <varname>config</varname> and another attribute
  named <varname>pkgs</varname>.  The <varname>config</varname> attribute
  contains the result of the merge of all modules.  This attribute is
  evaluated lazily, such as any Nix expression.  For more details on how
  options are merged, see the details in <xref linkend="para-opt-decl"/>.
  The <varname>pkgs</varname> attribute
  contains <emphasis>nixpkgs</emphasis> attribute set of packages.  This
  attribute is necessary for declaring options.</para>

<example xml:id='module-syntax'><title>Usual module content</title>
<programlisting>
{ config, pkgs, ... }: <co xml:id='module-syntax-1' />

{
  imports =
    [ <co xml:id='module-syntax-2' />
    ];

  options = {
    <co xml:id='module-syntax-3' />
  };

  config = {
    <co xml:id='module-syntax-4' />
  };
}</programlisting>
</example>

<para><xref linkend='module-syntax' /> Illustrates
  a <emphasis>module</emphasis> skeleton.

<calloutlist>
  <callout arearefs='module-syntax-1'>
    <para>This line makes the current Nix expression a function.  This
    line can be omitted if there is no reference to <varname>pkgs</varname>
    and <varname>config</varname> inside the module.</para>
  </callout>

  <callout arearefs='module-syntax-2'>
    <para>This list is used to enumerate path to other modules which are
    declaring options used by the current module.  In NixOS, default modules
    are listed in the file <filename>modules/module-list.nix</filename>.
    The default modules don't need to be added in the import list.</para>
  </callout>

  <callout arearefs='module-syntax-3'>
    <para>This attribute set contains an attribute set of <emphasis>option
    declaration</emphasis>.</para>
  </callout>

  <callout arearefs='module-syntax-4'>
    <para>This attribute set contains an attribute set of <emphasis>option
    definitions</emphasis>.  If the module does not have any imported
    modules or any option declarations, then this attribute set can be used
    in place of its parent attribute set.  This is a common case for simple
    modules such
    as <filename>/etc/nixos/configuration.nix</filename>.</para>
  </callout>
</calloutlist>

</para>

<!-- option definitions -->

<para xml:id="para-opt-def">A module defines a configuration which would be
  interpreted by other modules.  To define a configuration, a module needs
  to provide option definitions.  An option definition is a simple
  attribute assignment.</para>

<para>Option definitions are made in a declarative manner.  Without
  properties, options will always be defined with the same value.  To
  introduce more flexibility in the system, option definitions are guarded
  by <emphasis>properties</emphasis>.</para>

<para>Properties are means to introduce conditional values inside option
  definitions.  This conditional values can be distinguished in two
  categories.  The condition which are local to the current configuration
  and conditions which are dependent on others configurations.  Local
  properties are <varname>mkIf</varname>
  and <varname>mkAssert</varname>.  Global properties
  are <varname>mkOverride</varname>, <varname>mkDefault</varname>
  and <varname>mkOrder</varname>.</para>

<para><varname>mkIf</varname> is used to remove the option definitions which
  are below it if the condition is evaluated to
  false.  <varname>mkAssert</varname> expects the condition to be evaluated
  to true otherwise it raises an error message.</para>

<para><varname>mkOverride</varname> is used to mask previous definitions if
  the current value has a lower mask number.  The mask value is 100 (default)
  for any option definition which does not use this property.
  Thus, <varname>mkDefault</varname> is just a short-cut with a higher mask
  (1000) than the default mask value.  This means that a module can set an
  option definition as a preference, and still let another module defining
  it with a different value without using any property.</para>

<para><varname>mkOrder</varname> is used to sort definitions based on the
  rank number.  The rank number will sort all options definitions before
  giving the sorted list of option definition to the merge function defined
  in the option declaration.  A lower rank will move the definition to the
  beginning and a higher rank will move the option toward the end.  The
  default rank is 100.</para>

<!-- option declarations -->

<para xml:id="para-opt-decl">A module may declare options which are used by
  other module to change the configuration provided by the current module.
  Changes to the option definitions are made with properties which are using
  values extracted from the result of the merge of all modules
  (the <varname>config</varname> argument).</para>

<para>The <varname>config</varname> argument reproduce the same hierarchy of
  all options declared in all modules.  For each option, the result of the
  option is available, it is either the default value or the merge of all
  definitions of the option.</para>

<para>Options are declared with the
  function <varname>pkgs.lib.mkOption</varname>.  This function expects an
  attribute set which at least provides a description.  A default value, an
  example, a type, a merge function and a post-process function can be
  added.</para>

<para>Types are used to provide a merge strategy for options and to ensure
  the type of each option definitions.  They are defined
  in <varname>pkgs.lib.types</varname>.</para>

<para>The merge function expects a list of option definitions and merge
  them to obtain one result of the same type.</para>

<para>The post-process function (named <varname>apply</varname>) takes the
  result of the merge or of the default value, and produce an output which
  could have a different type than the type expected by the option.</para>

<!-- end -->

<example xml:id='locate-example'><title>Locate Module Example</title>
<programlisting>
{ config, pkgs, ... }:

with pkgs.lib;

let
  cfg = config.services.locate;
  locatedb = "/var/cache/locatedb";
  logfile = "/var/log/updatedb";
  cmd =''root  updatedb --localuser=nobody --output=${locatedb} > ${logfile}'';
in

{
  imports = [ /etc/nixos/nixos/modules/services/scheduling/cron.nix ];

  options = {
    services.locate = {
      enable = mkOption {
        default = false;
        example = true;
        type = with types; bool;
        description = ''
          If enabled, NixOS will periodically update the database of
          files used by the <command>locate</command> command.
        '';
      };

      period = mkOption {
        default = "15 02 * * *";
        type = with types; uniq string;
        description = ''
          This option defines (in the format used by cron) when the
          locate database is updated.
          The default is to update at 02:15 (at night) every day.
        '';
      };
    };
  };

  config = mkIf cfg.enable {
    services.cron = {
      enable = true;
      systemCronJobs = "${cfg.period}  root ${cmd}";
    };
  };
}</programlisting>
</example>

<para><xref linkend='locate-example' /> illustrates a module which handles
  the regular update of the database which index all files on the file
  system.  This modules has option definitions to rely on the cron service
  to run the command at predefined dates.  In addition, this modules
  provides option declarations to enable the indexing and to use different
  period of time to run the indexing.  Properties are used to prevent
  ambiguous definitions of option (enable locate service and disable cron
  services) and to ensure that no options would be defined if the locate
  service is not enabled.</para>

</section>


<!--===============================================================-->

<section>

<title>Building specific parts of NixOS</title>

<para>

<screen>
$ nix-build /etc/nixos/nixos -A <replaceable>attr</replaceable></screen>

where <replaceable>attr</replaceable> is an attribute in
<filename>/etc/nixos/nixos/default.nix</filename>.  Attributes of interest include:

<variablelist>

  <varlistentry>
    <term><varname>config</varname></term>
    <listitem><para>The computer configuration generated from
    the <envar>NIXOS_CONFIG</envar> environment variable (default
    is <filename>/etc/nixos/configuration.nix</filename>) with the NixOS
    default set of modules.</para></listitem>
  </varlistentry>

  <varlistentry>
    <term><varname>system</varname></term>
    <listitem><para>The derivation which build your computer system.  It is
    built by the command <command>nixos-rebuild
    build</command></para></listitem>
  </varlistentry>

  <varlistentry>
    <term><varname>vm</varname></term>
    <listitem><para>The derivation which build your computer system inside a
    virtual machine.  It is built by the command <command>nixos-rebuild
    build-vm</command></para></listitem>
  </varlistentry>
</variablelist>

</para>

<para>
Most parts of NixOS can be built through the <varname>config</varname>
attribute set.  This attribute set allows you to have a view of the merged
option definitions and all its derivations.  Important derivations are store
inside the option <option>system.build</option> and can be listed with the
command <command>nix-instantiate --xml --eval-only /etc/nixos/nixos -A
config.system.build</command>
</para>

</section>


<!--===============================================================-->

<section>

<title>Building your own NixOS CD</title>

<para>Building a NixOS CD is as easy as configuring your own computer. The
idea is to use another module which will replace
your <filename>configuration.nix</filename> to configure the system that
would be installed on the CD.</para>

<para>Default CD/DVD configurations are available
inside <filename>nixos/modules/installer/cd-dvd</filename>.  To build them
you have to set <envar>NIXOS_CONFIG</envar> before
running <command>nix-build</command> to build the ISO.

<screen>
$ export NIXOS_CONFIG=/etc/nixos/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix
$ nix-build /etc/nixos/nixos -A config.system.build.isoImage</screen>

</para>

<para>Before burning your CD/DVD, you can check the content of the image by mounting anywhere like
suggested by the following command:

<screen>
$ mount -o loop -t iso9660 ./result/iso/cd.iso /mnt/iso</screen>

</para>

</section>


<!--===============================================================-->

<section>

<title>Testing/building the NixOS Manual</title>

<para>A quick way to see if your documentation improvements
or option descriptions look good:

<screen>
$ nix-build -A config.system.build.manual</screen>

</para>

</section>


<!--===============================================================-->

<section>

<title>Testing the installer</title>

<para>Building, burning, and 
booting from an installation CD is rather
tedious, so here is a quick way to see if the installer works
properly:

<screen>
$ export NIXOS_CONFIG=/etc/nixos/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix
$ nix-build /etc/nixos/nixos -A config.system.build.nixosInstall
$ dd if=/dev/zero of=diskimage seek=2G count=0 bs=1
$ yes | mke2fs -j diskimage
$ mount -o loop diskimage /mnt
$ ./result/bin/nixos-install</screen>

</para>

</section>



<!--===============================================================-->

<section>

<title>Testing the <literal>initrd</literal></title>

<para>A quick way to test whether the kernel and the initial ramdisk
boot correctly is to use QEMU’s <option>-kernel</option> and
<option>-initrd</option> options:

<screen>
$ nix-build /etc/nixos/nixos -A config.system.build.initialRamdisk -o initrd
$ nix-build /etc/nixos/nixos -A config.system.build.kernel -o kernel
$ qemu-system-x86_64 -kernel ./kernel/bzImage -initrd ./initrd/initrd -hda /dev/null
</screen>

</para>

</section>

<section>

  <title>Whole-system testing using virtual machines</title>

  <para>
    Complete NixOS GNU/Linux systems can be tested in virtual machines
    (VMs).  This makes it possible to test a system upgrade or
    configuration change before rebooting into it, using the
    <command>nixos-rebuild build-vm</command> or
    <command>nixos-rebuild build-vm-with-bootloader</command> command.
  </para>

  <para>
    <!-- The following is adapted from
         http://wiki.nixos.org/wiki/NixOS_VM_tests, by Eelco Dolstra. -->

    The <filename>tests/</filename> directory in the NixOS source tree
    contains several <emphasis>whole-system unit tests</emphasis>.
    These tests can be run<footnote><para>NixOS tests can be run both from
    NixOS and from a non-NixOS GNU/Linux distribution, provided the
    Nix package manager is installed.</para></footnote> from the NixOS
    source tree as follows:

<screen>
$ nix-build tests/ -A nfs.test
</screen>

    This performs an automated test of the NFS client and server
    functionality in the Linux kernel, including file locking
    semantics (e.g., whether locks are maintained across server
    crashes).  It will first build or download all the dependencies of
    the test (e.g., all packages needed to run a NixOS VM). The test
    is defined in <link
    xlink:href="https://nixos.org/repos/nix/nixos/trunk/tests/nfs.nix">
    <filename>tests/nfs.nix</filename></link>.  If the test succeeds,
    <command>nix-build</command> will place a symlink
    <filename>./result</filename> in the current directory pointing at
    the location in the Nix store of the test results (e.g.,
    screenshots, test reports, and so on).  In particular, a
    pretty-printed log of the test is written to
    <filename>log.html</filename>, which can be viewed using a web
    browser like this:

<screen>
$ icecat result/log.html
</screen>
  </para>

  <para>
    It is also possible to run the test environment interactively,
    allowing you to experiment with the VMs.  For example:

<screen>
$ nix-build tests/ -A nfs.driver
$ ./result/bin/nixos-run-vms
</screen>

    The script <command>nixos-run-vms</command> starts the three
    virtual machines defined in the NFS test using QEMU/KVM.  The root
    file system of the VMs is created on the fly and kept across VM
    restarts in
    <filename>./</filename><varname>hostname</varname><filename>.qcow2</filename>.
  </para>

  <para>
    Finally, the test itself can be run interactively.  This is
    particularly useful when developing or debugging a test:

<screen>
$ nix-build tests/ -A nfs.driver
$ ./result/bin/nixos-test-driver
starting VDE switch for network 1
&gt;
</screen>

    Perl statements can now be typed in to start or manipulate the
    VMs:

<screen>
&gt; startAll;
(the VMs start booting)
&gt; $server-&gt;waitForJob("nfs-kernel-nfsd");
&gt; $client1-&gt;succeed("flock -x /data/lock -c 'sleep 100000' &amp;");
&gt; $client2-&gt;fail("flock -n -s /data/lock true");
&gt; $client1-&gt;shutdown;
(this releases client1's lock)
&gt; $client2-&gt;succeed("flock -n -s /data/lock true");
</screen>

    The function <command>testScript</command> executes the entire
    test script and drops you back into the test driver command line
    upon its completion.  This allows you to inspect the state of the
    VMs after the test (e.g. to debug the test script).
  </para>

  <para>
    This and other tests are continuously run on <link
    xlink:href="http://hydra.nixos.org/jobset/nixos/trunk/with-status">the
    Hydra instance at <literal>nixos.org</literal></link>, which
    allows developers to be notified of any regressions introduced by
    a NixOS or Nixpkgs change.
  </para>

  <para>
    The actual Nix programming interface to VM testing is in NixOS,
    under <link
    xlink:href="https://nixos.org/repos/nix/nixos/trunk/lib/testing.nix">
    <filename>lib/testing.nix</filename></link>.  This file defines a
    function which takes an attribute set containing a
    <literal>nixpkgs</literal> attribute (the path to a Nixpkgs
    checkout), and a <literal>system</literal> attribute (the system
    type).  It returns an attribute set containing several utility
    functions, among which the main entry point is
    <literal>makeTest</literal>.
  </para>

  <para>
    The <literal>makeTest</literal> function takes a function similar to
    that found in <link
    xlink:href="https://nixos.org/repos/nix/nixos/trunk/tests/nfs.nix">
    <filename>tests/nfs.nix</filename></link> (discussed above).  It
    returns an attribute set containing (among others):

    <variablelist>

      <varlistentry>
	<term><varname>test</varname></term>
	<listitem><para>A derivation containing the test log as an HTML file,
	as seen above, suitable for presentation in the Hydra continuous
	build system.</para></listitem>
      </varlistentry>

      <varlistentry>
	<term><varname>report</varname></term>
	<listitem><para>A derivation containing a code coverage report, with
	meta-data suitable for Hydra.</para></listitem>
      </varlistentry>

      <varlistentry>
	<term><varname>driver</varname></term>
	<listitem><para>A derivation containing scripts to run the VM test or
	interact with the VM network interactively, as seen above.</para>
	</listitem>
      </varlistentry>

    </variablelist>
  </para>
    

</section>

</chapter>