forked from mirrors/nixpkgs
Update section on writing tests
This commit is contained in:
Eelco Dolstra
parent
29027fd1e1
commit
e1a1146690
|
|
@ -1,5 +1,6 @@
|
|||
<chapter xmlns="http://docbook.org/ns/docbook"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||||
xml:id="ch-development">
|
||||
|
||||
<title>Development</title>
|
||||
|
||||
|
|
@ -9,7 +10,7 @@ NixOS.</para>
|
|||
|
||||
<!--===============================================================-->
|
||||
|
||||
<section>
|
||||
<section xml:id="sec-getting-sources">
|
||||
|
||||
<title>Getting the sources</title>
|
||||
|
||||
|
|
@ -74,7 +75,7 @@ in <filename>nixos/</filename> as packages.</para>
|
|||
|
||||
<!--===============================================================-->
|
||||
|
||||
<section>
|
||||
<section xml:id="sec-writing-modules">
|
||||
|
||||
<title>Writing NixOS modules</title>
|
||||
|
||||
|
|
@ -579,7 +580,7 @@ systemd.services.dhcpcd =
|
|||
|
||||
<!--===============================================================-->
|
||||
|
||||
<section>
|
||||
<section xml:id="sec-building-parts">
|
||||
|
||||
<title>Building specific parts of NixOS</title>
|
||||
|
||||
|
|
@ -692,7 +693,7 @@ $ systemctl start tmp-httpd.service
|
|||
|
||||
<!--===============================================================-->
|
||||
|
||||
<section>
|
||||
<section xml:id="sec-building-cd">
|
||||
|
||||
<title>Building your own NixOS CD</title>
|
||||
|
||||
|
|
@ -748,57 +749,310 @@ $ ./result/bin/nixos-install</screen>
|
|||
|
||||
<!--===============================================================-->
|
||||
|
||||
<section><title>Whole-system testing using virtual machines</title>
|
||||
<section xml:id="sec-nixos-tests">
|
||||
|
||||
<para>Complete NixOS GNU/Linux systems can be tested in virtual
|
||||
machines (VMs). This makes it possible to test a system upgrade or
|
||||
configuration change before rebooting into it, using the
|
||||
<command>nixos-rebuild build-vm</command> or <command>nixos-rebuild
|
||||
build-vm-with-bootloader</command> command.</para>
|
||||
<title>NixOS tests</title>
|
||||
|
||||
<!-- The following is adapted from
|
||||
http://wiki.nixos.org/wiki/NixOS_VM_tests, by Eelco Dolstra. -->
|
||||
<para>The <filename>tests/</filename> directory in the NixOS source
|
||||
tree contains several <emphasis>whole-system unit tests</emphasis>.
|
||||
These tests can be run<footnote><para>NixOS tests can be run both from
|
||||
NixOS and from a non-NixOS GNU/Linux distribution, provided the Nix
|
||||
package manager is installed.</para></footnote> from the NixOS source
|
||||
tree as follows:
|
||||
<para>When you add some feature to NixOS, you should write a test for
|
||||
it. NixOS tests are kept in the directory <filename
|
||||
xlink:href="https://github.com/NixOS/nixpkgs/tree/master/nixos/tests">nixos/tests</filename>,
|
||||
and are executed (using Nix) by a testing framework that automatically
|
||||
starts one or more virtual machines containing the NixOS system(s)
|
||||
required for the test.</para>
|
||||
|
||||
<simplesect><title>Writing tests</title>
|
||||
|
||||
<para>A NixOS test is a Nix expression that has the following structure:
|
||||
|
||||
<programlisting>
|
||||
import ./make-test.nix {
|
||||
|
||||
# Either the configuration of a single machine:
|
||||
machine =
|
||||
{ config, pkgs, ... }:
|
||||
{ <replaceable>configuration…</replaceable>
|
||||
};
|
||||
|
||||
# Or a set of machines:
|
||||
nodes =
|
||||
{ <replaceable>machine1</replaceable> =
|
||||
{ config, pkgs, ... }: { <replaceable>…</replaceable> };
|
||||
<replaceable>machine2</replaceable> =
|
||||
{ config, pkgs, ... }: { <replaceable>…</replaceable> };
|
||||
…
|
||||
};
|
||||
|
||||
testScript =
|
||||
''
|
||||
<replaceable>Perl code…</replaceable>
|
||||
'';
|
||||
}
|
||||
</programlisting>
|
||||
|
||||
The attribute <literal>testScript</literal> is a bit of Perl code that
|
||||
executes the test (described below). During the test, it will start
|
||||
one or more virtual machines, the configuration of which is described
|
||||
by the attribute <literal>machine</literal> (if you need only one
|
||||
machine in your test) or by the attribute <literal>nodes</literal> (if
|
||||
you need multiple machines). For instance, <filename
|
||||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix">login.nix</filename>
|
||||
only needs a single machine to test whether users can log in on the
|
||||
virtual console, whether device ownership is correctly maintained when
|
||||
switching between consoles, and so on. On the other hand, <filename
|
||||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs.nix">nfs.nix</filename>,
|
||||
which tests NFS client and server functionality in the Linux kernel
|
||||
(including whether locks are maintained across server crashes),
|
||||
requires three machines: a server and two clients.</para>
|
||||
|
||||
<para>There are a few special NixOS configuration options for test
|
||||
VMs:
|
||||
|
||||
<!-- FIXME: would be nice to generate this automatically. -->
|
||||
|
||||
<variablelist>
|
||||
|
||||
<varlistentry>
|
||||
<term><option>virtualisation.memorySize</option></term>
|
||||
<listitem><para>The memory of the VM in
|
||||
megabytes.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><option>virtualisation.vlans</option></term>
|
||||
<listitem><para>The virtual networks to which the VM is
|
||||
connected. See <filename
|
||||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix">nat.nix</filename>
|
||||
for an example.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><option>virtualisation.writableStore</option></term>
|
||||
<listitem><para>By default, the Nix store in the VM is not
|
||||
writable. If you enable this option, a writable union file system
|
||||
is mounted on top of the Nix store to make it appear
|
||||
writable. This is necessary for tests that run Nix operations that
|
||||
modify the store.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
</variablelist>
|
||||
|
||||
For more options, see the module <filename
|
||||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix">qemu-vm.nix</filename>.</para>
|
||||
|
||||
<para>The test script is a sequence of Perl statements that perform
|
||||
various actions, such as starting VMs, executing commands in the VMs,
|
||||
and so on. Each virtual machine is represented as an object stored in
|
||||
the variable <literal>$<replaceable>name</replaceable></literal>,
|
||||
where <replaceable>name</replaceable> is the identifier of the machine
|
||||
(which is just <literal>machine</literal> if you didn’t specify
|
||||
multiple machines using the <literal>nodes</literal> attribute). For
|
||||
instance, the following starts the machine, waits until it has
|
||||
finished booting, then executes a command and checks that the output
|
||||
is more-or-less correct:
|
||||
|
||||
<programlisting>
|
||||
$machine->start;
|
||||
$machine->waitForUnit("default.target");
|
||||
$machine->succeed("uname") =~ /Linux/;
|
||||
</programlisting>
|
||||
|
||||
The first line is actually unnecessary; machines are implicitly
|
||||
started when you first execute an action on them (such as
|
||||
<literal>waitForUnit</literal> or <literal>succeed</literal>). If you
|
||||
have multiple machines, you can speed up the test by starting them in
|
||||
parallel:
|
||||
|
||||
<programlisting>
|
||||
startAll;
|
||||
</programlisting>
|
||||
|
||||
</para>
|
||||
|
||||
<para>The following methods are available on machine objects:
|
||||
|
||||
<variablelist>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>start</methodname></term>
|
||||
<listitem><para>Start the virtual machine. This method is
|
||||
asynchronous — it does not wait for the machine to finish
|
||||
booting.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>shutdown</methodname></term>
|
||||
<listitem><para>Shut down the machine, waiting for the VM to
|
||||
exit.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>crash</methodname></term>
|
||||
<listitem><para>Simulate a sudden power failure, by telling the VM
|
||||
to exit immediately.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>block</methodname></term>
|
||||
<listitem><para>Simulate unplugging the Ethernet cable that
|
||||
connects the machine to the other machines.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>unblock</methodname></term>
|
||||
<listitem><para>Undo the effect of
|
||||
<methodname>block</methodname>.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>screenshot</methodname></term>
|
||||
<listitem><para>Take a picture of the display of the virtual
|
||||
machine, in PNG format. The screenshot is linked from the HTML
|
||||
log.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>sendMonitorCommand</methodname></term>
|
||||
<listitem><para>Send a command to the QEMU monitor. This is rarely
|
||||
used, but allows doing stuff such as attaching virtual USB disks
|
||||
to a running machine.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>sendKeys</methodname></term>
|
||||
<listitem><para>Simulate pressing keys on the virtual keyboard,
|
||||
e.g., <literal>sendKeys("ctrl-alt-delete")</literal>.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>sendChars</methodname></term>
|
||||
<listitem><para>Simulate typing a sequence of characters on the
|
||||
virtual keyboard, e.g., <literal>sendKeys("foobar\n")</literal>
|
||||
will type the string <literal>foobar</literal> followed by the
|
||||
Enter key.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>execute</methodname></term>
|
||||
<listitem><para>Execute a shell command, returning a list
|
||||
<literal>(<replaceable>status</replaceable>,
|
||||
<replaceable>stdout</replaceable>)</literal>.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>succeed</methodname></term>
|
||||
<listitem><para>Execute a shell command, raising an exception if
|
||||
the exit status is not zero, otherwise returning the standard
|
||||
output.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>fail</methodname></term>
|
||||
<listitem><para>Like <methodname>succeed</methodname>, but raising
|
||||
an exception if the command returns a zero status.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>waitUntilSucceeds</methodname></term>
|
||||
<listitem><para>Repeat a shell command with 1-second intervals
|
||||
until it succeeds.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>waitUntilFails</methodname></term>
|
||||
<listitem><para>Repeat a shell command with 1-second intervals
|
||||
until it fails.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>waitForUnit</methodname></term>
|
||||
<listitem><para>Wait until the specified systemd unit has reached
|
||||
the “active” state.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>waitForFile</methodname></term>
|
||||
<listitem><para>Wait until the specified file
|
||||
exists.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>waitForOpenPort</methodname></term>
|
||||
<listitem><para>Wait until a process is listening on the given TCP
|
||||
port (on <literal>localhost</literal>, at least).</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>waitForClosedPort</methodname></term>
|
||||
<listitem><para>Wait until nobody is listening on the given TCP
|
||||
port.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>waitForX</methodname></term>
|
||||
<listitem><para>Wait until the X11 server is accepting
|
||||
connections.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><methodname>waitForWindow</methodname></term>
|
||||
<listitem><para>Wait until an X11 window has appeared whose name
|
||||
matches the given regular expression, e.g.,
|
||||
<literal>waitForWindow(qr/Terminal/)</literal>.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
</variablelist>
|
||||
|
||||
</para>
|
||||
|
||||
</simplesect>
|
||||
|
||||
|
||||
<simplesect><title>Running tests</title>
|
||||
|
||||
<para>You can run tests using <command>nix-build</command>. For
|
||||
example, to run the test <filename
|
||||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix">login.nix</filename>,
|
||||
you just do:
|
||||
|
||||
<screen>
|
||||
$ nix-build tests/ -A nfs.test
|
||||
$ nix-build '<nixpkgs/nixos/tests/login.nix>'
|
||||
</screen>
|
||||
|
||||
This performs an automated test of the NFS client and server
|
||||
functionality in the Linux kernel, including file locking semantics
|
||||
(e.g., whether locks are maintained across server crashes). It will
|
||||
first build or download all the dependencies of the test (e.g., all
|
||||
packages needed to run a NixOS VM). The test is defined in <link
|
||||
xlink:href="https://nixos.org/repos/nix/nixos/trunk/tests/nfs.nix">
|
||||
<filename>tests/nfs.nix</filename></link>. If the test succeeds,
|
||||
<command>nix-build</command> will place a symlink
|
||||
<filename>./result</filename> in the current directory pointing at the
|
||||
location in the Nix store of the test results (e.g., screenshots, test
|
||||
reports, and so on). In particular, a pretty-printed log of the test
|
||||
is written to <filename>log.html</filename>, which can be viewed using
|
||||
a web browser like this:
|
||||
or, if you don’t want to rely on <envar>NIX_PATH</envar>:
|
||||
|
||||
<screen>
|
||||
$ cd /my/nixpkgs/nixos/tests
|
||||
$ nix-build login.nix
|
||||
…
|
||||
running the VM test script
|
||||
machine: QEMU running (pid 8841)
|
||||
…
|
||||
6 out of 6 tests succeeded
|
||||
</screen>
|
||||
|
||||
After building/downloading all required dependencies, this will
|
||||
perform a build that starts a QEMU/KVM virtual machine containing a
|
||||
NixOS system. The virtual machine mounts the Nix store of the host;
|
||||
this makes VM creation very fast, as no disk image needs to be
|
||||
created. Afterwards, you can view a pretty-printed log of the test:
|
||||
|
||||
<screen>
|
||||
$ firefox result/log.html
|
||||
</screen>
|
||||
|
||||
</para>
|
||||
|
||||
<para>It is also possible to run the test environment interactively,
|
||||
allowing you to experiment with the VMs. For example:
|
||||
|
||||
<screen>
|
||||
$ nix-build tests/ -A nfs.driver
|
||||
$ nix-build login.nix -A driver
|
||||
$ ./result/bin/nixos-run-vms
|
||||
</screen>
|
||||
|
||||
The script <command>nixos-run-vms</command> starts the three virtual
|
||||
machines defined in the NFS test using QEMU/KVM. The root file system
|
||||
of the VMs is created on the fly and kept across VM restarts in
|
||||
The script <command>nixos-run-vms</command> starts the virtual
|
||||
machines defined by test. The root file system of the VMs is created
|
||||
on the fly and kept across VM restarts in
|
||||
<filename>./</filename><varname>hostname</varname><filename>.qcow2</filename>.</para>
|
||||
|
||||
<para>Finally, the test itself can be run interactively. This is
|
||||
|
|
@ -811,17 +1065,11 @@ starting VDE switch for network 1
|
|||
>
|
||||
</screen>
|
||||
|
||||
Perl statements can now be typed in to start or manipulate the VMs:
|
||||
You can then take any Perl statement, e.g.
|
||||
|
||||
<screen>
|
||||
> startAll;
|
||||
(the VMs start booting)
|
||||
> $server->waitForJob("nfs-kernel-nfsd");
|
||||
> $client1->succeed("flock -x /data/lock -c 'sleep 100000' &");
|
||||
> $client2->fail("flock -n -s /data/lock true");
|
||||
> $client1->shutdown;
|
||||
(this releases client1's lock)
|
||||
> $client2->succeed("flock -n -s /data/lock true");
|
||||
> startAll
|
||||
> $machine->succeed("touch /tmp/foo")
|
||||
</screen>
|
||||
|
||||
The function <command>testScript</command> executes the entire test
|
||||
|
|
@ -829,54 +1077,7 @@ script and drops you back into the test driver command line upon its
|
|||
completion. This allows you to inspect the state of the VMs after the
|
||||
test (e.g. to debug the test script).</para>
|
||||
|
||||
<para>This and other tests are continuously run on <link
|
||||
xlink:href="http://hydra.nixos.org/jobset/nixos/trunk">the Hydra
|
||||
instance at <literal>nixos.org</literal></link>, which allows
|
||||
developers to be notified of any regressions introduced by a NixOS or
|
||||
Nixpkgs change.</para>
|
||||
|
||||
<para>The actual Nix programming interface to VM testing is in NixOS,
|
||||
under <link
|
||||
xlink:href="https://nixos.org/repos/nix/nixos/trunk/lib/testing.nix">
|
||||
<filename>lib/testing.nix</filename></link>. This file defines a
|
||||
function which takes an attribute set containing a
|
||||
<literal>nixpkgs</literal> attribute (the path to a Nixpkgs checkout),
|
||||
and a <literal>system</literal> attribute (the system type). It
|
||||
returns an attribute set containing several utility functions, among
|
||||
which the main entry point is <literal>makeTest</literal>.
|
||||
</para>
|
||||
|
||||
<para>The <literal>makeTest</literal> function takes a function
|
||||
similar to that found in <link
|
||||
xlink:href="https://nixos.org/repos/nix/nixos/trunk/tests/nfs.nix">
|
||||
<filename>tests/nfs.nix</filename></link> (discussed above). It
|
||||
returns an attribute set containing (among others):
|
||||
|
||||
<variablelist>
|
||||
|
||||
<varlistentry>
|
||||
<term><varname>test</varname></term>
|
||||
<listitem><para>A derivation containing the test log as an HTML
|
||||
file, as seen above, suitable for presentation in the Hydra
|
||||
continuous build system.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><varname>report</varname></term>
|
||||
<listitem><para>A derivation containing a code coverage report, with
|
||||
meta-data suitable for Hydra.</para></listitem>
|
||||
</varlistentry>
|
||||
|
||||
<varlistentry>
|
||||
<term><varname>driver</varname></term>
|
||||
<listitem><para>A derivation containing scripts to run the VM test or
|
||||
interact with the VM network interactively, as seen above.</para>
|
||||
</listitem>
|
||||
</varlistentry>
|
||||
|
||||
</variablelist>
|
||||
|
||||
</para>
|
||||
</simplesect>
|
||||
|
||||
</section>
|
||||
|
||||
|
|
|
|||
|
|
@ -60,7 +60,7 @@
|
|||
<xi:include href="release-notes.xml" />
|
||||
|
||||
<appendix xml:id="ch-options">
|
||||
<title>List of options</title>
|
||||
<title>Configuration options</title>
|
||||
<xi:include href="options-db.xml" />
|
||||
</appendix>
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue