Keeping the VM state test across several run sometimes lead to subtle
and hard to spot errors in practice. We delete the VM state which
contains (among other things) the qcow volume.
We also introduce a -K (--keep-vm-state) flag making VM state to
persist after the test run. This flag makes test-driver.py to match
its previous behaviour.
Turns out, on smaller images (~800MiB uncompressed sdcard image size),
the current fudge factor is way too small to even get the system to the
phase where it can resize itself.
I first tried with 1.05, but it wasn't enough.
xchg is advertised as a bidirectional exchange dir, but file content
transfer from host to VM fails due to caching:
If a file is read in the VM and then modified on the host, subsequent
re-reads in the VM can yield old, cached data.
This is caused by the use of 9p's cache=loose mode that is explicitly
meant for read-only mounts.
9p doesn't provide any suitable cache modes, so fix this by disabling
caching.
Also, remove a now unnecessary sync in the test driver.
These syncs have the goal to transfer host filesystem changes to the VM,
but they have no effect because 1) syncing in the VM can't possibly pull
in host data and 2) 9p is accessing the host filesystem on the cached
layer anyways, so even syncing on the host would have no effect in the
VM.
The test harness provides the commands it wishes to run in Bourne
syntax. This fails if the user uses a different shell. For example,
with fish:
machine.wait_for_unit("graphical-session.target", "alice")
machine # fish: Unsupported use of '='. To run '-u`' with a modified environment, please use 'env XDG_RUNTIME_DIR=/run/user/`id -u`…'
machine # XDG_RUNTIME_DIR=/run/user/`id -u` systemctl --user --no-pager show "graphical-session.target"
machine # ^
machine # [ 16.329957] su[1077]: pam_unix(su:session): session closed for user alice
error: retrieving systemctl info for unit "graphical-session.target" under user "alice" failed with exit code 127
This completes the removal of the nested log feature, which previously
got removed from Nix, Hydra, stdenv and GNU Make. In particular, this
means that the output of VM builds no longer contains a copy of
jQuery.
If a program (e.g. nixos-install) writes more than 1000 lines to
stderr during execute(), then process_serial_output() deadlocks
waiting for the queue to be processed. So use an unbounded queue
instead.
We should probably get rid of the structured log output (log.xml),
since then we don't need the log queue anymore.
This avoids a possible surprise if the user is using `nixpkgs.system`
and `nesting.children`. `nesting.children` is expected to ignore all
parent configuration so we shouldn't propagate the user-facing option
`nixpkgs.system`. To avoid doing so, we introduce a new internal
option for holding the value passed to eval-config.nix, and use that
when recursing for nesting.
Most VM tests have been migrated to use the python test driver
(introduced in #71684), the migration is tracked in #72828 (which also
thankfully uncovered and fixed many currently broken tests)
While increasing the acceptance and adoption of NixOS integration tests
by using a more popular language, there was also nobody willing to do
larger refactors in the currently very convoluted test infrastructure.
We plan to remove the perl infrastructure between the 20.03 and 20.09
release, to be able to do these refactorings.
Some people might be using Perl tests in their internal CI, so print a
warning for 20.03, and give users time to move to the python testing
infrastructure.
According to https://repology.org/repository/nix_unstable/problems, we have a
lot of packages that have http links that redirect to https as their homepage.
This commit updates all these packages to use the https links as their
homepage.
The following script was used to make these updates:
```
curl https://repology.org/api/v1/repository/nix_unstable/problems \
| jq '.[] | .problem' -r \
| rg 'Homepage link "(.+)" is a permanent redirect to "(.+)" and should be updated' --replace 's@$1@$2@' \
| sort | uniq > script.sed
find -name '*.nix' | xargs -P4 -- sed -f script.sed -i
```
The docstring says it uses a directory shared among all vms, although
that doesn't seem necessary for the functionality. However, it does need
to be consistent between the guest and host.
The codec format 'unicode_escape' was introduced in 52ee102 to handle
undecodable bytes in boot menus.
This made the problem worse as unicode chars outside of iso-8859-1
produce garbled output and valid utf-8 strings (such as "\x" ) trigger
decoding errors.
Fix this by using the default 'utf-8' codec and by explicitly ignoring
decoding errors.
This changes the python test driver to match the behavior of the perl
test driver. I.e. the directory mounted into /tmp/shared should be the
same for all machines.
This probably fixes many tests, but I found this while investigating
failures in nixos/tests/ceph-multi-node.nix.
While it's a good idea to automate the linting of the python code used
for our tests, I think that it can be quite distracting when hacking on
a NixOS test.
I figured that it might be more convenient to add an option as a
shortcut for this to avoid that everyone needs to dig into the test
driver again.
The upstream session files display managers use have no concept of sessions being composed from
desktop manager and window manager. To be able to set upstream session files as default
session, we need a single option. Having two different ways to set default session would be confusing,
though, so we decided to deprecate the old method.
We also created separate script for each session, just like we already had a separate desktop
file for each one, and started using displayManager.sessionPackages mechanism to make the
session handling more uniform.
When using `documentation.nixos.includeAllModules = true;` with external
modules, the string context might contain dependencies to derivations
and so `toFile` refuses to evaluate;
```
error: in 'toFile': the file 'options.xml' cannot refer to derivation outputs, at
[...]/nixpkgs/nixos/lib/make-options-doc/default.nix:89:16
```
This is not an issue when using `writeText` (instead of manually
stripping the context).
The SLIM project is abandoned and their last release was in 2013.
Because of this it poses a security risk to systems, no one is working
on it or picked up maintenance. It also lacks compatibility with systemd
and logind sessions. For users, there liikely isn't anything like slim
that's as lightweight in terms of dependencies.
we previously immediately returned the first commands output, and didn't
execute any of the other commands.
Now, return the last commands output.
This should be documented in the method docstring.
Condition seems to be inverted. Crash and shutdown only make sense, when
the machine is booted; i.e. we return immediately otherwise.
In the Perl test driver this is:
return unless $self->{booted};
This reverts commit e9bf955fd6. We use
nixos-install to ensure that make-disk-image produces the same result
as a regular installation (9802da517f)
and to reduce code duplication. If there is something broken in
nixos-install, it should be fixed there.
Introduce new functions which allows modules to define options where,
if the input is an attrset and the output is JSON, the user can define
arbitrary secrets.
Because the copy process inside the VM does not reliably
give "No space" error message leaving the user wondering what
went wrong:
unable to create directory /mnt/0000fe01///nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share/locale/ro: Success
[ 5.462365] reboot: Restarting system
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share/locale/ro, aborting
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share/locale, aborting
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7/share, aborting
error processing entry /build/root/nix/store/yknzxx7w2ck9p30k81gpi5yfjlrq41lr-libsecret-0.18.7, aborting
error processing entry /build/root/nix/store, aborting
error processing entry /build/root/nix, aborting
builder for '/nix/store/fsdvqxq92iai7f3w8wcsncgfwag7cj2l-libvirtd-ssh-image.drv' failed with exit code 228
Motivation is to support other repositories containing nixos
modules that would like to generate options documentation:
- nix-darwin
- private repos
- arion
- ??
When IPXE tests were added, an option was added for configuring only
the frontend, and the backend configuration was dropped entirely. This
caused most installer tests to fail.
We differentiate between modules and baseModules in the
VM builder for NixOS tests. This way, nesting.children, eventhough
it doesn't inherit from parent, still has enough config to
actually complete the test. Otherwise, the qemu modules
would not be loaded, for example, and a nesting.children
statement would not evaluate.
Before this change `man 5 configuration.nix` would only show options of modules in
the `baseModules` set, which consists only of the list of modules in
`nixos/modules/module-list.nix`
With this change applied and `documentation.nixos.includeAllModules` option enabled
all modules included in `configuration.nix` file will be used instead.
This makes configurations with custom modules self-documenting. It also means
that importing non-`baseModules` modules like `gce.nix` or `azure.nix`
will make their documentation available in `man 5 configuration.nix`.
`documentation.nixos.includeAllModules` is currently set to `false` by
default as enabling it usually uncovers bugs and prevents evaluation.
It should be set to `true` in a release or two.
This was originally implemented in #47177, edited for more configurability,
documented and rebased onto master by @oxij.
Fixes#5185856e12aae54 ends up passing config to pkgs. Unfortunately this might be null and pkgs/top-level/default.nix assumes it is an attrset. To fix this, we just make the default for config = {}. Thanks to @kristoff3r for tracking this down.
/cc @domenkozar
See #49441 for an earlier attempt, which was subsequently reverted. I am
assuming that doubling the time will be sufficient if the machine is
overloaded since so many of the tests already pass at 5 minutes, while
still not holding back failures for needlessly long.
cleanSource does not appear to work correctly in this case. The path
does not get coerced to a string, resulting in a dangling symlink
produced in channel.nix. Not sure why, but this
seems to fix it.
Fixes#51025.
/cc @elvishjericco
Since 113a6b9325 the test driver
explicitly ensures if the node names won't break the resulting Perl
script at runtime. This slightly improves the correctness of the error
message.
These names are referenced by Perl variables inside the testing
frameworks which don't allow chars like `-` as character inside. An exemplary
expression may look like this:
```
{
x11-vm = {
services.xserver.enable = true;
};
}
```
This expression evaluates, e.g. when running `nixos-build-vms`, but when
trying to run `./result/bin/nixos-run-vms`, an error like this occurs:
```
starting VDE switch for network 1
running the VM test script
error: Can't modify subtraction (-) in scalar assignment at (eval 17) line 1, at EOF
Bareword "test" not allowed while "strict subs" in use at (eval 17) line 1.
Can't modify subtraction (-) in scalar assignment at (eval 17) line 1, at EOF
Bareword "test" not allowed while "strict subs" in use at (eval 17) line 1.
vde_switch: EOF on stdin, cleaning up and exiting
cleaning up
```
This can be very confusing for beginners, this change breaks evaluation
if such names are used for machines.
This is to try and squeeze more lost space from the image, so that hydra
starts building it again.
The fsck previous to the resize2fs is required so resize2fs works.
The one afterwards is a sanity check.
Using `-M` from resize2fs will not give much saved space due to a known
(in the manual) issue.
```
[samueldr@aarch64:~/nixpkgs]$ ls -lh result-*/*/*.img
-r--r--r-- 1 root root 2.2G Jan 1 1970 result-original/sd-image/nixos-sd-image-18.09.git.a7fd431-aarch64-linux.img
-r--r--r-- 1 root root 2.1G Jan 1 1970 result-M/sd-image/nixos-sd-image-18.09.git.a7fd431-aarch64-linux.img
-r--r--r-- 1 root root 1.9G Jan 1 1970 result-slimmed/sd-image/nixos-sd-image-18.09.git.a7fd431-aarch64-linux.img
```
```
[samueldr@aarch64:~/nixpkgs]$ nix path-info -S ./result-original
/nix/store/c8k9n78gylx293rjh762fr05a069kxp2-nixos-sd-image-18.09.git.a7fd431-aarch64-linux.img 3844125000
[samueldr@aarch64:~/nixpkgs]$ nix path-info -S ./result-slimmed
/nix/store/962238skj5mnzhrsmjy23dyzmxk77sp4-nixos-sd-image-18.09.git.a7fd431-aarch64-linux.img 3447473208
```
