This seems to be the root cause of the random page allocation failures
and @wizeman did a very good job on not only finding the root problem
but also giving a detailed explanation of it in #10828.
Here is an excerpt:
The problem here is that the kernel is trying to allocate a contiguous
section of 2^7=128 pages, which is 512 KB. This is way too much:
kernel pages tend to get fragmented over time and kernel developers
often go to great lengths to try allocating at most only 1 contiguous
page at a time whenever they can.
From the error message, it looks like the culprit is unionfs, but this
is misleading: unionfs is the name of the userspace process that was
running when the system ran out of memory, but it wasn't unionfs who
was allocating the memory: it was the kernel; specifically it was the
v9fs_dir_readdir_dotl() function, which is the code for handling the
readdir() function in the 9p filesystem (the filesystem that is used
to share a directory structure between a qemu host and its VM).
If you look at the code, here's what it's doing at the moment it tries
to allocate memory:
buflen = fid->clnt->msize - P9_IOHDRSZ;
rdir = v9fs_alloc_rdir_buf(file, buflen);
If you look into v9fs_alloc_rdir_buf(), you will see that it will try
to allocate a contiguous buffer of memory (using kzalloc(), which is a
wrapper around kmalloc()) of size buflen + 8 bytes or so.
So in reality, this code actually allocates a buffer of size
proportional to fid->clnt->msize. What is this msize? If you follow
the definition of the structures, you will see that it's the
negotiated buffer transfer size between 9p client and 9p server. On
the client side, it can be controlled with the msize mount option.
What this all means is that, the reason for running out of memory is
that the code (which we can't easily change) tries to allocate a
contiguous buffer of size more or less equal to "negotiated 9p
protocol buffer size", which seems to be way too big (in our NixOS
tests, at least).
After that initial finding, @lethalman tested the gnome3 gdm test
without setting the msize parameter at all and it seems to have resolved
the problem.
The reason why I'm committing this without testing against all of the
NixOS VM test is basically that I think we can only go better but not
worse than the current state.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Previously is was assumed that bash was in the path when calling the
environment setup script. This changes all of the references of bash to
be absolute paths so that the user doesn't have to worry about the
environment they call it with.
Otherwise, if the upstream mirror changes (rather than deletes) a
file, then tarballs.nixos.org won't be used even if it has a copy of
the original file, and so we'll get a hash mismatch.
Emacs packages are commonly distributed as single .el files. This
unpackCmd handles them correctly and sets up sourceRoot. Other sources
are treated in the default manner.
If "fetcher" is a string, then Nix will execute it with bash already, so
the additional bash argument in that string was redundant and apparently
causes trouble on non-Linux platforms.
Hopefully fixes https://github.com/NixOS/nixpkgs/issues/11496.
The list we had before contained a lot of junk, i.e. sites that were no
longer online or no longer in sync. The new list of sites comes from
https://gnupg.org/download/index.html.
It turns out that cargo implicitly depends on rustc at runtime: even
`cargo help` will fail if rustc is not in the PATH.
This means that we need to wrap the cargo binary to add rustc to PATH.
However, I have opted into doing something slightly unusual: instead of
tying down a specific cargo to use a specific rustc (i.e., wrap cargo so
that "${rustc}/bin" is prefixed into PATH), instead I'm adding the rustc
used to build cargo as a fallback rust compiler (i.e., wrap cargo so
that "${rustc}/bin" is suffixed into PATH). This means that cargo will
prefer to use a rust compiler that is in the default path, but fallback
into the one used to build cargo only if there wasn't any rust compiler
in the default path.
The reason I'm doing this is that otherwise it could cause unexpected
effects. For example, if you had a build environment with the
rustcMaster and cargo derivations, you would expect cargo to use
rustcMaster to compile your project (since rustcMaster would be the only
compiler available in $PATH), but this wouldn't happen if we tied down
cargo to use the rustc that was used to compile it (because the default
cargo derivation gets compiled with the stable rust compiler).
That said, I have slightly modified makeRustPlatform so that a rust
platform will always use the rust compiler that was used to build cargo,
because this prevents mistakenly depending on two different versions of
the rust compiler (stable and unstable) in the same rust platform,
something which is usually undesirable.
Fixes#11053
The fetch-cargo-deps script is written in bash syntax, but it
erroneously ran under the /bin/sh interpreter.
This wasn't noticed because /bin/sh is actually bash in NixOS, but on
some other systems this is not true.
While debugging an issue with running NixOps tests, I found out that the
output from debClosureGenerator is not deterministic.
The reason behind this is the way how Provides and Replaces fields are
handled. I haven't yet found out what's the exact issue, but so far
packages "Provides" are more or less picked at random.
So, running the NixOps Hetzner tests we get either mawk, original-awk or
gawk altering on every invocation.
While for the test it isn't poisionous whether wi have mawk or gawk,
having original-awk certainly is, because live-build only works with
mawk or gawk.
The best solution would obviously be to make debClosureGenerator
deterministic, but in the case of "Provides: awk", we can safely pick
mawk by default, because the latter has a "Priority: required" in its
package description.
This also has the advantage that we can safely cherry-pick this to
release-15.09 because it's very unlikely that we'll break the
debClosureGenerator by adding a dependency to commonDebPackages.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This should avoid accidential expansion of variables, i.e. in
"export PATH=/some/path:$PATH"
$PATH would have been expanded in the environment builder!
For practical purposes, here are the changes in behavior:
- When fetching from a subdirectory of a repo, do not rebuild because of
changes elsewhere in the repo
- Fetch (not-ignored) untracked files too
It does this by letting git hash and export the directory in question,
which I believes makes for a cleaner implementation than the ad-hoc copying
and hashing that was there before.
Close#9790.
This fixes checkouting for a nasty combination:
1. To be checkouted is a revision which corresponds to tag in a form "<tag>^{}".
2. This revision is not fetched by default.
You can now pass
separateDebugInfo = true;
to mkDerivation. This causes debug info to be separated from ELF
binaries and stored in the "debug" output. The advantage is that it
enables installing lean binaries, while still having the ability to
make sense of core dumps, etc.
Fixes#9044, close#9667. Thanks to @taku0 for suggesting this solution.
Now we have no modes starting with `/` or `+`.
Rewrite the `-perm` parameters of find:
- completely safe: rewrite `/0100` and `+100` to `-0100`,
- slightly semantics-changing: rewrite `+111` to `-0100`.
I cross-verified the `find` manual pages for Linux, Darwin, FreeBSD.
Upstream likes to move "old" releases to an archive mirror as soon as a
new one is released. This is now handled for free by mirrors.nix.
(No idea why cs.utah.edu was used to begin with; it's now added to
mirrors.nix. Note that it doesn't support SSL, but that applies to
several others so I don't see the harm.)
By default `makeWrapper` will not set argv[0] (this is a reversion to
the old default behavior). Based on the breakage we have seen from
changing the default, this is what most people want. The `wrapProgram`
function will send `--argv0 '"$0"'` to `makeWrapper`, i.e. it will
continue to pass-through the argv[0] that the wrapper is called with.
Also, in some cases, the result of fetchBower is different depending on the
value of $out. For now, it seems that it works best if using a local output
directory before copying to $out.
(cherry picked from commit aa4c6b027163abe0891f9ad438899f9679298a64)
