`gcc-unwrapped` basically replaces `gccPlain`. It may seem like an ugly
polution to stick it in all-packages, but a future PR will enshrine this
`*-unwrapped` pattern. In any event, the long term goal is stdenvs might
need to tweak how compilers are booted and wrapped, but the code to build
the unwrapped compilers themselves should be generic.
Introduce new abstraction, `stdenv/booter.nix` for composing bootstraping
stages, and use it everywhere for consistency. See that file for more doc.
Stdenvs besides Linux and Darwin are completely refactored to utilize this.
Those two, due to their size and complexity, are minimally edited for
easier reviewing.
No hashes should be changed.
On one hand, don't want to pass garbage that affects hash, on the other
hand footguns are bad.
Now, factored out the derivation so only need to pass in what is used.
- The darwin test can now force the use of the freshly-booted darwin stdenv
- The linux test now passes enough dummy arguments
This may make debugging harder, if so, check out #20889
- Non-cross stdenvs are honest and assert that `crossSystem` is null
- `crossSystem` is a mandatory argument to top-level/stage.nix, just like
`system` and `platform`
- Broken default arguments on stdenvs for testing are gone.
- All stdenvs (but little-used stdenvNix) take the same arguments for easy
testing.
This makes the flow of data easier to understand. There's little downside
because the args in question are already inspected by the stdenvs.
cross-compiling in particular is simpler because we don't need to worry
about overriding the config closed over by `allPackages`.
This commit changes the dependencies of stdenv, and clean-up the stdenv
story by removing the `defaultStdenv` attribute as well as the `bootStdenv`
parameter.
Before, the final bootstrapping stage's stdenv was provided by
all-packages, which was iterating multiple times over the
top-level/default.nix expression, and non-final bootstrapping stages'
stdenvs were explicitly specified with the `bootStdenv` parameter.
Now, all stages' stdenvs are specified with the `stdenv` parameter.
For non-final bootstrapping stages, this is a small change---basically just
rename the parameter.
For the final stage, top-level/default.nix takes the chosen stdenv and
makes the final stage with it.
`allPackages` is used to make all bootstrapping stages, final and
non-final alike. It's basically the expression of `stage.nix` (along with a
few partially-applied default arguments)
Note, the make-bootstrap-tools scripts are temporarily broken
This reinstates the libSystem selective symbol export machinery we used
to have, but locks it to the symbols that were present in 10.11 and skips
the actual compiled code we put into that library in favor of the system
initialization code. That should make it more stable and less likely to
do weird stuff than the last time we did this.
`stripHash` documentation states that it prints out the stripped name to
the stdout, but the function stored the value in `strippedName`
instead.
Basically all usages did something like
`$(stripHash $foo | echo $strippedName)` which is just braindamaged.
Fixed the implementation and all invocations.
This is similar to `overrideDerivation`, but overrides the arguments to
`mkDerivation` instead of the underlying `derivation` call.
Also update `makeOverridable` so that uses of `overrideAttrs` can be
followed by `override` and `overrideDerivation`, i.e. they can be
mix-and-matched.
The 3.8 upgrade was premature and appears to be breaking a load of stuff
that I can't look into right now. Will take it back to 3.8 and test more
thoroughly before pushing more broadly. I should learn to change fewer
variables at a time :)
It's a long build and generally painful to split into smaller commits,
so I apologize for lumping many changes into one commit but this is far
easier.
There are still several outdated parts of the darwin stdenv but these
changes should bring us closer to the goal.
Fixes#18461
I broke this in the cleanups I did in 171c7f0, the gcc inside the bootstrap
tarball is not getting built with the correct --with-fpu, --with-float
etc. options.
Revert a revert of a merge that shouldn't have been in master but was intentionally in staging.
Next time I'll do this right after the revert instead of so far down the line...
This reverts commit 9adad8612b.
This will break part of the bootstrap tools tests because the new tools
need some changes in the stdenv, but if I change them all at once, the
stdenv breaks with the old bootstrap tools. So I'm doing this first, then
will make changes to the stdenv once this bundle is built and I can use
it.
I also added some functionality to let me test one set of bootstrap tools
on another nixpkgs tree, which makes testing a lot more pleasant.
- cloog, ppl, cloogppl aren't used by recent GCCs. Kill references to them.
- Use correct versions of isl, as the current GCC depends
on non-default versions of them.
- Also clarify isl dynamic libraries are needed in cross
builds, but not in native builds
- Since aeb3d8c (bzip2: fix cross build on mingw by using autoconf patch),
it seems that the bzip2 binary depends on libbz2 when cross compiling.
So copy libbz2 into the bootstrap tarball as well.
- Curl isn't used in the bootstrap tools since e6f61b4cf3.
This reverts commit debd401b0f.
We must not use a single-binary build for the bootstrap since the common
binary gains a dynamic linkage to gmp (due to 'factor' and 'expr'
handling arbitrary-precision arithmetic).
Our coreutils now uses single-binary-build mode where, by default,
simple shebang scripts are used for all the binaries. That doesn't work
e.g. with the Linux unpacker which only handles standard binaries and
symlinks. Let's use the symlinked mode instead for boostrapping.
This does NOT change any stdenv hashes.
I only tested the case most important to me:
$ nix-build pkgs/top-level/release.nix -A stdenvBootstrapTools.x86_64-linux.test
Close#15803. This avoids the error:
while setting up the build environment: executing
‘/nix/store/7sb42axk5lrxqz45nldrb2pchlys14s1-bash-4.3-p42/bin/bash’:
Argument list too long
Note: I wanted to make it optional based on buildCommand length,
but that seems pointless as I'm sure it's less performant.
Amended by vcunat:
https://github.com/NixOS/nixpkgs/pull/15803#issuecomment-224841225
On Linux, paxctl's setup hook should overwrite the paxmark stub, but the
stub is defined after the setup hooks are sourced, so the stub ends up
overwriting the real function. The result is that paxmark fails to do
anything. The fix is to define the stub before any setup hooks are
sourced. Thanks to @vcunat for figuring this out.
Closes#15492
This changes cygwin stdenv, but I don't think it will hurt much people.
This allows mkDerivation to get "dontRebase=true" to skip the usual cygwin
rebase. This is required, if we are using this stdenv to build DLLs for win32
inside x86_64-cygwin, because /bin/rebase crashes at finding an arch mismatch.
Additionally, we don't need any rebase for libraries built by visual studio and
meant for visual studio (my use case).
I'm using nix in x86_64-cygwin to build libraries with visual studio, both for
x86_64 and x86.
I'm giving this up. Feel free to find some reasonable variant that works
at least on Linux and Darwin. Problems encountered:
- During bootstrap of Darwin stdenv `env -0` and some bash features
don't work.
- Without `env -0` the contents of some multi-line phases is taken as
variable declarations, which wouldn't typically matter, but the PR
wanted to refuse bash-invalid names which would be occasionally
triggered. This commit dowgrades that to a warning with explanation.
It turned out that process substitution fed into a while-cycle
isn't recognized during darwin bootstrap:
http://hydra.nixos.org/build/35382446/nixlog/1/raw
Also fix broken NIX_DEBUG output, noticed by abbradar.
The set/env fix in #14907 wasn't very good, so let's use a null-delimited
approach. Suggested by Aszlig.
In particular, this should fix a mass-breakage on Darwin, though I was
unable to test that.
bash variable names may only contain alphanumeric ASCII-symbols and _,
and must not start with a number. Nix expression attribute names however
might contain nearly every character (in particular spaces and dashes).
Previously, a substitution that was not a valid bash name would be
expanded to an empty string. This commit introduce a check that throws
a (hopefully) helpful error when a wrong name is used in a substitution.
For every *.{exe,dll} in $output/bin/ we try to find all (potential)
transitive dependencies and symlink those DLLs into $output/bin
so they are found on invocation.
(DLLs are first searched in the directory of the running exe file.)
The links are relative, so relocating whole /nix/store won't break them.
The hook is activated on cygwin and when cross-compiling to mingw.
Close#14335.
Since 89036ef76a, when a package doesn't include a configure script,
the build complains with:
grep: : No such file or directory
grep: : No such file or directory
This prevents that.
Commit 2040a9ac57 changed the order of
$PATH elements, causing initialpath to appear after buildInputs. Thus
gnugrep ended up depending on bin/sh from bootstrapTools, rather than
from pkgs.bash. The fix is to provide pkgs.bash via buildInputs rather
than initialPath.
http://hydra.nixos.org/build/33276697
For some reason, the current bootstrap tools fail to build gettext:
init2.c:37: MPFR assertion failed: (64 - 0) == ((64 - 0)/8) * 8 && sizeof(mp_limb_t) == ((64 - 0)/8)
libxml/xpath.c: In function 'xmlXPathCompPathExpr':
libxml/xpath.c:10627:1: internal compiler error: Aborted
xmlXPathCompPathExpr(xmlXPathParserContextPtr ctxt) {
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
make[5]: *** [libxml/libxml_rpl_la-xpath.lo] Error 1
I didn't investigate why this is the case but rebuilding the bootstrap
tools seems to help.
I used this old-ish WIP branch https://github.com/dezgeg/nixpkgs/commits/arm-bootstrap
since latest master has even more problems with cross-compiling anything.
(I will eventually push this stuff and make the ARM bootstraps build on hydra.)
The $lib output refers to the terminfo database in $out, which is about
10x larger than the ncurses shared library. Splitting these outputs
saves a small amount of space for any derivations that use the terminfo
database but not the ncurses library, but we do not have evidence that
any such exist.
Otherwise, when building glibc and other packages, the "strip" from
bootstrapTools is used, which doesn't recognise some tags produced by
the newer "ld" from binutils.
I assume there's not much use for it during bootstrapping.
This fixes them as well, as curl was compiled against libnghttp2 but the
lib wasn't copied to the bootstrap tools.
Fixes#12632.
I think it's better to quote this variable in general, because it is
common and even documented to pass space-separated commands in there.
The greps should just fail in that case and `if` won't proceed
which seems fine for such cases, and it's certainly better than
passing additional unintended parameters to grep
(which was happening all the time before).
Doing it in an openssl setup hook only works if packages have openssl
as a build input - it doesn't work if they're using a program linked
against openssl.
Commit 6d928ab684 changed this to not
preserve timestamps. However, that results in non-determinism; in
particular, it gives us a broken $SOURCE_DATE_EPOCH (especially for
everything using fetchFromGitHub). Builds affected by timestamps <
1980 should be fixed in some other way (e.g. changing the timestamp to
some fixed date > 1980).
The ld-wrapper.sh script calls `readlink` in some circumstances. We need
to ensure that this is the `readlink` from the `coreutils` package so
that flag support is as expected.
This is accomplished by explicitly setting PATH at the top of each shell
script.
Without doing this, the following happens with a trivial `main.c`:
```
nix-env -f "<nixpkgs>" -iA pkgs.clang
$ clang main.c -L /nix/../nix/store/2ankvagznq062x1gifpxwkk7fp3xwy63-xnu-2422.115.4/Library -o a.out
readlink: illegal option -- f
usage: readlink [-n] [file ...]
```
The key element is the `..` in the path supplied to the linker via a
`-L` flag. With this patch, the above invocation works correctly on
darwin, whose native `/usr/bin/readlink` does not support the `-f` flag.
The explicit path also ensures that the `grep` called by `cc-wrapper.sh`
is the one from Nix.
Fixes#6447
This is used by some build tools to provide reproducible builds. See
https://reproducible-builds.org/specs/source-date-epoch/
for more info.
Later, we'll want to set this to a more intelligent value (such as the
most recent mtime of any source file).
So far if no configure script is found or no makefile,
the rest of the phase is skipped, *including* post-hooks.
I find that behavior unexpected/unintuitive.
Earlier version of this patch had problems due to me assuming
that $configureScript is always a simple path, but that turned out
to be false in many cases, e.g. perl.
This un-hardcodes the bootstrap tools passed into the Darwin stdenv and
thus allows us to quickly iterate on improving the design of the full
bootstrap process. We can easily change the contents of the bootstrap
tools and evaluate an entire bootstrap all the way up to real packages.
The most complex problems were from dealing with switches reverted in
the meantime (gcc5, gmp6, ncurses6).
It's likely that darwin is (still) broken nontrivially.
You can now pass
separateDebugInfo = true;
to mkDerivation. This causes debug info to be separated from ELF
binaries and stored in the "debug" output. The advantage is that it
enables installing lean binaries, while still having the ability to
make sense of core dumps, etc.
Otherwise this fails on ARM:
/nix/store/jipqp9739n7wrjz40igbk85pqk13s0ad-binutils-2.23.1/bin/ld: /nix/store/92pdpqrqkdf8wjciq1cisvsp8kdz8p2i-gmp-5.1.3/lib/libgmp.a(mp_get_fns.o): relocation R_ARM_MOVW_ABS_NC against `__gmp_allocate_func' can not be used when making a shared object; recompile with -fPIC
/nix/store/92pdpqrqkdf8wjciq1cisvsp8kdz8p2i-gmp-5.1.3/lib/libgmp.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [libisl.la] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/tmp/nix-build-isl-0.11.1.drv-3/isl-0.11.1'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/nix-build-isl-0.11.1.drv-3/isl-0.11.1'
make: *** [all] Error 2
builder for ‘/nix/store/a8ghniifd8d8agqx0cqsh41daa08v11c-isl-0.11.1.drv’ failed with exit code 2
Several places in the tree associate the ARMv7 system
with the beaglebone platform. Change them to point to
armv7l-hf-multiplatform as it supports several boards (including the
beaglebone as well)
Now development stuff is propagated from the first output,
and userEnvPkgs from the one with binaries.
Also don't move *.la files (yet). It causes problems, and they're small.
- there were many easy merge conflicts
- cc-wrapper needed nontrivial changes
Many other problems might've been created by interaction of the branches,
but stdenv and a few other packages build fine now.
Attrnames and package names should be as close as possible to avoid confusion.
I took care not to confuse the two mpc things during the mass-replace,
so hopefully I suceeded (tarball still builds).
Conflicts (simple):
pkgs/os-specific/linux/util-linux/default.nix
It seems this merge creates a new stdenv hash,
because we had changes on both branches :-/
- IMO using a temporary is not needed here (anymore),
- temporary at that location can cause a problem (in a specific case):
for example, when using the substituteAll function from nixpkgs
on a single file directly under /nix/store/ (or ./foo-file),
the stdenv's substitute tries to create a temporary directly under
/nix/store, which causes problems on chrooted darwin
(according to @copumpkin earlier today on IRC)
The old boot.spl.hostid option was not working correctly due to an
upstream bug.
Instead, now we will create the /etc/hostid file so that all applications
(including the ZFS kernel modules, ZFS user-space applications and other
unrelated programs) pick-up the same system-wide host id. Note that glibc
(and by extension, the `hostid` program) also respect the host id configured in
/etc/hostid, if it exists.
The hostid option is now mandatory when using ZFS because otherwise, ZFS will
require you to force-import your ZFS pools if you want to use them, which is
undesirable because it disables some of the checks that ZFS does to make sure it
is safe to import a ZFS pool.
The /etc/hostid file must also exist when booting the initrd, before the SPL
kernel module is loaded, so that ZFS picks up the hostid correctly.
The complexity in creating the /etc/hostid file is due to having to
write the host ID as a 32-bit binary value, taking into account the
endianness of the machine, while using only shell commands and/or simple
utilities (to avoid exploding the size of the initrd).
Getting the names of all environment variables is tricky. The previous
implementation easily got confused by multi-line variables. The new
one is more reliable but not still not perfect.
This works around a segfault in Bash 4.3, where the expression
"${!var}" (where var="-9") crashes under certain conditions.
http://hydra.nixos.org/build/16693445
Otherwise, stdenv won't have a reference to e.g. patchelf on Linux
(because it was passed in by mkDerivation). This causes the installer
tests to fail, because having "stdenv" in the installation CD closure
is not enough to pull in all stdenv packages.
http://hydra.nixos.org/build/16546643
This allows licenses like the Amazon Software License to be identified
properly while still preventing packages with those licenses from
being distributed in the Nixpkgs/NixOS channels.
This makes stuff build with Xcode 6.1 on Mac OS X 10.9 (where we got
errors like "ld: file not found:
/usr/lib/system/libsystem_coreservices.dylib for architecture x86_64" due to the use of the 10.10 SDK).
The static curl program is gone, replaced by curl inside of the
bootstrap tools tarball. Also, we generate a .tar.xz archive rather
than .cpio.bz2, making the download smaller. The separate
{sh,cpio,mkdir,ln,bzip2} programs have been replaced by a single
busybox program.
In 3.3, a C++ class defined in a header will get a typeinfo symbol
like this (e.g. in Nix's src/libutil/util.o):
(__DATA,__datacoal_nt) weak external typeinfo for nix::BaseError
But in 3.4, this has changed to:
(__DATA,__datacoal_nt) weak external automatically hidden typeinfo for nix::BaseError
This causes the linker to change the symbol to:
(__DATA,__data) non-external (was signed char private external) typeinfo for nix::BaseError
i.e. losing its weak linkage. But without weak linkage, dynamic_cast
and other RTTI-based mechanisms (such as catching an exception of a
certain type) don't work across shared libraries / executables.
The clang compiler in the SDK doesn't have this behaviour, but it's
not clear exactly which version it is (it just says "based on LLVM
3.4svn").
This should fix the OpenJDK build, which was failing because paxctl is
in sbin and therefore not automatically added to $PATH.
http://hydra.nixos.org/build/15658346
Copying /usr/lib/system/libunwind.dylib at evaluation time doesn't
work (e.g. on Hydra). And copying binary system libraries is a bad
idea anyway for license reasons.
Make thread disabling explicit. This changes the semantics of the perl
derivation, so on other platforms it may require setting
enableThreading = false
This commit doesn't change the derivation or out hash of stdenvLinux.
The stage3.extraAttrs.glibc argument was required for this whole build
procedure to correctly work and it was very-very hard to see why (the
comment said something about gcc47, but we're using gcc48 now).
This stage3.extraAttrs.glibc goes into stage3.stdenv.glibc after some
arg passing, and in pkgs/development/compiler/gcc an (stdenv ? glibc)
boolean expression decides to override /usr/include during the GCC
build.
All of our stages are built with glibc, so this refactoring moves this
repeating specification of glibc (once for gcc wrapper and once here for
extraAttrs) to stageFun, by getting rid of wrapGCC, as we were using
that in all of the stages anyways.
Incidentally it turned out, that this stdenv.glibc inconsistency caused
some random other stuff to behave differently:
- stage1.pkgs.perl has threading disabled,
- stage4.pkgs.coreutils (the production coreutils) has testing disabled.
Leave this historical accidents as they are in this commit, so the scope
of this commit can stay as a refactoring only, these issues will be
fixed in separate commits.
This commit doesn't change the derivation or the output hash of
stdenvLinux.
Use the new allowedRequisites feature in stdenvLinux.
This way we properly check that the end-result stdenv of the quite
complicated multi-stage stdenvLinux building procedure is sane, and only
depends on the stuff that we know about.
Alternative would be to just disallowRequisites bootstrapTools, which is
the most common offender, but we have had other offenders in the past.
For these checks to actually fire, you currently have to use nixUnstable,
as the necessary feature will be released in Nix 1.8.
Previously stdenv depended on two different zlibs and there was a third
one in the top-level package set for other purposes. This commit merges
all this zlibs to one.
Actually this have been committed once as 1f2b636, but then got lost
while resolving merge conflicts. Hopefully it survives this time.
The point here is that it's always possible to debug this staging
logic by using `nix-repl pkgs/stdenv/linux'. The modular-stdenv
change introduced the lib dependency, which we now default to
../../../lib.
No derivation or out hashes of stdenvLinux is changed by this commit.
Previously stdenv depended on two different zlibs and there was a third
one in the top-level package set for other purposes. This commit merges
all this zlibs to one.
Now it should contain *all* information from stdenv/setup.sh of
the original mutiple-output branch.
However, the configurability of the output paths is much greater.
This is needed for multiple-output derivations,
where it is desirable to propagate deps and setup-hooks into $dev instead of $out.
Also drop an unused simple function which will not even make sense.
Make stages explicit and generalize the pattern of having an stdenv and
a pkgs collection for all stages to a common stage generating function
called stageFun.
Rewrite all stage handling with this new function.
This commit doesn't change the outhash (or drvhash) of the stdenv.
Don't use default parameter values, to make the callsites more readable
and for easier debuggability/changability. Also reordered the
callsites' parameter ordering for consistency.
In the final stdenv don't repeat the name of the shell.
This commit doesn't change the outhash (or drvhash) of the stdenv.
All the different stages of stdenv had the fetchurl inherited anyways,
so make this generic in stdenvBootFun.
This commit doesn't change the outhash (or drvhash) of the stdenv.
