Just a few comments added:
- added a few one-liners to explore which tools are rebuilt at each
stdenv iteration during bootstrap
- explicitly listed available toolchains and their sources for on each
bootstrap step: glibc, binutils, gcc, coreutils.
- added mention of static libraries linked into gcc
Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
The documentation for `meta.sourceProvenance` in
`doc/stdenv/meta.chapter.md` says: "the `meta.sourceProvenance`
attribute should be a list containing one or more value..."
Let's update check-meta.nix to require that `meta.sourceProvenance` is
a list, as the documentation says, rather than a single element.
Adding two extra keystrokes `[` and `]` when filling out this field is
an insignificant burden for package authors, and being able to assume
that the `meta.sourceProvenance` field is always a list greatly
simplifies any code that acts on the value of this field.
Since `meta.sourceProvenance` was just merged a few hours ago now is
the easiest time to fix this: nobody is using the feature yet.
In specific cases, combining the `checkMeta` and `checkMetaRecursively`
config options would result in `error: infinite recursion encountered`
fixes #193296
This is needed in order to mark a certain derivation containing a Nix
expression tarball to Hydra so that it is recognised as a channel.
When I first got an evaluation error due to using this meta attribute, I
was under the impression that nobody outside of Vuizvui[1] is using this
feature and that we don't have any occurrence of isHydraChannel in
Nixpkgs.
However, when working around[2] the issue I assumed that it's not
something that should be included in Nixpkgs because we're not using it
there.
It turned out that my assumption was wrong and we *do* use the attribute
in Nixpkgs, namely via releaseTools.channel, which is similar to what
we're doing in Vuizvui.
Since we already include a bunch of undocumented attributes in
metaTypes, it only makes sense to add isHydraChannel as well since it's
actually documented in the Hydra documentation[3].
[1]: https://github.com/openlab-aux/vuizvui
[2]: https://github.com/openlab-aux/vuizvui/commit/e0685e81b3fdc43a272f0
[3]: 53335323ae/doc/manual/src/jobs.md (meta-fields)
Signed-off-by: aszlig <aszlig@nix.build>
Passing `-l$NIX_BUILD_CORES` improperly limits the overall system load.
For a build machine which is configured to run `$B` builds where each
build gets `total cores / B` cores (`$C`), passing `-l $C` to make will
improperly limit the load to `$C` instead of `$B * $C`.
This effect becomes quite pronounced on machines with 80 cores, with
40 simultaneous builds and a cores limit of 2. On a machine with this
configuration, Nix will run 40 builds and make will limit the overall
system load to approximately 2. A build machine with this many cores
can happily run with a load approaching 80.
A non-solution is to oversubscribe the machine, by picking a larger
`$C`. However, there is no way to divide the number of cores in a way
which fairly subdivides the available cores when `$B` is greater than
1.
There has been exploration of passing a jobserver in to the sandbox,
or sharing a jobserver between all the builds. This is one option, but
relatively complicated and only supports make. Lots of other software
uses its own implementation of `-j` and doesn't support either `-l` or
the Make jobserver.
For the case of an interactive user machine, the user should limit
overall system load using `$B`, `$C`, and optionally systemd's
cpu/network/io limiting features.
Making this change should significantly improve the utilization of our
build farm, and improve the throughput of Hydra.
Stdenv on aarch64-darwin pulls in (bootstrap-stage4) objc4, unlike
x86_64. However derivations that otherwise depend on objc4 would use a
a different objc4 derivation on top of the final stdenv.
Because this library defines an LLVM module, having multiple instances
of it in the import path will interfere with builds.
inherit_errexit wasn’t available in bash 3. We have a check to show a
nice error message, but that check is after we set inherit_errexit in
setup.sh. So we can just move this to below the BASH_VERSINFO check.
gcc stopped using libelf in commit 48215350c24 ("re PR lto/46273 (Failed
to bootstrap)") around 2010, before gcc-4.6.0.
Bootstrap tools don't use it either.
This PR updates the Hydra-generated bootstrap tarballs for
powerpc64le-linux. The bootstrap-files referenced prior to this
commit will only bootstrap in a nixpkgs which has
https://github.com/NixOS/nixpkgs/pull/181802. That PR was closed in
favor of https://github.com/NixOS/nixpkgs/pull/181943, which is a
cleaner solution but which requires regenerating the bootstrap-files.
I'll be following the script established in
https://github.com/NixOS/nixpkgs/pull/151399, which I previously used
in https://github.com/NixOS/nixpkgs/pull/168199.
Files came from [this](https://hydra.nixos.org/build/186237511) Hydra build, which used nixpkgs revision ac43c44478 to instantiate:
```
/nix/store/nhjbza9vlcyhp9zxfz6lwpc3m2ghrpzj-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu.drv
```
and then built:
```
/nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
```
I downloaded these files from Hydra with the following commands:
```
STOREPATH=fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store add-file \
--name bootstrap-tools.tar.xz \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
--name busybox \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox
```
I then prefetched them into `/nix/store` with:
```
$ nix store prefetch-file --executable file:///nix/store/p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox
Downloaded 'file:///nix/store/p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox' to '/nix/store/a42qf2kf5hychcsw5sz0pvghy9vli1im-p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox' (hash 'sha256-jtPEAsht4AUAG4MLK8xocQSfveUR4ppU1lS4bGI1VN4=').
$ nix store prefetch-file file:///nix/store/y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz
Downloaded 'file:///nix/store/y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz' to '/nix/store/kgzyq9q08nll28ccqjcbv8angq5hyvdp-y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz' (hash 'sha256-MpIDnpZUK3M17qlnuoxfnK0EgxRosm3TMW1WfPZ1+jU=').
```
And started the bootstrap with the following command:
```
nix build -f . -L hello
```
As @lovesegfault requested in #151399, here are the the `sha256sum`s of all the `on-server` components for extra verification:
```
$ sha256sum /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/*
3292039e96542b7335eea967ba8c5f9cad04831468b26dd3316d567cf675fa35 /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
3d078dff7b4087d82442937667c91dace3321493aae4d3a4160d046b7eabcc2c /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/busybox
```
This is a change for `powerpc-linux` but that is ancient and I don't
think it matters. The impure bootstrap that was previously assigned to
it has probably bitrotted anyways.
Before the change an attempt to use `gnumake.override { guileSupport = true; }`
caused recursion in bootstrap stages as guileSupport pulls in guile and it's
dependencies.
To restore the bootstrap the change unconditionally sets
`guileSupport = false;` for `gnumake`.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
