Passing `-l$NIX_BUILD_CORES` improperly limits the overall system load.
For a build machine which is configured to run `$B` builds where each
build gets `total cores / B` cores (`$C`), passing `-l $C` to make will
improperly limit the load to `$C` instead of `$B * $C`.
This effect becomes quite pronounced on machines with 80 cores, with
40 simultaneous builds and a cores limit of 2. On a machine with this
configuration, Nix will run 40 builds and make will limit the overall
system load to approximately 2. A build machine with this many cores
can happily run with a load approaching 80.
A non-solution is to oversubscribe the machine, by picking a larger
`$C`. However, there is no way to divide the number of cores in a way
which fairly subdivides the available cores when `$B` is greater than
1.
There has been exploration of passing a jobserver in to the sandbox,
or sharing a jobserver between all the builds. This is one option, but
relatively complicated and only supports make. Lots of other software
uses its own implementation of `-j` and doesn't support either `-l` or
the Make jobserver.
For the case of an interactive user machine, the user should limit
overall system load using `$B`, `$C`, and optionally systemd's
cpu/network/io limiting features.
Making this change should significantly improve the utilization of our
build farm, and improve the throughput of Hydra.
Before the change separate-debug-info.sh did the stripping itself.
This scheme has a few problems:
1. Stripping happens only on ELF files. *.a and *.o files are skipped.
Derivations have to do it manually. Usually incorrectly
as they don't run $RANLIB (true for `glibc` and `musl`).
2. Stripping happens on all paths. Ideally only `stripDebugList` paths
should be considered.
3. Host strip is called on Target files.
This change offloads stripping logic to strip hook. This strips more
files for `glibc` and `musl`. Now we can remove most $STRIP calls
from individual derivations.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
commit e938c0274 "Don't add access size hints to fortifiable functions"
converted a few '__attr_access ((...))' into '__fortified_attr_access (...)'
calls.
But one of conversions had double parentheses of '__fortified_attr_access (...)'.
Noticed as a gnat6 build failure:
/<<NIX>>-glibc-2.34-210-dev/include/bits/string_fortified.h:110:50: error: macro "__fortified_attr_access" requires 3 arguments, but only 1 given
The change fixes parentheses.
Rather than `$BASH`, `glibc` now hardcodes `/bin/bash` as
interpreter[1] in several scripts (including `ldd`).
This is a problem because relevant programs such as `ldd(1)` won't work
properly without this since we set `BASH` to `/bin/sh` to avoid
dependencies to the bootstrap tools for `bash` as runtime-dependency (since
NixOS doesn't have `/bin/bash`).
Considering that this was only done as an "improvement" to their
build-system and not because they wanted to use some bashisms here (the
variable was always called `BASH` and we still used `/bin/sh` anyways),
I'd consider this to be relatively safe.
[1] 5188a9d0265cc6f7235a8af1d31ab02e4a24853d
Linking via `-lpthread` (or `-pthread`) is not needed anymore since
`glibc-2.34` since all the functionality is part of `libc.so.6` and
`libpthread.so.6` only exists for backwards-compatibility.
However, e.g. `gcc` (`libgomp` to be precise) expects a `libpthread.so`
to link against, otherwise the configure script will fail. As already
stated in the glibc release-notes itself, it is to expect that a lot
more applications will have issues with this, so I decided to re-add
`libpthread.so` as well.
For `librt.so.1`, the same thing is needed to make sure that Perl still
compiles:
/nix/store/d6y5r7m93x14bmgn2p75fannz39jz66f-binutils-2.35.1/bin/ld: cannot find -lrt
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:490: ../../lib/auto/Time/HiRes/HiRes.so] Error 1
make[1]: Leaving directory '/build/perl-5.34.0/dist/Time-HiRes'
Since the script interpreters are not patched (anymore?), no reference
to any shell is generated by the scripts in glibc's `bin` output:
```
$ nix-store --query --references /nix/store/nmfj3a2k2kr2lypndn37x6qbd8537f9b-glibc-riscv64-unknown-linux-gnu-2.33-50-bin
/nix/store/afgfd3vl3x5zx89awn3fjayn83yl9vcy-glibc-riscv64-unknown-linux-gnu-2.33-50
/nix/store/nmfj3a2k2kr2lypndn37x6qbd8537f9b-glibc-riscv64-unknown-linux-gnu-2.33-50-bin
```
This means that it's safe to include the scripts in glibc's bin output
even in cross builds. One interesting improvement for the future could
be to split these scripts into a separate derivation and properly patch
their script interpreter (currently they are using impure /bin/sh).
`mtrace(1)` is a small Perl script that interprets and provides
human-readable output for `malloc(3)` traces.
Even though this is actually part of `glibc` itself I decided to place
this into its own package. The main reason for this is that this script
has a runtime dependency on Perl which would complicate `stdenv`
bootstrapping since we'd have to compile another Perl that doesn't depend on
the bootstrap tools that is used as runtime dependency for the
stage2 glibc.
Since this is only a dev/debugging tool, splitting this up seemed like a
reasonable choice to me.
On a leaking C program, this can be used like this:
$ env MALLOC_TRACE=$(pwd)/trace ./a.out
$ ./result/bin/mtrace ./trace
Memory not freed:
-----------------
Address Size Caller
0x0000000001875690 0x4 at 0x401151
Closes#141924
Without this patch, setting the same locale twice, e.g. like this in
NixOS:
```nix
{
i18n.supportedLocales = [
(config.i18n.defaultLocale + "/UTF-8")
(config.i18n.defaultLocale + "/UTF-8")
];
}
```
Would make the glibcLocales build fail with
```
Error: unsupported locales detected:
en_US.UTF-8/UTF-8 \
You should choose from the list above the error.
```
Add an explicit dependency on netbase for /etc/protocols
Certain functions in glibc look for files present in /etc such as getprotobyname which reads /etc/protocols.
If you are using Nix over a Linux installation, this file may not be present, and therefore it will cause errors.
- add netbase as a new package in nixpks
- add a dependency in glibc on it using postPatchPhase and substitute
the path
Fixes#124401
