libtool's libtool.m4 script assumes that `file` is available, and can
be found at `/usr/bin/file` (this path is hardwired). Furthermore,
the script with this assumption is vendored into the ./configure
scripts of an enormous number of packages. Without this commit, you
will frequently see errors like this during the configurePhase with
the sandbox enabled:
./configure: line 9595: /usr/bin/file: command not found
Due mostly to luck, this error does not affect native compiles on
nixpkgs' two most popular platforms, x86_64-linux and aarch64-linux.
However it will cause incorrect linker flag detection and a failure to
generate shared libraries for sandboxed cross-builds to a x86_64-linux
host as well as any sandboxed build (cross or native) for the following
hosts: x86_64-freebsd, *-hpux, *-irix, mips64*-linux, powerpc*-linux,
s390x-linux, s390x-tpf, sparc-linux, and *-solaris.
This commit fixes the problem by adding an extra line to fixLibtool()
in pkgs/stdenv/generic/setup.sh. This extra line will scan the
unpacked source code for executable files named "configure" which
contain the following text:
'GNU Libtool is free software; you can redistribute it and/or modify'
This text is taken to be an indicator of a vendored libtool.m4. When
it is found, the configure script containing it is subjected to `sed
-i s_/usr/bin/file_file_` which replaces all occurrences of
`/usr/bin/file` with `file`.
Additionally, the `file` package is now considered to be part of
`stdenv`. It has been added to `common-path.nix` so that the `file`
binary will be found in the `$PATH` of every build, except for the
bootstrap-tools and the first few stages of stdenv boostrapping.
Verified no regressions under:
nix-build --arg pkgs 'import ./. {}' ./lib/tests/release.nix
This commit allows the following commands to complete, which should
enable Hydra to produce bootstrap-files for mips64el:
nix-build \
--option sandbox true \
--option sandbox-fallback false \
pkgs/top-level/release-cross.nix \
-A bootstrapTools.mips64el-linux-gnuabi64.build
nix-build \
--option sandbox true \
--option sandbox-fallback false \
. \
-A pkgsCross.mips64el-linux-gnuabi64.nix_2_4
Instead of requiring the platforms be equal, use `isCompatible` to
determine if we can execute tests. The upside of this is that we now
can execute tests for natively cross compiled package sets like
pkgsStatic, pkgsLLVM and pkgsCross.musl64 etc.
canExecute is like isCompatible, but also checks that the Kernels are
_equal_, i.e. that both platforms use the same syscall interface. This
is crucial in order to actually be able to execute binaries for the
other platform.
isCompatible is dropped, since it has changed semantically and there's
no use case left in nixpkgs.
the motivation for this is to simplify stdenv and ease the job of
reviewers due to them needing to tell contributors about the defacto
rule that configureFlags should be a list of strings
stdenv: deprecate addCoverageInstrumentation adapter
this used to be used in nixos/tests but it hasn't been used in nixpkgs
for years
stdenv: deprecate replaceMaintainersField adapter
it was added in 2009 in 01e98e49b1
by nbp
there are no uses of it in nixpkgs now
stdenv: deprecate validateLicenses adapter
it was added in 2009 in b29073af25
unfreePredicate is now handled in ./pkgs/stdenv/generic/check-meta.nix
so this is unnecessary
This warning logs when a package has no maintainers. It will stay silent
if `meta.maintainers` is not set at all, only complaining when it is an
empty list. In the future a separate warning could be added to allow for
that stricter behavior. Or this warning could be changed.
This will allow for adding more validity types in the future, such as a
warning type. (which is in the next commit in this series)
This is NOT a breaking change because validity.valid is never exposed
outside of `stdenv.mkDerivation`.
This PR adds Hydra-generated bootstrap tarballs for powerpc64le-linux.
I'll be following the script established in PR to tarballs.nixos.org.
Files came from this Hydra build:
https://hydra.nixos.org/build/172142499
Which used nixpkgs revision 49a83445c2
to instantiate:
/nix/store/gj272sd56gsj6qpyzh4njpfzwdhviliz-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu.drv
and then built:
/nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
I downloaded these files from Hydra with the following commands:
```
STOREPATH=n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org"
nix store add-file \
--name bootstrap-tools.tar.xz \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
--name busybox \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox
```
As @lovesegfault requested in #151399, here are the the `sha256sum`s
of all the `on-server` components for extra verification:
```
$ sha256sum /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/*
036d062869f7accf0ad89714d12029469dfe6af504f9b226d61eb7d808ad4735 /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
6bc05832180f5075f4458c32eb0a5e2b673f605884dce01822be474f4e0a63ee /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/busybox
```
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
This isn't really desirable in general, but given that Nix itself
currently relies on this behaviour and that we don't want to break
backwards compatibility we should support it for now, maybe
deprecating it in the future.
This reverts commit c1ef87e8d0.
I didn't realize that it would rebuild all linux stdenvs;
I certainly don't want to incur such a rebuild at this point.
At some point, I'd like to make another attempt at
71f1f4884b ("openssl: stop static binaries referencing libs"), which
was reverted in 195c7da07d. One problem with my previous attempt is
that I moved OpenSSL's libraries to a lib output, but many dependent
packages were hardcoding the out output as the location of the
libraries. This patch fixes every such case I could find in the tree.
It won't have any effect immediately, but will mean these packages
will automatically use an OpenSSL lib output if it is reintroduced in
future.
This patch should cause very few rebuilds, because it shouldn't make
any change at all to most packages I'm touching. The few rebuilds
that are introduced come from when I've changed a package builder not
to use variable names like openssl.out in scripts / substitution
patterns, which would be confusing since they don't hardcode the
output any more.
I started by making the following global replacements:
${pkgs.openssl.out}/lib -> ${lib.getLib pkgs.openssl}/lib
${openssl.out}/lib -> ${lib.getLib openssl}/lib
Then I removed the ".out" suffix when part of the argument to
lib.makeLibraryPath, since that function uses lib.getLib internally.
Then I fixed up cases where openssl was part of the -L flag to the
compiler/linker, since that unambigously is referring to libraries.
Then I manually investigated and fixed the following packages:
- pycurl
- citrix-workspace
- ppp
- wraith
- unbound
- gambit
- acl2
I'm reasonably confindent in my fixes for all of them.
For acl2, since the openssl library paths are manually provided above
anyway, I don't think openssl is required separately as a build input
at all. Removing it doesn't make a difference to the output size, the
file list, or the closure.
I've tested evaluation with the OfBorg meta checks, to protect against
introducing evaluation failures.
When implementing this reviewer request:
https://github.com/NixOS/nixpkgs/pull/161158#discussion_r822256070
to move mips64el-unknown-linux-* from platforms.nix to examples.nix, I
neglected to update the reference in make-bootstrap-tools-cross.nix.
As a result, the hydra jobs to generate a bootstrap tarball for
mips64el are not running:
https://hydra.nixos.org/eval/1750828?filter=mips&compare=1750620&full=#tabs-errors
This commit fixes the problem, so the hydra job can run. Once it
does, I will submit a PR adding the trusted bootstrap tarball hash to
pkgs/stdenv/linux/bootstrap-files/.
Co-authored-by: sterni <sternenseemann@systemli.org>
This change is prompted by the following, admittedly cursed tarball:
```
> curl https://registry.npmjs.org/char-regex/-/char-regex-1.0.2.tgz 2>/dev/null \
| tar -ztv
drw-rw-rw- 0/0 0 2020-02-18 10:50 package
-rw-rw-rw- 0/0 297 2020-02-18 10:50 package/index.d.ts
-rw-rw-rw- 0/0 1920 2020-02-18 10:50 package/index.js
-rw-rw-rw- 0/0 1092 2020-01-31 11:31 package/LICENSE
-rw-rw-rw- 0/0 937 2020-02-18 10:51 package/package.json
-rw-rw-rw- 0/0 713 2020-02-18 10:50 package/README.md
```
The minimal reproducer for the issue is the following derivation trying
to work around the uid 0 issue with `dontMakeSourcesWritable = true`:
```nix
{ stdenv, fetchurl }:
stdenv.mkDerivation {
name = "test";
src = fetchurl {
sha1 = "d744358226217f981ed58f479b1d6bcc29545dcf";
url = "https://registry.npmjs.org/char-regex/-/char-regex-1.0.2.tgz";
};
dontMakeSourcesWritable = true;
installPhase = ''
cp -R . $out
'';
}
```
This currently fails in the following way:
```
these derivations will be built:
/nix/store/pc3jbydl0xcc8nrndf5xkf7hdhpgpb41-test.drv
building '/nix/store/pc3jbydl0xcc8nrndf5xkf7hdhpgpb41-test.drv'...
unpacking sources
unpacking source archive /nix/store/v9p98kqplf4kflmy91p0687xlvr6klb1-char-regex-1.0.2.tgz
source root is package
find: 'package/index.d.ts': Permission denied
find: 'package/index.js': Permission denied
find: 'package/LICENSE': Permission denied
find: 'package/package.json': Permission denied
find: 'package/README.md': Permission denied
/nix/store/6c47azxacncswc1pllzj28zfzqw40d7c-stdenv-linux/setup: line 1311: cd: package: Permission denied
builder for '/nix/store/pc3jbydl0xcc8nrndf5xkf7hdhpgpb41-test.drv' failed with exit code 1
error: build of '/nix/store/pc3jbydl0xcc8nrndf5xkf7hdhpgpb41-test.drv' failed
```
As you can see, the issue is that `$sourceRoot` isn't executable,
prohibiting the call to `cd`. This can be fixed by running
`chmod +x "${sourceRoot}"` before `cd` regardless of
`dontMakeSourcesWritable` in `unpackPhase` since if `chmod` fails, `cd`
would fail as well and we are out of options.
Verified that the workaround works locally.
Another thing to investigate is investigating if we should use
`--no-same-owner` for `tar` and if it helps in this case as well.
See also <https://github.com/Profpatsch/yarn2nix/issues/56>.
MIPS has a large space of {architecture,abi,endianness}; this commit
adds all of them to lib/systems/platforms.nix so we can be done with
it.
Currently lib/systems/inspect.nix has a single "isMips" predicate,
which is a bit ambiguous now that we will have both mips32 and mips64
support, with the latter having two ABIs. Let's add four new
predicates (isMips32, isMips64, isMips64n32, and isMips64n64) and
treat the now-ambiguous isMips as deprecated in favor of the
more-specific predicates. These predicates are used mainly for
enabling/disabling target-specific workarounds, and it is extremely
rare that a platform-specific workaround is needed, and both mips32
and mips64 need exactly the same workaround.
The separate predicates (isMips64n32 and isMips64n64) for ABI
distinctions are, unfortunately, useful. Boost's user-scheduled
threading (used by nix) does does not currently supports mips64n32,
which is a very desirable ABI on routers since they rarely have
more than 2**32 bytes of DRAM.
7459a40 `stdenv-bootstrap: force using new libc from stage2` overrode
`gcc-wrapper`. 91fa7657 `stdenv: revert gcc hack` dod not restore the initial
value.
As a result attempt to build glibc-2.35 fails early on perl as:
ld: cannot find -lnsl: No such file or directory
collect2: error: ld returned 1 exit status
I can't compile the test program.
(The supplied flags or libraries might be incorrect.)
The change restores `gcc-wrapper` as it was before 7459a40.
This fix is needed to work around linker-errors such as
undefined reference to `__libc_csu_fini'
which I got in almost every derivation which is part of stage2. The
reasoning behind this is that the startup-code was simplified[1] and
thus `__libc_csu_fini` doesn't exist anymore.
A workable solution is to use a newer libc which properly links in
stage3. And actually this seems expected given the rationale for stage3:
# Construct a third stdenv identical to the 2nd, except that this
# one uses the rebuilt Glibc from stage2. It still uses the recent
# binutils and rest of the bootstrap tools, including GCC.
So this patch basically overrides the libraries inside `gcc-unwrapped` -
which is basically the bootstrap tools and thus also contains the libc
used in stage3 - with the shared objects from the freshly built libc
from stage2.
[1] https://sourceware.org/pipermail/libc-alpha/2021-March/123079.html
Flake users that use a command like `nix build nixpkgs#hello` on a
broken/insecure package will not be able to use an environment variable
to override that behavior, unless they pass `--impure` to the command.
Co-authored-by: pkharvey <kayharvey@protonmail.com>
Adds an easy method of appending compiler flags to your stdenv via a
list.
Co-authored-by: tomberek <tomberek@users.noreply.github.com>
Co-authored-by: Gytis Ivaskevicius <gytis02.21@gmail.com>
Co-authored-by: sternenseemann <sternenseemann@systemli.org>
The `curlMinimal` is to be used throughout the early bootstrap
stages. The final stage will allow the new references of the `curl`.
Fixes: 29526bc2 ('curl: IDN support requires libidn2 package')
llvmPackages_*.clang should check the default compiler for the package
set it is targeting (targetPackages.stdenv.cc) instead of the compiler
that has been used to build it (stdenv.cc) in order to get some sense of
whether to use libc++ or libstdc++.
Since we are now inspecting targetPackages in the llvmPackages.clang
attribute, we need to avoid using it in the cross stdenv — which just
forces us to explicitly request libcxxClang for darwin instead of
relying on the clang attribute to pick it for us.
We also need to do something similar for targetPackages.stdenv.cc: Here
the llvmPackages.clang logic would work as we want (inspect
targetPackages.stdenv.cc and if it doesn't exist, make the choice based
on stdenv.cc), but it gets locked in a cycle with the previous package.
We can easily break this, however: We know that the previous set had
clang and the next one doesn't exist, so we'd choose libcxxClang any day
of the week.
Add `shellDryRun` to the generic stdenv and substitute it for uses of
`${stdenv.shell} -n`. The point of this layer of abstraction is to add
the flag `-O extglob`, which resolves#126344 in a more direct way.
`TargetConditionals.h` was missing several definitions, like
`TARGET_OS_TV` that are part of SDK 10.12 at least. And one that doesn't
seem to occur in any SDK afaict, `TARGET_OS_EMBEDDED_OTHER`.
I added the definitions from SDK 10.12 verbatim and defined
`TARGET_OS_EMBEDDED_OTHER` to be equal to `0`.
I think none of this works if `darwin.Libsystem` is used to build for
linux or iOS though so maybe this needs a more thorough fix?
This reverts 336d82617f because it's no
longer necessary.
Packages that use libtool run it as a wrapper around the linker.
Before calling the linker, libtool will determine what libraries would
be linked, and check if there's a corresponding libtool
archive (libfoo.la) file in the same directory . This file
contains extra information about the library. This is especially
important for static linking, because static archives don't contain
dependency information, so we need libtool to use the .la files to
figure out which libraries actually need to be linked against.
But in Nixpkgs, this has never worked. libtool isn't able to find any
libraries, because only the compiler wrapper knows how to find them,
and the compiler wrapper is opaque to libtool. This is why
pkgsStatic.util-linuxMinimal doesn't build prior to this patch — it
depends on libpam, which depends on libaudit, and if libtool can't
find the .la file, nothing will tell the linker to also link against
libaudit when linking libpam. (It was previously possible to build a
static util-linux, because linux-pam only recently had the audit
dependency added.)
There are a couple of ways we could fix this, so that libtool knows
where to look for .la files.
* Set LD_LIBRARY_PATH/DYLD_LIBRARY_PATH/whatever, which libtool will
examine. This would have major side effects though, because the
dynamic linker looks at it too.
* Inject libtool scripts with the appropriate information. That's
what I've done here. It was the obvious choice because we're
already finding and modifying the libtool scripts, to remove paths
outside the Nix store that libtool might check in unsandboxed
builds. Instead of emptying out the system paths, we can
repopulate it with our own library paths.
(We can't use a wrapper like we do for other tools in Nixpkgs, because
libtool scripts are often distributed in source tarballs, so we can't
just add a wrapped version of libtool as a dependency. That's why
there's already the fixLibtool function in stdenv.)
With this change, libtool is able to discover .la files, and
pkgsStatic.util-linuxMinimal can build again, linking correctly
against libpam and libaudit.
`--enable-deterministic-archives` is a GNU specific strip flag and
causes other strip implementations (for example LLVM's, see #138013)
to fail. Since strip failures are ignored, this means that stripping
doesn't work at all in certain situation (causing unnecessary
dependencies etc.).
To fix this, no longer pass `--enable-deterministic-archives`
unconditionally, but instead add it in a GNU binutils specific strip
wrapper only.
`commonStripFlags` was only used for this flag, so we can remove
it altogether.
Future work could be to make a generic strip wrapper, with support for
nix-support/strip-flags-{before,after} and NIX_STRIP_FLAGS_{BEFORE,AFTER}.
This possibly overkill and unnecessary though -- also with the
additional challenge of incorporating the darwin strip wrapper somehow.
This reverts commit 488395c0f8.
Currently, `nix print-dev-env` fails to execute if this function is present, because of its use of hex literals.
Until this issue (https://github.com/NixOS/nix/issues/5262) is solved, we should revert this to prevent breakage.
somehow `read -N 0` behavior changed in bash 5. `read -d ''` has identical behavior
the purpose of the function is to read stdin and exit 1 on a null byte (i.e. if stdin is the content of a binary)
(cherry picked from commit 5d0acf20f8)
The old stdenv adapters were subtly wrong in two ways:
- `overrideAttrs` leaked the original, unoverridden `mkDerivation`.
- `stdenv.override` would throw away any manually-set `mkDerivation`
from a stdenv reverting to the original.
Now, `mkDerivation` is controlled (nearly directly) via an argument, and
always correctly closes over the final ("self") stdenv. This means the
adapters can work entirely via `.override` without any manual `stdenv //
...`, and both those issues are fixed.
Note hashes are changed, because stdenvs no previously overridden like
`stdenvNoCC` and `crossLibcStdenv` now are. I had to add some
`dontDisableStatic = true` accordingly. The flip side however is that
since the overrides compose, we no longer need to override anything but
the default `stdenv` from which all the others are created.
When we "fix" libtool, we empty out its system library path to avoid
it discovering libraries in e.g. /usr when the sandbox is disabled.
But this also means that the checks libtool does to make sure it can
find the libraries its supposed to be linking to won't work. On Linux
and Darwin, this isn't a problem, because libtool doesn't actually
perform any checks, but it is on at least NetBSD and Cygwin[1].
So, we force libtool not to do these checks on any platform, bringing
the more exotic platforms into line with the existing behaviour on
Linux and Darwin.
Without this change, lots of library packages produce warnings like
this in their build output on the platforms with checks by default:
*** Warning: linker path does not have real file for library -lz.
*** I have the capability to make that library automatically link in when
*** you link to this library. But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libz but no candidates were found. (...for regex pattern test)
*** The inter-library dependencies that have been dropped here will be
*** automatically added whenever a program is linked with this library
*** or is declared to -dlopen it.
And dependent packages break because libtool doesn't link their
transitive dependencies. So making this change fixes _lots_ of
packages on those platforms.
[1]: https://git.savannah.gnu.org/cgit/libtool.git/tree/m4/libtool.m4?id=544fc0e2c2a03129a540aebef41ad32bfb5c06b8#n3445
At least for now. Such changes are risky (we have very many packages),
and apparently it needs more testing/review without blocking other
changes.
This reverts the whole range 4d0e3984918^..8752c327377,
except for one commit that got reverted in 6f239d7309 already.
(that MR didn't even get its merge commit)
