When running `nix develop` for a package, Nix records the stdenv
environment with NIX_LOG_FD set. That is, when the actual development
shell runs, it uses the functions that attempt to log to NIX_LOG_FD, but
this variable is not actually set.
As a workaround, check whether NIX_LOG_FD is set at runtime.
Example (before this change):
```console
$ nix develop --file . bash
$ echo "${NIX_LOG_FD-unset}"
unset
$ runPhase unpackPhase
bash: "$NIX_LOG_FD": Bad file descriptor
Running phase: unpackPhase
unpacking source archive /nix/store/v28dv6l0qk3j382kp40bksa1v6h7dx9p-bash-5.2.tar.gz
bash: "$NIX_LOG_FD": Bad file descriptor
source root is bash-5.2
bash: "$NIX_LOG_FD": Bad file descriptor
setting SOURCE_DATE_EPOCH to timestamp 1663942708 of file bash-5.2/y.tab.h
```
After this change:
```console
$ nix develop --file . bash
$ runPhase unpackPhase
Running phase: unpackPhase
unpacking source archive /nix/store/v28dv6l0qk3j382kp40bksa1v6h7dx9p-bash-5.2.tar.gz
source root is bash-5.2
setting SOURCE_DATE_EPOCH to timestamp 1663942708 of file bash-5.2/y.tab.h
```
This reduces the number of Python builds in the bootstrap to two: a minimal build and a normal build. Both have LTO disabled, which is required due to missing LLVM LTO libraries. This is necessary to correctly enable LTO builds in Python because it needs `llvm-ar` from `stdenv.cc.cc.libllvm`, which does not exist in the bootstrap.
The old stdenv didn't work, and was also impure. The new one works, and
is pure. Presently, the bootstrap tools are cross compiled into one small
nar and one large tar, which is then unpacked, patched, and split into
smaller derivations. Efforts were made to make the boot process as short
as possible - there are only two clangs built, and as many packages are
propagated between stages as possible while leaving the bootstrap tools
out of the final stdenv's closure.
Fetchers can use the `curl` binary from the bootstrap tools. Allowing packages in the Darwin bootstrap to link curl makes any curl update cause a full rebuild on Darwin, which is undesirable.
After https://github.com/NixOS/nixpkgs/pull/322388, darwin.cctools is just Apple’s cctools again. The replacement for what the bootstrap tools wants is darwin.binutils-unwrapped.
GNU binutils is not preferred on Darwin, and newer versions have issues building. Make it an evaluation error to use it in the Darwin stdenv bootstrap.
- Only link binaries that exist for stage 0 cctools and LLVM bintools;
- Drop cctools-llvm in favor of the updated darwin.binutils; and
- Update llvm-manages Python overrides (needed for newer versions of LLVM).
I realized what rhelmot did in 61202561d9
(specify what packages just need `stdenvNoLibc`) is definitely the right
approach for this, and adjusted NetBSD and OpenBSD to likewise use it.
With that change, we don't need these confusing and ugly `*bsdCross`
package sets at all!
We can get rid of a lot more libc-related `*Cross`, and I will do so
soon, but this is the first step.
(adapted from commit 51f1ecaa59)
For a long time, we've had `crossLibcStdenv`, `*Cross` libc attributes,
and `*bsdCross` pre-libc package sets. This was always bad because
having "cross" things is "not declarative": the naming doesn't reflect
what packages *need* but rather how we *provide* something. This is
ugly, and creates needless friction between cross and native building.
Now, almost all of these `*Cross` attributes are gone: just these are
kept:
- Glibc's and Musl's are kept, because those packages are widely used
and I didn't want to risk changing the native builds of those at this
time.
- generic `libcCross`, `theadsCross`, and friends, because these relate
to the convolulted GCC bootstrap which still needs to be redone.
The BSD and obscure Linux or freestnanding libcs have conversely all
been made to use a new `stdenvNoLibc`, which is like the old
`crossLibcStdenv` except:
1. It usable for native and cross alike
2. It named according to what it *is* ("a standard environment without
libc but with a C compiler"), rather than some non-compositional
jargon ("the stdenv used for building libc when cross compiling",
yuck).
I should have done this change long ago, but I was stymied because of
"infinite recursions". The problem was that in too many cases we are
overriding `stdenv` to *remove* things we don't need, and this risks
cyles since those more minimal stdenvs are used to build things in the
more maximal stdenvs.
The solution is to pass `stage.nix` `stdenvNoCC`, so we can override to
*build up* rather than *tear down*. For now, the full `stdenv` is also
passed, so I don't need to change the native bootstraps, but I can see
this changing as we make things more uniform and clean those up.
Finally, the BSDs also had to be cleaned up, since they have a few
pre-libc dependencies, demanding a systematic approach. I realized what
rhelmot did in 61202561d9 (specify what
packages just need `stdenvNoLibc`) is definitely the right approach for
this, and adjusted NetBSD and OpenBSD to likewise use it.
Without the change `runPhase` fails on tarballs like
`diffoscope-269` that contain single top-level `-269` root as:
diffoscope> unpacking source archive /nix/store/p620nidkm73vrp0z6kk5krmrm4vg7bxd-diffoscope-269.tar.bz2
diffoscope> source root is -269
diffoscope> setting SOURCE_DATE_EPOCH to timestamp 1717143039 of file ./-269/tests/utils/versions.py
diffoscope> chmod: invalid mode: ‘-269’
diffoscope> Try 'chmod --help' for more information.
Currently `diffoscope-269` has a `sourceRoot = "./-269";` workaround to
bypass the failure.
Most Linux distributions are enabling this these days and it does
protect against real world vulnerabilities as demonstrated by
CVE-2018-16864 and CVE-2018-16865.
Fix #53753.
Information on llvm version support gleaned from
6609892a2d68e07da3e5092507a730
Information on gcc version support a lot harder to gather,
but both 32bit and 64bit arm do appear to be supported
based on the test suite.
A second take at eb28e5e72e, which was reverted for the extra logging
during the internals of `nix-shell -p`. This commit does the same
logging, but to $NIX_LOG_FD instead, which is echoed during any normal
build, but not during the internals of `nix-shell -p`.
[1]: eb28e5e72e
libiconv-darwin depends on Meson, which (indirectly) depends on
libiconv. When libiconv-darwin is set as libiconv, it will cause an
infinite recursion. Avoid the infinite recursion by using libiconvReal
in stage 1. Every stage after that can use libiconv-darwin.
The cc and bintools wrapper contained ad hoc bootstrapping logic for
expand-response-params (which was callPackage-ed in a let binding). This
lead to the strange situation that the bootstrapping logic related to
expand-response-params is split between the wrapper derivations (where
it is duplicated) and the actual stdenv bootstrapping.
To clean this up, the wrappers simply should take expand-response-params
as an ordinary input: They need an adjacent expand-response-params (i.e.
one that runs on their host platform), but don't care about the how.
Providing this is only problematic during stdenv bootstrapping where we
have to pull it from the previous stage at times.
We don't need to artificially make sure that we can execute the wrapper
scripts on the build platform by using stdenv's shell (which comes from
buildPackages) since our cross infrastructure will get us the wrapper
from buildPackages. The upside of this change is that cross-compiled
wrappers (e.g. pkgsCross.aarch64-multiplatform.gcc) will actually work
when executed!
For bootstrapping this is also not a problem, since we have a long
build->build platform chain so runtimeShell is just as good as
stdenvNoCC.shell. We do fall back to old ways, though, by explicitly
using the bootstrap-tools shell in stage2, so the adjacent bash is only
used from stage4 onwards. This is unnecessary in principle (I'll try
removing this hack in the future), but ensures this change causes zero
rebuilds.
As of late, `final: prev: …` for overlays has become more prevalent in
newer code. This is also exhibited in some code (presumably added
recently) in stdenv. This change is not about any merits or demerits of
any naming convention, but rather aims to make the nomenclature in
stdenv bootstrapping consistent to lessen confusion.
I've chosen to stick to `self: super: …` convention because:
1. It is more common in the code as it stands.
2. Using `final: prev: …` makes the code more confusing, als it causes
`prev` to be in scope alongside `prevStage`. `prevStage` actually
bears no relation to `prev` even though their naming suggests it,
making it easy to confuse them (the former is the (final) package set
of the previous stage while the latter is just the `prev`/`super` of
the overlay “chaining” on a completely fresh package set, i.e. `prev`
doesn't even relate to the previous stage's `overrides` argument).
This change also corrects a naming error in stdenv/native which had no
effect, as the variables were unused.
This fixes using inputDerivation on derivations that are fixed-output.
Previously:
```
nix-repl> drv = runCommand "huh" { outputHash = "sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU="; outputHashAlgo = "sha256"; outputHashType = "flat"; } "touch $out"
nix-repl> drv.inputDerivation
«derivation /nix/store/d8mjs6cmmvsr1fv7psm6imis5pmh9bcs-huh.drv»
nix-repl> :b drv.inputDerivation
error: fixed output derivation 'huh' is not allowed to refer to other store paths.
You may need to use the 'unsafeDiscardReferences' derivation attribute, see the manual for more details.
```
Fixes: https://github.com/NixOS/nixpkgs/issues/304209
Nixpkgs tries to print a helpful message when it blocks unfree packages,
but the suggestion is subtly broken. The predicate only matches on the
package's name, but the suggestion includes the full name-version pair.
Fixed by formatting the message with the same function as the predicate.
This issue arises because check-meta defines its own local getName with
semantics divergent from lib.getName. The former includes the version,
the latter does not.
Example Before:
Alternatively you can configure a predicate to allow specific packages:
{ nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
"obsidian-1.5.12"
];
}
Example After:
Alternatively you can configure a predicate to allow specific packages:
{ nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
"obsidian"
];
}
Fixes #303116
checkValidity has the responsibility to check if a derivation's attributes are valid.
Previously it also had the overloaded task of creating a subset of meta attributes:
- unfree
- broken
- unsupported
- insecure
Not only is this overloading strange, these attributes were only ever consumed by `commonMeta`.
This change makes checkValidity _only_ check for validity, and removes the creation of any meta attributes from `checkValidity` and moves them to `commonMeta`.
This is technically a breaking change but I don't expect any external nixpkgs consumers to rely on these implementation details.
