a4630c65ca was incorrect in assuming $SHELL would be a path to the
bash derivation. In fact $SHELL will be a path to the bash executable.
Unfortunately this did not fix the original issue. So instead, we just
have to reuse initialPath can be added like PATH is.
Sorry for the inconvenience! I hadn’t thought through the effects of
the last commit.
/cc @copumpkin @ericson2314
To avoid breaking things, we need to make sure SHELL goes into
HOST_PATH. This reflects my changes to patch-shebangs to make it cross
compilation ready. When a script is patched from the Nix store it now
looks to HOST_PATH to get the targeted machine’s executables.
Unfortunately, this only works in native builds.
LTO is disabled during bootstrap to keep the bootstrap tools small and
avoid unnecessary LLVM rebuilds, but is enabled in the final stdenv
stage and should be usable by normal packages.
This also updates the bootstrap tool builder to LLVM 5, but not the ones
we actually use for bootstrap. I'll make that change in a subsequent commit
so as to provide traceable provenance of the bootstrap tools.
Intuitively, one cares mainly about the host platform: Platforms differ
in meaningful ways but compilation is morally a pure process and
probably doesn't care, or those difference are already abstracted away.
@Dezgeg also empirically confirmed that > 95% of checks are indeed of
the host platform.
Yet these attributes in the old cross infrastructure were defined to be
the build platform, for expediency. And this was never before changed.
(For native builds build and host coincide, so it isn't clear what the
intention was.)
Fixing this doesn't affect native builds, since again they coincide. It
also doesn't affect cross builds of anything in Nixpkgs, as these are no
longer used. It could affect external cross builds, but I deem that
unlikely as anyone thinking about cross would use more explicit
attributes for clarity, all the more so because the rarity of inspecting
the build platform.
Works similarly to `enableParallelBuilding`, but is set by default when
`enableParallelBuilding` is set. In my experience most packages that build
fine in parallel also check fine in parallel.
Derivations where drawing their `system` attribute from `hostPlatform`
instead of `buildPlatform`. Fix that, and add an explanatory commment.
Fixes #45993
This has been not touched in 6 years. Let's remove it to cause less
problems when adding new cross-compiling infrastructure.
This also simplify gcc significantly.
This reverts commit a809fdc8e1 and then
achieves the same result (not rebuilding texinfo three times)
but without dragging bootstrap tools into the closure.
I *want* cross-specific overrides to be verbose, so I rather not have
this shorthand. This makes the syntactic overhead more proportional to
the maintainence cost. Hopefully this pushes people towards fewer
conditionals and more abstractions.
On darwin llvmPackages is built using python-boot to avoid dependencies
in the stdenv, but we can't and shouldn't use that when building the
manpages since it depends on python packages.
* substitute(): --subst-var was silently coercing to "" if the variable does not exist.
* libffi: simplify using `checkInputs`
* pythonPackges.hypothesis, pythonPackages.pytest: simpify dependency cycle fix
* utillinux: 2.32 -> 2.32.1
https://lkml.org/lkml/2018/7/16/532
* busybox: 1.29.0 -> 1.29.1
* bind: 9.12.1-P2 -> 9.12.2
https://ftp.isc.org/isc/bind9/9.12.2/RELEASE-NOTES-bind-9.12.2.html
* curl: 7.60.0 -> 7.61.0
* gvfs: make tests run, but disable
* ilmbase: disable tests on i686. Spooky!
* mdds: fix tests
* git: disable checks as tests are run in installcheck
* ruby: disable tests
* libcommuni: disable checks as tests are run in installcheck
* librdf: make tests run, but disable
* neon, neon_0_29: make tests run, but disable
* pciutils: 3.6.0 -> 3.6.1
Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/pciutils/versions.
* mesa: more include fixes
mostly from void-linux (thanks!)
* npth: 1.5 -> 1.6
minor bump
* boost167: Add lockfree next_prior patch
* stdenv: cleanup darwin bootstrapping
Also gets rid of the full python and some of it's dependencies in the
stdenv build closure.
* Revert "pciutils: use standardized equivalent for canonicalize_file_name"
This reverts commit f8db20fb3a.
Patching should no longer be needed with 3.6.1.
* binutils-wrapper: Try to avoid adding unnecessary -L flags
(cherry picked from commit f3758258b8895508475caf83e92bfb236a27ceb9)
Signed-off-by: Domen Kožar <domen@dev.si>
* libffi: don't check on darwin
libffi usages in stdenv broken darwin. We need to disable doCheck for that case.
* "rm $out/share/icons/hicolor/icon-theme.cache" -> hicolor-icon-theme setup-hook
* python.pkgs.pytest: setupHook to prevent creation of .pytest-cache folder, fixes #40273
When `py.test` was run with a folder as argument, it would not only
search for tests in that folder, but also create a .pytest-cache folder.
Not only is this state we don't want, but it was also causing
collisions.
* parity-ui: fix after merge
* python.pkgs.pytest-flake8: disable test, fix build
* Revert "meson: 0.46.1 -> 0.47.0"
With meson 0.47.0 (or 0.47.1, or git)
things are very wrong re:rpath handling
resulting in at best missing libs but
even corrupt binaries :(.
When we run patchelf it masks the problem
by removing obviously busted paths.
Which is probably why this wasn't noticed immediately.
Unfortunately the binary already
has a long series of paths scribbled
in a space intended for a much smaller string;
in my testing it was something like
lengths were 67 with 300+ written to it.
I think we've reported the relevant issues upstream,
but unfortunately it appears our patches
are what introduces the overwrite/corruption
(by no longer being correct in what they assume)
This doesn't look so bad to fix but it's
not something I can spend more time on
at the moment.
--
Interestingly the overwritten string data
(because it is scribbled past the bounds)
remains in the binary and is why we're suddenly
seeing unexpected references in various builds
-- notably this is is the reason we're
seeing the "extra-utils" breakage
that entirely crippled NixOS on master
(and probably on staging before?).
Fixes #43650.
This reverts commit 305ac4dade.
(cherry picked from commit 273d68eff8)
Signed-off-by: Domen Kožar <domen@dev.si>
`depsHostBuild` is not a thing, would never be a thing per the rules,
and isn't used anywhere. This is just my typo, hitherto unnoticed
because "host -> host" dependencies are by far the most obscure form.
HOST_PATH contains the path of the host package. This will include the
packages listed in buildInputs & depsHostHost. Use this to find
runtime commands that the host needs.
For instance to find the runtime version of perl,
$ PATH="$HOST_PATH" command -v perl
/nix/store/...-perl-5.28.0-aarch64-unknown-linux-android/bin/perl
This path should not be executed directly (it will break for cross
compilation). Only use it to find the location of executables that
will be run by your host system. Your build tools will, as always, be
available on the default PATH.
The line was essentially checking whether /bin/sh exists and is
executable and if that's the case, the isScript function returns
successfully.
When asking the author of this line on IRC it seems that even they can't
remember or imagine what this was supposed to be.
In summary: Whenever /bin/sh doesn't exist during a build, *any* file
given to isScript is reported as being a script even if it isn't.
This is kinda counter-intuitive and not something what somebody would
expect from a function called "isScript".
Signed-off-by: aszlig <aszlig@nix.build>
Cc: @edolstra
Not only does the suffix unnecessarily reduce sharing, but it also breaks
unpacker setup hooks (e.g. that of `unzip`) which identify interesting tarballs
using the file extension.
This also means we can get rid of the splicing hacks for fetchers.
It wasn’t exactly clear which NDK you were using previously. This adds
an attribute to system that handles what version of the NDK we should
use when building things.
/cc @Ericson2314
We already did them on non-mass-rebuild llvm 6. Also, this allows
simplifying the stdenv booting.
We were missing the libcxxabi dep in compile-rt in llvm 6, so fixed that
too.
It may seem nice and abstract to just override the default version, but
that breaks the alias relationship where the original llvmPackages_* is
no longer in sync. Put another away, modifying the referee rather
instead of breaking the reference "copy-on-write" is impossible.
We want `buildPackages` to be almost the same as
`buildPackages.buildPackges`, but that is only true if most packages
don't care about the target platform. The commented code however made
them all care about whether the target platform was Darwin.
The hack of using `crossConfig` to enforce stricter handling of
dependencies is replaced with a dedicated `strictDeps` for that purpose.
(Experience has shown that my punning was a terrible idea that made more
difficult and embarrising to teach teach.)
Now that is is clear, a few packages now use `strictDeps`, to fix
various bugs:
- bintools-wrapper and cc-wrapper
Note that a bunch of non-python packages use this attribute already.
Some of those are clearly unaware of the fact that this attribute does
not exists in stdenv because they define it but don't to add it to
their `bulidInputs` :)
Also note that I use `buildInputs` here and only handle regular
builds because python and haskell builders do it this way and I'm not
sure how to properly handle the cross-compilation case.
As in:
$ nix eval -f . bash
Also remove the glibc propagation inherit that made these necessary,
stages handle propagating libc themselves (apparently) and
AFAICT no hashes are changed as a result of this.
Following legacy packing conventions, `isArm` was defined just for
32-bit ARM instruction set. This is confusing to non packagers though,
because Aarch64 is an ARM instruction set.
The official ARM overview for ARMv8[1] is surprisingly not confusing,
given the overall state of affairs for ARM naming conventions, and
offers us a solution. It divides the nomenclature into three levels:
```
ISA: ARMv8 {-A, -R, -M}
/ \
Mode: Aarch32 Aarch64
| / \
Encoding: A64 A32 T32
```
At the top is the overall v8 instruction set archicture. Second are the
two modes, defined by bitwidth but differing in other semantics too, and
buttom are the encodings, (hopefully?) isomorphic if they encode the
same mode.
The 32 bit encodings are mostly backwards compatible with previous
non-Thumb and Thumb encodings, and if so we can pun the mode names to
instead mean "sets of compatable or isomorphic encodings", and then
voilà we have nice names for 32-bit and 64-bit arm instruction sets
which do not use the word ARM so as to not confused either laymen or
experienced ARM packages.
[1]: https://developer.arm.com/products/architecture/a-profile
(cherry picked from commit ba52ae5048)
Following legacy packing conventions, `isArm` was defined just for
32-bit ARM instruction set. This is confusing to non packagers though,
because Aarch64 is an ARM instruction set.
The official ARM overview for ARMv8[1] is surprisingly not confusing,
given the overall state of affairs for ARM naming conventions, and
offers us a solution. It divides the nomenclature into three levels:
```
ISA: ARMv8 {-A, -R, -M}
/ \
Mode: Aarch32 Aarch64
| / \
Encoding: A64 A32 T32
```
At the top is the overall v8 instruction set archicture. Second are the
two modes, defined by bitwidth but differing in other semantics too, and
buttom are the encodings, (hopefully?) isomorphic if they encode the
same mode.
The 32 bit encodings are mostly backwards compatible with previous
non-Thumb and Thumb encodings, and if so we can pun the mode names to
instead mean "sets of compatable or isomorphic encodings", and then
voilà we have nice names for 32-bit and 64-bit arm instruction sets
which do not use the word ARM so as to not confused either laymen or
experienced ARM packages.
[1]: https://developer.arm.com/products/architecture/a-profile
This allows one to force a compiler to use native machine optimizations. This
goes contrary to all the usual guarantees of Nix and so should be used only by
end-user and only in specific cases when they know what are they doing.
In my case this is needed to get a noticeable FPS boost in RPCS3 which is very
CPU-hungry PlayStation 3 emulator.
- `localSystem` is added, it strictly supercedes system
- `crossSystem`'s description mentions `localSystem` (and vice versa).
- No more weird special casing I don't even understand
TEMP
Since at least d7bddc27b2, we've had a
situation where one should depend on:
- `stdenv.cc.bintools`: for executables at build time
- `libbfd` or `libiberty`: for those libraries
- `targetPackages.cc.bintools`: for exectuables at *run* time
- `binutils`: only for specifically GNU Binutils's executables,
regardless of the host platform, at run time.
and that commit cleaned up this usage to reflect that. This PR flips the
switch so that:
- `binutils` is indeed unconditionally GNU Binutils
- `binutils-raw`, which previously served that role, is gone.
so that the correct usage will be enforced going forward and everything
is simple.
N.B. In a few cases `binutils-unwrapped` (which before and now was
unconditionally actual GNU binutils), rather than `binutils` was used to
replace old `binutils-raw` as it is friendly towards some cross
compilation usage by avoiding a reference to the next bootstrapping
change.
First, we need check against the host platform, not the build platform.
That's simple enough.
Second, we move away from exahustive finite case analysis (i.e.
exhaustively listing all platforms the package builds on). That only
work in a closed-world setting, where we know all platforms we might
build one. But with cross compilation, we may be building for arbitrary
platforms, So we need fancier filters. This is the closed world to open
world change.
The solution is instead of having a list of systems (strings in the form
"foo-bar"), we have a list of of systems or "patterns", i.e. attributes
that partially match the output of the parsers in `lib.systems.parse`.
The "check meta" logic treats the systems strings as an exact whitelist
just as before, but treats the patterns as a fuzzy whitelist,
intersecting the actual `hostPlatform` with the pattern and then
checking for equality. (This is done using `matchAttrs`).
The default convenience lists for `meta.platforms` are now changed to be
lists of patterns (usually a single pattern) in
`lib/systems/for-meta.nix` for maximum flexibility under this new
system.
Fixes #30902
Resolved the following conflicts (by carefully applying patches from the both
branches since the fork point):
pkgs/development/libraries/epoxy/default.nix
pkgs/development/libraries/gtk+/3.x.nix
pkgs/development/python-modules/asgiref/default.nix
pkgs/development/python-modules/daphne/default.nix
pkgs/os-specific/linux/systemd/default.nix
We go out of our way (see top of file) to build a single binary
with symlinks for all of the tools, but were losing them
when preparing the bootstrap tools.
For the cc of the intermediate stages, to be precise. Doing the same for
bintools requires lots of refactoring.
This is mainly for the future extensibility as now you can change
documentation generation with impunity without rebuilding the
whole of stdenv.
Existing "mips64el" should be "mipsel".
This is just the barest minimum so that nixpkgs can recognize them as
systems - although required for building individual derivations onto
MIPS boards, it is not sufficient if you want to actually build nixos on
those targets
Aarch64 tools tested briefly with qemu-aarch64,
but neither have been actually used yet :).
For now only "host" indirectly via binary cache
at cache.allvm.org.
This is a temporary workaround to make `nix-env -qa` and `nix search` ignore
broken packages as they they did before this patchset.
This patch should be reverted after `nix` gets a proper fix for this.
See NixOS/nix#1771.
This option makes `meta.evaluate` into a close approximation of the result of
evaluating `.outPath` by checking all the dependencies recursively at a cost of
2x slowdown. Note that actually evaluating `.outPath` costs some
5x-7x more because `.outPath` also computes all the hashes.
I hope this will be a temporary measure. If there is consensus around
issue #33599, then we can follow an explicit `dontCheck`, but default to
not checking during cross builds when none is given.
This pushes check-meta evaluation to derivation evaluation step, leaving all other
attributes accessible.
Before this commit:
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.
as expected
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.name --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.meta.description --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.
which is unfortunate since its impossible to use packages in autogenerated
documentation on all platforms.
After this commit:
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen --argstr system aarch64-linux
still fails
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.name --argstr system aarch64-linux
> "xen-4.5.5"
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.meta.description --argstr system aarch64-linux
> "Xen hypervisor and related components (vanilla)"
- All deps go on the PATH
- CC and Bintools wrappers with their host != depender's host still get their
setup hooks run.
- Environment hooks get applied to all packages
This isn't so elegent, but eases the transition on a very significant
PR.
We now have the information to properly determine the role the
cc-wrapper dependency has, by taking advantage of `offset`. No longer
use the soon-to-be-deprecated crossConfig environment variable, the
temp hack used before this change.
4 far-reaching changes: Smaller PATH, New vars, different propagation
logic, and different hook logic
Smaller PATH
------------
`buildInputs` no longer go on the PATH at build time, as they cannot be
run when cross compiling and we don't want to special case. Simply make
a `nativeBuildInput` too if one needs them on the PATH. Fixes #21191.
Many new depedendency variables
-------------------------------
See the stdenv chapter of the nixpkgs manual. I pulled out the existing
documentation of dependency specification into a new section, and added
language for these two (and their propagated equivalents) along side
the others'.
More complex propagation logic
------------------------------
Before a propagated*XXX*Input always acted as if it was specified
directly as a *XXX*Input downstream. That's simple enough, but violates
the intended roles of each sort of dep, which has functional and not
just stylistic consequences.
The new algorithm is detailed in the manual, and ensures everything
ends up in the right place. I tried to give both an informal and formal
description, but I suspect in practice it will not make much sense
until one tries cross compiling, after which it will immediately make
sense as the only sane option.
Simplified hook logic
---------------------
Rather than `envHook` and `crossEnvHook`, whose behavior differs
depending on whether we are cross compiling or not, there is now one
hook per sort (or rather non-propagated and propagated pair of sorts)
of dependency. These new hooks have the same meaning regardless of
cross compilation. See the setup hook section of stdenv chapter of the
Nixpkgs manual for more details.
stdenvNoCC should not inject any C++ standard library, just as it
doesn't inject any C standard library. stdenv still does, but only
indirectly through stdenv.cc. Wrapped clangs can be simplified now that
they don't need to worry about clobbering CoreFoundation when replacing
the C++ standard library implementation.
This generally-good cleanup should assist with debugging some C++
failures in #26805.
- tracing seems annoying enough
- we get errors for all packages instead of aborting on the first one
- easier to differentiate from unwanted packages (broken, unfree, etc.)
This continues #23374, which always kept around both attributes, by
always including both propagated files: `propgated-native-build-inputs`
and `propagated-build-inputs`. `nativePkgs` and `crossPkgs` are still
defined as before, however, so this change should only barely
observable.
This is an incremental step to fully keeping the dependencies separate
in all cases.
I find the separation of concerns, accumulating, then processing, easier
to follow. Also, with my yet-to-be-merged cross work, the accumulation
part will become more complex.
One should do this when needed executables at run time. It is more
honest and cross-friendly than refering to binutils directly, if one
neeeds the default binary tools for the target platform, rather than
binutils in particular.
This requires some small changes in the stdenv, then working around the
weird choice LLVM made to hardcode @rpath in its install name, and then
lets us remove a ton of annoying workaround hacks in many of our Go
packages. With any luck this will mean less hackery going forward.
cc-wrapper may wrap a cc-compiler, but it doesn't need one to build
itself. (c.f. expand-response-params is a separate derivation.) This
helps avoid cycles on the cross stuff, in addition to removing a
useless dependency edge.
I could have been super careful with overrides in the stdenv to avoid
the mass rebuild, but I don't think it's worth it.
Why 6? It seems a decently high number, giving us room for more degrees
of debugging before the `set -x` sledgehammer without incurring a
mass-rebuild.
By default, all previous overrides are discarded as before, as they
would only apply to the old host platform. But sometimes it is useful to
add some new ones, and this optional parameter allows that.
This is needed when cross-compiling for iOS (Aarch64 + Darwin). I also
changed the syntax of the Linux stdenv for visual consistency, though
that has no effect on semantics as the os is already guaranteed to be
Linux.
This reverts commit 0a944b345e, reversing
changes made to 61733ed6cc.
I dislike these massive stdenv changes with unclear motivation,
especially when they involve gratuitous mass renames like NIX_CC ->
NIX_BINUTILS. The previous such rename (NIX_GCC -> NIX_CC) caused
months of pain, so let's not do that again.
Environment variable filter in substituteAll was not precise and produced
undefined and invalid variable names. Vladimír Čunát tried to fix that in [1],
but `env -0` did not work during Darwin bootstrap, so [2] reverted this change
and replaced an error due to invalid variables with a warning. Recently in #28057
John Ericson added `set -u` to `setup.sh` and undefined variables made the setup
fail during e.g. `nix-build -A gnat` with `setup: line 519: !varName: unbound
variable`.
[1] 62fc8859c1
[2] 81df035429
This becomes necessary if more wrappers besides cc-wrapper start
supporting hardening flags. Also good to make the warning into an
error.
Also ensure interface is being used right: Not as a string, not just in
bash.
- Don't build with libsigsegv by default. The build apparently attempted
to link against it, but it never retained the reference anyway...
- Side effect: stdenv bootstrapping needs no libsigsegv anymore.
- Run checks, but only in the interactive gawk by default on Linux,
so that stdenv bootstrap isn't slowed down (by glibc locales, etc.).
- xz should be no longer needed in inputs, as we have it in stdenvs now.
The whole change was triggered by some used kernel versions still
breaking libsigsegv tests #28464.
Older bash version, like those in the bootstrap tools and on macOS,
currently confuse variables defined as an empty array with undefined
variables. `${foo+"${foo[@]}"}` will prevent `set -u` problems with
empty arrays and older without making a single '' in the empty case.
Care is taken to `set +u` when running hooks so as to not break existing
packages.
This reverts commit eeabf85780.
This change suddenly makes tons of stdenv internals visible in
nativeBuildInputs of every derivation, which doesn't seem desirable.
E.g:
````
nix-repl> hello.nativeBuildInputs
[ «derivation /nix/store/bcfkyf6bhssxd2vzwgzmsbn7b5b9rpxc-patchelf-0.9.drv»
«derivation /nix/store/4wnshnz9wwanpfzcrdd76rri7pyqn9sk-paxctl-0.9.drv»
<< snip 10+ lines >>
«derivation /nix/store/d35pgh1lcg5nm0x28d899pxj30b8c9b2-gcc-wrapper-6.4.0.drv»
]
````
Additionally, instead of pulling them from `setup.sh`, route them via
Nix. This gets us one step closer to making stdenv be a plain attribute
set instead of a derivation.
Currently the logic of generating nixpkgs Hydra jobs is to walk through
the pkgs evaluated for system = "x86_64-linux", collect any derivations
and their meta.platforms values. However, that doesn't work for
packages whose meta.platforms doesn't include x86_64-linux, as just
evaluating their meta attribute raises an error so they get skipped
completely.
As a less-intrusive fix (i.e. anything than rewriting the current package
enumeration logic), allow passing `config.allowUnsupportedSystem = true`
to permit evaluating packages regardless of their platform and use that
in the package listing phase.
Fixes #25200
@vcunat and others rightly point out that it's easier to quote always,
than learn Bash's idiosyncrasies enough to know when it doesn't make a
difference.
This reverts commit 2743078f66, which
removes quotes that don't do anything, and then goes further adding
even more quotes.
The logic was made pure for the normal libSystem, but this change never
made it to the bootstrap tools. Deduplication the logic as the comment
suggests would have prevented this, but here's a stop-gap until we do
so.
It's better than the eval solution this is adding back, but until we can
rely on a particular version of bash in nix-shell, this just breaks too
much stuff.
See c94f3d5575
and https://github.com/NixOS/nix/pull/1483 for the better long-term
solution.
As @oxij points out in [1], this breakage is especially serious because
it changes the contents of built environments without a corresonding
change in their hashes. Also, the revert is easier than I thought.
This reverts commit 3cb745d5a6.
[1]: https://github.com/NixOS/nixpkgs/pull/27427#issuecomment-317293040
