buildRustCrate has a handy `include` helper, that only imports those whitelisted
files and folders to the store.
However, the function's matching logic is broken and includes all files,
regardless of whether or not they're whitelisted, as long as the whitelist
contains at least one name (regardless of whether that name exists). This is
because it doesn't take into account that
`lib.strings.removePrefix "foo" "bar" == "bar"` (that is, paths that don't match
the prefix are passed straight through).
This rare sitation was caught when building zoom-us package:
```
automatically fixing dependencies for ELF files
/nix/store/71d65fplq44y9yn2fvkpn2d3hrszracd-auto-patchelf-hook/nix-support/setup-hook: line 213: echo: write error: Broken pipe
/nix/store/71d65fplq44y9yn2fvkpn2d3hrszracd-auto-patchelf-hook/nix-support/setup-hook: line 210: echo: write error: Broken pipe
```
The worst is that derivation continued and resulted into broken package:
https://github.com/NixOS/nixpkgs/pull/55566#issuecomment-470065690
I hope, replacing `grep -q` with `grep` will remove this race condition.
When a package provides both executables and gio modules, it is quite
probable the executables will need those modules. wrapGAppsHook wraps
executables with GIO_EXTRA_MODULES picked up from dependencies
but forgets about the package being built. Let’s add to consideration.
Closes: https://github.com/NixOS/nixpkgs/issues/50254
On very large graphs (14k+ paths), we'd end up with a massive in
memory tree of mostly duplication.
We can safely cache trees and point back to them later, saving
memory.
While it is not obvious from the source, cargo sets CARGO_MANIFEST_DIR to an absolute directory. This let to a build problem with the popular "tera" crate using the "pest" crate.
## Cargo details
The variable is set here:
f7c91ba622/src/cargo/core/compiler/compilation.rs (L229)
and computed from the `manifest_path`:
f7c91ba622/src/cargo/core/package.rs (L163)
The manifest path is also exported via `cargo metadata` where you can see that it is absolute.
Whenever we create scripts that are installed to $out, we must use runtimeShell
in order to get the shell that can be executed on the machine we create the
package for. This is relevant for cross-compiling. The only use case for
stdenv.shell are scripts that are executed as part of the build system.
Usages in checkPhase are borderline however to decrease the likelyhood
of people copying the wrong examples, I decided to use runtimeShell as well.
The execlineb program is the launcher (and lexer) of execline scripts.
So it makes a lot of sense to have all the small tools in scope by
default.
We append to the end of PATH so that they can be easily overwritten by
the user.
Co-authored-by: Alyssa Ross <hi@alyssa.is>
The appimageTools attrset contains utilities to prevent
the usage of appimage-run to package AppImages, like done/attempted
in #49370 and #53156.
This has the advantage of allowing for per-package environment changes,
and extracts into the store instead of the users home directory.
The package list was extracted into appimageTools to prevent
duplication.
bcf54ce5bb introduced a treewide change to
use ${stdenv.shell} where-ever possible. However, this broke a script
used by dockerTools, store-path-to-layer.sh, as it did not preserve the
+x mode bit. This meant the file got put into the store as mode 0444,
resulting in a build-time error later on that looked like:
xargs: /nix/store/jixivxhh3c8sncp9xlkc4ls3y5f2mmxh-store-path-to-layer.sh: Permission denied
However, in a twist of fate, bcf54ce5bb
not only introduced this regression but, in this particular instance,
didn't even fix the original bug: the store-path-to-layer.sh script
*still* uses /bin/sh as its shebang line, rather than an absolute path
to stdenv. (Fixing this can be done in a separate commit.)
Signed-off-by: Austin Seipp <aseipp@pobox.com>
The original change in #55372 was supposed to fix the case where a store
path which is a file should be placed into `buildEnv` which broke with a
fairly misleading Perl error.
Unfortunately this introduced a regression, `findFiles` can have targets
that are files if the file isn't a store path. Rather than adding more
obscure checks with probably further regressions, I figured that it's
better to replicate the behavior of `lib.isStorePath` and explicitly
check if the store path is a file and break in this case only.
This should also fix recent staging issues.
I noticed by creating `buildEnv` where I accidentally put a derivation
from `pkgs.writeText` into `paths` and got a broken build with the
following misleading error message:
```
Use of uninitialized value $stat1 in numeric ne (!=) at /nix/store/9g4wc31j7a2xp22xpgwr0qssfxahxdzl-builder.pl line 74.
Use of uninitialized value $stat1 in bitwise and (&) at /nix/store/9g4wc31j7a2xp22xpgwr0qssfxahxdzl-builder.pl line 75.
different permissions in `' and `/nix/store/0vy5ss91laxvwkyvrbld5hv27i88qk5w-noise': 0000 <-> 0444 at /nix/store/9g4wc31j7a2xp22xpgwr0qssfxahxdzl-builder.pl line 75.
```
It can be reproduced with an expression like this:
``` nix
{ pkgs ? import <nixpkgs> { } }:
let
file = pkgs.writeText "test" ''
content
'';
in
pkgs.buildEnv {
name = "test-env";
paths = [ /* ... */ file ];
}
```
In combination with carnix we can now build crates that require a
specific edition of rust features. A few crates started requiring that
already and having this in nixpkgs is just logical.
This patch preserves the ordering of layers of a parent image when the
new image is packed.
It is currently not the case: layers are stacked in the reverse order.
Fixes#55290
This round is without the systemd CVE,
as we don't have binaries for that yet.
BTW, I just ignore darwin binaries these days,
as I'd have to wait for weeks for them.
Emacs loads all the elisp files in the top-level of the site-lisp
directory. However some packages (e.g. mu4e) put their elisp files in a
subdirectory of site-lisp. Emacs will not load these packages unless
subdirs.el is present.
This commit links the subdirs.el file from the emacs package into the
emacs-package-deps package so that packages that put their elisp files
in a subdirectory of site-lisp are loaded.
Comments on conflicts:
- llvm: d6f401e1 vs. 469ecc70 - docs for 6 and 7 say the default is
to build all targets, so we should be fine
- some pypi hashes: they were equivalent, just base16 vs. base32
Unless dontWrapGapps is set, the wrap-gapps-hook.sh will currently
wrap all executables (and symbolic links to executables) found under
the target directories: bin and libexec.
As a result, if a symbolic link in a target directory points to an
executable in a target directory, both will get wrapped. This
causes an extra shell/exec when following the symbolic link,
as well as increasing the size of the final executable's environment.
To avoid wrapping a link to an already wrapped executable, this
commit splits the determination of what gets wrapped into two phases:
1. All binaries under the target directories are wrapped and logged
with "Wrapping program ..."
2. All links to executables under the target directories are
identified and checked to see if they reference an executable
under one of the target directories.
If yes, the required wrapping has already been performed on
the associated binary (in phase 1), so no wrapping is done
and "Not wrapping link: ... (already wrapped)" is logged.
If no, the link points at an executable that hasn't been
wrapped, so the link is wrapped and "Wrapping link: ..." is logged.
As an example, the yelp package has a bin directory that contains
an executable "yelp" and a symbolic link "gnome-help" -> "yelp".
Prior to this commit, the bin directory would contain these files
after wrapping:
gnome-help -- wrapper to exec .gnome-help-wrapped
.gnome-help-wrapped -- a symbolic link to yelp
yelp -- wrapper to exec .yelp-wrapped
.yelp-wrapped -- the original yelp binary
After this commit, the bin directory will instead contain:
gnome-help -- a symbolic link to yelp
yelp -- wrapper to exec .yelp-wrapped
.yelp-wrapped -- the original yelp binary
NOTE: The primary motivation for this commit is to avoid obscuring
the fact that two or more paths are simple aliases and expected to
behave identically. It also reduces the likelihood of hitting
limits related to environment variable size.
LIMITATION: The method used above is intended to be conservative
and will still wrap symbolic links to other symbolic links when
the ultimate target is outside of bin or libexec.
There is a function params `kernel' intended to specify which kernel to use.
It defaults to `pkgs.linux`.
But when we override `kernel', compiling and using two kernels seems not to be the intendend bevavior.
clang needs to find headers + libraries for compiling with libc++. We
need to add a libcxx argument to cc-wrapper. This means you do not
have to pass in c++ headers directly.
This resolves the last case remaining of #30670. Darwin clang++ now
works properly.
Fixes#30670
This is useful when running tools like NixOps or nix-review
on workstations where the upload to the builder is significantly
slower then downloading the source on the builder itself.
* add generic x86_32 support
- Add support for i386-i586.
- Add `isx86_32` predicate that can replace most uses of `isi686`.
- `isi686` is reinterpreted to mean "exactly i686 arch, and not say i585 or i386".
- This branch was used to build working i586 kernel running on i586 hardware.
* revert `isi[345]86`, remove dead code
- Remove changes to dead code in `doubles.nix` and `for-meta.nix`.
- Remove `isi[345]86` predicates since other cpu families don't have specific model predicates.
* remove i386-linux since linux not supported on that cpu
* fetchurl: fix and add extra KDE mirrors (#51480)
- The gwdg.de mirror has moved the relative path of its KDE tarballs
- Add new mirrors from Berkeley and MIT, which are on the list of officially supported mirrors
https://download.kde.org/extra/download-mirrors.html
* More KDE mirror touchups
- The official one is a redirect to HTTPS anyways, so go directly to HTTPS
- Add China USTC for Asian users
- Swap Finland university from FTP to HTTP
* fetchurl: keep one ftp KDE mirror
Docker images used to be, essentially, a linked list of layers. Each
layer would have a tarball and a json document pointing to its parent,
and the image pointed to the top layer:
imageA ----> layerA
|
v
layerB
|
v
layerC
The current image spec changed this format to where the Image defined
the order and set of layers:
imageA ---> layerA
|--> layerB
`--> layerC
For backwards compatibility, docker produces images which follow both
specs: layers point to parents, and images also point to the entire
list:
imageA ---> layerA
| |
| v
|--> layerB
| |
| v
`--> layerC
This is nice for tooling which supported the older version and never
updated to support the newer format.
Our `buildImage` code only supported the old version, so in order for
`buildImage` to properly generate an image based on another image
with `fromImage`, the parent image's layers must fully support the old
mechanism.
This is not a problem in general, but is a problem with
`buildLayeredImage`.
`buildLayeredImage` creates images with newer image spec, because
individual store paths don't have a guaranteed parent layer. Including
a specific parent ID in the layer's json makes the output less likely
to cache hit when published or pulled.
This means until now, `buildLayeredImage` could not be the input to
`buildImage`.
The changes in this PR change `buildImage` to only use the layer's
manifest when locating parent IDs. This does break buildImage on
extremely old Docker images, though I do wonder how many of these
exist.
This work has been sponsored by Target.
This ensures that RPATH entries like "/foo/build/bar" doesn't trigger a
match when TMPDIR is "/build/bar". (I've had this problem with a
prebuilt package.)
If there was no older file than $NIX_BUILD_TOP this would result in a
warning, e.g. with nix-info.
```
/nix/store/15kgcm8hnd99p7plqzx7p4lcr2jni4df-set-source-date-epoch-to-latest.sh: line 13: [: : integer expression expected
```
This commit causes melpaBuild to use package-build from melpa/package-build
instead of melpa/melpa. Development of package-build happens in the former
repository whereas the latter is much larger, containing also the MELPA
recipes. We do not need to fetch the MELPA recipes from melpa/melpa, as we fetch
them one-by-one for Nixpkgs.
No real function change here, but this updates the trivial and melpa builders to
be formatted more consistently with the rest of the builders, and swaps
`eval "$preBuild"` for the more standard `runHook preBuild`.
If the file in question is not a shared object file but an ELF, we
really want to skip the file, because we won't have anything to patch
there.
For example if the file is created via "gcc -c -o foo.o foo.c", we don't
get a segment header and so far autoPatchelf was trying to patch such a
file.
By checking for missing segment headers, we're now no longer going to
attempt patching such a file.
Signed-off-by: aszlig <aszlig@nix.build>
Reported-by: Sander van der Burg <svanderburg@gmail.com>
While declaring it as an array doesn't do any harm in our usage, it
might be a bit confusing when reading the code.
Signed-off-by: aszlig <aszlig@nix.build>
This function is useful if autoPatchelf is invoked during some of the
phases of a build and allows to add arbitrary shared objects to the
search path.
So far the same functionality was in autoPatchelf itself, but not
available as a separate function, so when adding shared objects to the
dependency cache one would have to do so manually.
The function also has the --no-recurse flag, which prevents recursing
into subdirectories.
Signed-off-by: aszlig <aszlig@nix.build>
This is to be used with the autoPatchelf command and allows to only
patch a specific file or directory without recursing into
subdirectories.
Apart from being able to run the command in a standalone way, as
detailled in the previous commit this is also needed for the Android SDK
emulator, because according to @svanderburg there are subdirectories we
don't want to patch.
The reason why I didn't use GNU getopt is that it might not be available
on all operating systems and the getopts bash builtin doesn't support
long arguments. Apart from that, the implementation for recognizing the
flag is pretty trivial and it's also using bash builtins only, so if we
want to do something really fancy someday, we can still change it.
Signed-off-by: aszlig <aszlig@nix.build>
If you want to only run autoPatchelf on a specific path and leave
everything else alone, we now have a $dontAutoPatchelf environment
variable, which causes the postFixup hook to not run at all.
The name "dontAutoPatchelf" probably is a bit weird in conjunction with
putting "autoPatchelfHook" in nativeBuildInputs, but unless someone
comes up with a better name I keep it that way because it's consistent
with all the other dontStrip, dontPatchShebangs, dontPatchELF and
whatnot.
A specific example where this is needed is when building the Android SDK
emulator, which contains a few ARM binaries in subdirectories that
should not be patched. If we were to run autoPatchelf on all outputs
unconditionally we'd run into errors because some ARM libraries couldn't
be found.
Signed-off-by: aszlig <aszlig@nix.build>
The autoPatchelf main function which is run against all of the outputs
was pretty much tailored towards this specific setup-hook and was
relying on $prefix to be set globally.
So if you wanted to run autoPatchelf manually - let's say during
buildPhase - you would have needed to run it like this:
prefix=/some/directory autoPatchelf
This is now more intuitive and all you need to do is run the following:
autoPatchelf /some/directory
Signed-off-by: aszlig <aszlig@nix.build>
On Linux the `$TMPDIR` is `/build`. The TMPDIR audit looks for `$TMPDIR`
in the build output, which will then fail with packages like
/buildkite-agent.
This fixes the heuristic to look for `$TMPDIR/` instead.
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.
This reverts commit bde99096a8.
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.
This reverts commit eb7c50a993.
This allows to simplify the usage of libredirect inside of nix build
sandboxes. Add "libredirect.hook" to the build inputs to get everything
linked in automaticall. All that's left is to set NIX_REDIRECTS and call
the target program.
Since Nix 2 is now the stable Nix version, we can use closureInfo
which simplifies the Nix database initialisation (size and hash are
included in the "dump").
Pull request #50246 was merged a bit too quickly and it was supposed to
fix libredirect on Darwin. However it still failed on Darwin and this
was missed by the person merging the pull request.
The reason this was failing was that there is no __xstat* on Darwin.
So I'm adding a wrapper for stat() as well as it works on Darwin and it
still doesn't hurt on GNU/Linux.
Signed-off-by: aszlig <aszlig@nix.build>
Cc: @edolstra, @zimbatm
This is just a sanity check on whether the library correctly wraps the
syscalls and it's using the "true" executable for posix_spawn() and
execv().
The installCheckPhase is not executed if we are cross-compiling, so this
shouldn't break cross-compilation.
One thing I'm not actually sure is whether ${coreutils}/bin/true is
universally available on all the platforms, nor whether all the
functions we use in the test are available, but we can still fix that
after we've found out about that.
Signed-off-by: aszlig <aszlig@nix.build>
This is to make sure we get the correct shared library suffix of the
target platform. While for example on Darwin it would even work with the
hardcoded .so prefix it's IMHO a bit nicer to have the actual native
extension.
Signed-off-by: aszlig <aszlig@nix.build>
The library can be used also on Darwin using it like this:
NIX_REDIRECTS='foo=bar' \
DYLD_INSERT_LIBRARIES=${libredirect}/lib/libredirect.so \
DYLD_FORCE_FLAT_NAMESPACE=1 \
some_program
So let's actually not hardcade gcc and add Darwin to meta.platforms.
No other changes seem to be required.
Signed-off-by: aszlig <aszlig@nix.build>
This adds the shell shebang to the wrapper script. Without this,
emacs and in particular agda2-mode (but probably other applications as
well) return a format error when trying to execute agda.
With the previous commit `propagateDoc` is now always given the correct value
(i.e. it is never set to `true` when there are no `man` and `info` outputs).
Hence, we can simply symlink the original outputs to the wrapper outputs.
Pros:
- simpler, less indirection compared to `propagated-user-env-packages`,
- uses less inodes (1 symlink, which nix then simply automatically resolves
and removes, vs. two directories and a file),
- makes direct references like "export MANPATH=${stdenv.cc.man}/share/man"
simply work.
Cons:
- I'm not aware of any.
This and the previous commit together almost completely revert commits
fde7296a47,
fa41297209, and
c981787db9.
I originally thought it would just be enough to just check for an INTERP
section in isExecutable, however this would mean that we don't detect
statically linked ELF files, which would break our recent improvement to
gracefully handle those.
In theory, we are only interested in ELF files that have an INTERP
section, so checking for INTERP would be enough. Unfortunately the
isExecutable function is already used outside of autoPatchelfHook, so we
can't easily get rid of it now, so let's actually strive for more
correctness and make isExecutable actually match ELF files that are
executable.
So what we're doing instead now is to check whether either the ELF type
is EXEC *or* we have an INTERP section and if one of them is true we
should have an ELF executable, even if it's statically linked.
Along the way I also set LANG=C for the invocations of readelf, just to
be sure we don't get locale-dependent output.
Tested this with the following command (which contains almost[1] all the
packages using autoPatchelfHook), checking whether we run into any
library-related errors:
nix-build -E 'with import ./. { config.allowUnfree = true; };
runCommand "test-executables" {
drvs = [
anydesk cups-kyodialog3 elasticsearch franz gurobi
masterpdfeditor oracle-instantclient powershell reaper
sourcetrail teamviewer unixODBCDrivers.msodbcsql17 virtlyst
vk-messenger wavebox zoom-us
];
} ("for i in $drvs; do for b in $i/bin/*; do " +
"[ -x \"$b\" ] && timeout 10 \"$b\" || :; done; done")
'
Apart from testing against library-related errors I also compared the
resulting store paths against the ones prior to this commit. Only
anydesk and virtlyst had the same as they didn't have self-references,
everything else differed only because of self-references, except
elasticsearch, which had the following PIE binaries:
* modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/autoconfig
* modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/autodetect
* modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/categorize
* modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller
* modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/normalize
These binaries were now patched, which is what this commit is all about.
[1]: I didn't include the "maxx" package (MaXX Interactive Desktop)
because the upstream URLs are no longer existing and I couldn't
find them elsewhere on the web.
Signed-off-by: aszlig <aszlig@nix.build>
Fixes: https://github.com/NixOS/nixpkgs/issues/48330
Cc: @gnidorah (for MaXX Interactive Desktop)
In strictDeps=false, autoPatchshebangs should use
--build (corresponding to PATH) to lookup commands. This restores the
previous behavior of patchshebangs so that we don’t break stuff that
isn’t careful in the buildInputs vs. nativeBuildInputs distinction.
Unfortunately this won’t work under cross compilation.
- respect libc’s incdir and libdir
- make non-unix systems single threaded
- set LIMITS_H_TEST to false for avr
- misc updates to support new libc’s
- use multilib with avr
For threads we want to use:
- posix on unix systems
- win32 on windows
- single on everything else
For avr:
- add library directories for avrlibc
- to disable relro and bind
- avr5 should have precedence over avr3 - otherwise gcc uses the wrong one
Usuage: Add breakpointHook to your `buildInputs` like this:
stdenv.mkDerivation rec {
# ...
buildInputs = [ breakpointHook ];
});
When the build fails as show in this example:
pkgs.hello.overrideAttrs (old: {
buildInputs = [ breakpointHook ];
postPatch = ''
false
'';
});
It will halt execution printing the following message:
build failed in patchPhase with exit code 1
To attach to this build run the following command as root:
cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10
Installing cntr and running the command will provide shell access to the
build sandbox of failed build:
sudo cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10
WARNING: bad ownership on /nix/var/nix/profiles/per-user/root, should be 1000
[nixbld@localhost:/var/lib/cntr]$
At /var/lib/cntr the sandbox filesystem is mounted. All commands and
files of the system are still accessible within the shell.
To execute commands from the sandbox use the `cntr exec` subcommand.
Bazel computes the default value of output_user_root before parsing the
flag[0]. The computation of the default value involves getting the $USER
from the environment. I don't have that variable when building with
sandbox enabled.
[0]: 9323c57607/src/main/cpp/startup_options.cc (L123-L124)
Create a many-layered Docker Image.
Implements much less than buildImage:
- Doesn't support specific uids/gids
- Doesn't support runninng commands after building
- Doesn't require qemu
- Doesn't create mutable copies of the files in the path
- Doesn't support parent images
If you want those feature, I recommend using buildLayeredImage as an
input to buildImage.
Notably, it does support:
- Caching low level, common paths based on a graph traversial
algorithm, see referencesByPopularity in
0a80233487993256e811f566b1c80a40394c03d6
- Configurable number of layers. If you're not using AUFS or not
extending the image, you can specify a larger number of layers at
build time:
pkgs.dockerTools.buildLayeredImage {
name = "hello";
maxLayers = 128;
config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
};
- Parallelized creation of the layers, improving build speed.
- The contents of the image includes the closure of the configuration,
so you don't have to specify paths in contents and config.
With buildImage, paths referred to by the config were not included
automatically in the image. Thus, if you wanted to call Git, you
had to specify it twice:
pkgs.dockerTools.buildImage {
name = "hello";
contents = [ pkgs.gitFull ];
config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
};
buildLayeredImage on the other hand includes the runtime closure of
the config when calculating the contents of the image:
pkgs.dockerTools.buildImage {
name = "hello";
config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
};
Minor Problems
- If any of the store paths change, every layer will be rebuilt in
the nix-build. However, beacuse the layers are bit-for-bit
reproducable, when these images are loaded in to Docker they will
match existing layers and not be imported or uploaded twice.
Common Questions
- Aren't Docker layers ordered?
No. People who have used a Dockerfile before assume Docker's
Layers are inherently ordered. However, this is not true -- Docker
layers are content-addressable and are not explicitly layered until
they are composed in to an Image.
- What happens if I have more than maxLayers of store paths?
The first (maxLayers-2) most "popular" paths will have their own
individual layers, then layer #(maxLayers-1) will contain all the
remaining "unpopular" paths, and finally layer #(maxLayers) will
contain the Image configuration.
Using a simple algorithm, convert the references to a path in to a
sorted list of dependent paths based on how often they're referenced
and how deep in the tree they live. Equally-"popular" paths are then
sorted by name.
The existing writeReferencesToFile prints the paths in a simple
ascii-based sorting of the paths.
Sorting the paths by graph improves the chances that the difference
between two builds appear near the end of the list, instead of near
the beginning. This makes a difference for Nix builds which export a
closure for another program to consume, if that program implements its
own level of binary diffing.
For an example, Docker Images. If each store path is a separate layer
then Docker Images can be very efficiently transfered between systems,
and we get very good cache reuse between images built with the same
version of Nixpkgs. However, since Docker only reliably supports a
small number of layers (42) it is important to pick the individual
layers carefully. By storing very popular store paths in the first 40
layers, we improve the chances that the next Docker image will share
many of those layers.*
Given the dependency tree:
A - B - C - D -\
\ \ \ \
\ \ \ \
\ \ - E ---- F
\- G
Nodes which have multiple references are duplicated:
A - B - C - D - F
\ \ \
\ \ \- E - F
\ \
\ \- E - F
\
\- G
Each leaf node is now replaced by a counter defaulted to 1:
A - B - C - D - (F:1)
\ \ \
\ \ \- E - (F:1)
\ \
\ \- E - (F:1)
\
\- (G:1)
Then each leaf counter is merged with its parent node, replacing the
parent node with a counter of 1, and each existing counter being
incremented by 1. That is to say `- D - (F:1)` becomes `- (D:1, F:2)`:
A - B - C - (D:1, F:2)
\ \ \
\ \ \- (E:1, F:2)
\ \
\ \- (E:1, F:2)
\
\- (G:1)
Then each leaf counter is merged with its parent node again, merging
any counters, then incrementing each:
A - B - (C:1, D:2, E:2, F:5)
\ \
\ \- (E:1, F:2)
\
\- (G:1)
And again:
A - (B:1, C:2, D:3, E:4, F:8)
\
\- (G:1)
And again:
(A:1, B:2, C:3, D:4, E:5, F:9, G:2)
and then paths have the following "popularity":
A 1
B 2
C 3
D 4
E 5
F 9
G 2
and the popularity contest would result in the paths being printed as:
F
E
D
C
B
G
A
* Note: People who have used a Dockerfile before assume Docker's
Layers are inherently ordered. However, this is not true -- Docker
layers are content-addressable and are not explicitly layered until
they are composed in to an Image.
This causes problems for packages built using a bootstrap stdenv,
resulting in references to /bin/sh or even bootstrap-tools. The darwin
stdenv is much stricter about what requisites/references are allowed but
using /bin/sh on linux is also undesirable.
eg. https://hydra.nixos.org/build/81754896
$ nix-build -A xz
$ head -n1 result-bin/bin/xzdiff
#!/nix/store/yvc7kmw98kq547bnqn1afgyxm8mxdwhp-bootstrap-tools/bin/sh
This reverts commit f06942327a.
This reverts commit f777d2b719.
cc #34409
This breaks evaluation of the tested job:
attribute 'diskInterface' missing, at /nix/store/5k9kk52bv6zsvsyyvpxhm8xmwyn2yjvx-source/pkgs/build-support/vm/default.nix:316:24
This includes the initialy commit was done by @Mic92 plus a few fixes
from my side. So essentially this avoids patching statically linked
executables and also speeds up searching for ELF files altogether.
I've tested this by comparing the outputs of all the derivations which
make use of this hook using the following Nix expression:
let
getPackagesForRev = rev: with import (builtins.fetchGit {
url = ./.;
inherit rev;
}) { config.allowUnfree = true; }; [
cups-kyodialog3 elasticsearch franz gurobi javacard-devkit
masterpdfeditor maxx oracle-instantclient powershell reaper
teamviewer unixODBCDrivers.msodbcsql17 virtlyst wavebox zoom-us
];
pkgs = import <nixpkgs> {};
baseRev = "ef764eb0d8314b81a012dae04642b4766199956d";
in pkgs.runCommand "diff-contents" {
chset = pkgs.lib.zipListsWith (old: new: pkgs.runCommand "diff" {
inherit old new;
nativeBuildInputs = [ pkgs.nukeReferences ];
} ''
mkdir -p "''${NIX_STORE#/}"
cp --no-preserve=all -r "$old" "''${NIX_STORE#/}"
cp --no-preserve=all -r "$new" "''${NIX_STORE#/}"
find "''${old#/}" "''${new#/}" \
\( -type f -exec nuke-refs {} + \) -o \( -type l -delete \)
mkdir "$out"
echo "$old" > "$out/old-path"
echo "$new" > "$out/new-path"
diff -Nur "''${old#/}" "''${new#/}" > "$out/diff" || :
'') (getPackagesForRev baseRev) (getPackagesForRev "");
} ''
err=0
for c in $chset; do
if [ -s "$c/diff" ]; then
echo "$(< "$c/old-path") -> $(< "$c/new-path")" \
"differs, report: $c/diff" >&2
err=1
fi
done
[ $err -eq 0 ] && touch "$out"
''
With these changes there is only one derivation which has altered
contents, which is "franz". However the reason why it has differing
contents is not directly because of the autoPatchelfHook changes, but
because the "env-vars" file from the builder is in
"$out/opt/franz/env-vars" (Cc: @gnidorah) and we now have different
contents for NIX_CFLAGS_COMPILE and other environment variables.
I also tested this against a random static binary and the hook no longer
tries to patch it.
Merges: #47222
The "maxx" package recursively runs isExecutable on a bunch of files and
since the change to use "readelf" instead of "file" a lot of errors like
this one are printed during build:
readelf: Error: Not an ELF file - it has the wrong magic bytes at the
start
While the isExecutable was never meant to be used outside of the
autoPatchelfHook, it's still a good idea to silence the errors because
whenever readelf fails, it clearly indicates that the file in question
is not a valid ELF file.
Signed-off-by: aszlig <aszlig@nix.build>
If the ELF file is not an executable, we do not get a PT_INTERP section,
because after all, it's a *shared* library.
So instead of checking for PT_INTERP (to avoid statically linked
executables) for all ELF files, we add another check to see if it's an
executable and *only* skip it when it is and there's no PT_INTERP.
Signed-off-by: aszlig <aszlig@nix.build>
The `overrideScope` bound by `makeScope` (via special `callPackage`)
took an override in the form `super: self { … }`. But this is
dangerously close to the `self: super { … }` form used by *everything*
else, even other definitions of `overrideScope`! Since that
implementation did not even share any code either until I changed it
recently in 3cf43547f4, this inconsistency
is almost certainly an oversight and not intentional.
Unfortunately, just as the inconstency is hard to debug if one just
assumes the conventional order, any sudden fix would break existing
overrides in the same hard-to-debug way. So instead of changing the
definition a new `overrideScope'` with the conventional order is added,
and old `overrideScope` deprecated with a warning saying to use
`overrideScope'` instead. That will hopefully get people to stop using
`overrideScope`, freeing our hand to change or remove it in the future.
02c09e0171 (NixOS/nixpkgs#44558) was reverted in
c981787db9 but, as it turns out, it fixed an issue
I didn't know about at the time: the values of `propagateDoc` options were
(and now again are) inconsistent with the underlying things those wrappers wrap
(see NixOS/nixpkgs#46119), which was (and now is) likely to produce more instances
of NixOS/nixpkgs#43547, if not now, then eventually as stdenv changes.
This patch (which is a simplified version of the original reverted patch) is the
simplest solution to this whole thing: it forces wrappers to directly inspect the
outputs of the things they are wrapping instead of making stdenv guess the correct
values.
Because dates are an impurity, by default buildImage will use a static
date of one second past the UNIX Epoch. This can be a bit frustrating
when listing docker images in the CLI:
$ docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
hello latest 08c791c7846e 48 years ago 25.2MB
If you want to trade the purity for a better user experience, you can
set created to now.
pkgs.dockerTools.buildImage {
name = "hello";
tag = "latest";
created = "now";
contents = pkgs.hello;
config.Cmd = [ "/bin/hello" ];
}
and now the Docker CLI will display a reasonable date and sort the
images as expected:
$ docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
hello latest de2bf4786de6 About a minute ago 25.2MB
This commit adds test based on real-world crates (brotli).
There were a few more edge cases that were missing beforehand. Also it
turned out that we can get rid of the `finalBins` list since that will
now be handled during runtime.
The build expression got quiet large over time and to make it a bit
easier to grasp the different scripts involved in the build are now
separated from the nix file.
Cargo has a few odd (old) ways of picking source files if the `bin.path`
attribute isn't given in the Cargo.toml. This commit adds support for
some of those. The previous behaviour always defaulted to `src/main.rs`
which was not always the right choice.
Since there is look-ahead into the unpacked sources before running the
actual builder the path selection logic has to be embedded within the
build script.
`buildRustCrate` currently supports two ways of running building
binaries when processing a crate:
- Explicit definition of all the binaries (& optionally the paths to
their respective `main.rs`) and,
- if not binary was explictly configured all files matching the patterns
`src/main.rs`, `src/bin/*.rs`.
When the explicit list is given without path information paths are now
being picked from a list of candidates. The first match wins. The order
is the same as within the cargo compatibility code.
If the crate does not provide any libraries the path `src/{bin_name}.rs`
is also considered.
All underscores within the binary names are translated into dashes (`-`)
before the lookups are made. This seems to be a common convention.
Previously the Release.xz URL would show up with a new hash whenever
debian releases an update. By using archive.org we should have a stable
source for those. I wasn't able to find the equivalent in the debian
world. Maybe they don't keep all the different Release files around..
By setting useRealVendorConfig explicitly to true, the actual (slightly
modified) config generated by cargo-vendor is used.
This solves a problem, where the static vendor config in
pkgs/build-support/rust/default.nix would not sufficiently replace all
crates Cargo is looking for.
As useRealVendorConfig (and writeVendorConfig in fetchcargo) default to
false, there should be no breakage in existing cargoSha256 hashes.
Nethertheless, imho using this new feature should become standard. A
possible deprecation path could be:
- introduce this patch
- set useRealVendorConfig explicitly to false whereever cargoSha256 is
set but migration is not wanted yet.
- after some time, let writeVendorConfig default to true
- when useRealVendorConfig is true everywhere cargoSha256 is set and
enough time is passed, `assert cargoVendorDir == null ->
useRealVendorConfig;`, remove old behaviour
- after some time, remove all appearences of useRealVendorConfig and the
parameter itself
Introduce a `skawarePackages.buildPackage` function that contains the
common setup, removing a lot of duplication.
In particular, we require that the build directory has to be empty
after the `fixupPhase`, to make sure every relevant file is moved to
the outputs.
A next step would be to deduplicate the `configureFlags` attributes
and only require a `skawareInputs` field.
There's no reason `linkFarm` can't be used for symlinks in
subdirectories, except that currently it doesn't ensure the directory
of the link exists. This backwards-compatible change expands the utility
of the function.
This hopefully makes patchShebangs respect cross compilation. It
introduces the concept of the HOST_PATH. Nothing is ever executed on
it but instead used as a way to get the proper path using ‘command
-v’. Needs more testing.
/cc @ericson2314 @dtzwill
Fixes#33956Fixes#21138
