While looking at the graph of all the outputs in my personal binary
cache it became obvious that we have a lot of self references within the
package set. That isn't an isuse by itself. However it increases the
size of the binary cache for every (reproducible) build of a package
that carries references to itself. You can no longer deduplicate the
outputs since they are all unique. One of the ways to get rid of (a few)
references is to rewrite all the symlinks that are currently used to be
relative symlinks. Two build of something that didn't really change but
carries a self-reference can the be store as the same NAR file again.
I quickly hacked together this change to see if that would yield and
success. My bash scripting skills are probably not great but so far it
seem to somewhat work.
Adding empty variables can lead to this problem:
```diff
wrapProgram \
./pye_menu_shell \
--prefix PATH : /nix/store/4c3z5r6yxsf2cxwwyazhdn92xixn4j5b-python3-3.7.5/bin:/nix/store/b3l3niilvqcxcsbxmd6sgqk1dy1rk81c-pye-menu-1.0/bin:/nix/store/y8j1cfj8d9r5rbbxc22w7hnfjw5f4fd3-cairo-1.16.0-dev/bin:/nix/store/6mg7lfbdh9pgx7pbxr3544qqbrigdl1q-freetype-2.10.1-dev/bin:/nix/store/gpszqcy0xi0lavbbjdq82zkkjp3jbp2a-bzip2-1.0.6.0.1-bin/bin:/nix/store/031c5pk5lzabgmpqpyd46hzi625as6bp-libpng-apng-1.6.37-dev/bin:/nix/store/f8kl7kmpv130aw9zm542p74a3hg0yc13-fontconfig-2.12.6-bin/bin:/nix/store/bqp30vkncmm222mjvwggz0s7p318sflj-expat-2.2.7-dev/bin:/nix/store/w57xa8g4s4aviwmqwgra7m5hwj2b005m-glib-2.60.7-dev/bin:/nix/store/v5d4966ahvfir2hwpv003022f3pb7vik-gettext-0.19.8.1/bin:/nix/store/qpvxhl1jr0fxnrx9idnpdagqs00m5m2z-glib-2.60.7/bin \
--set PYTHONNOUSERSITE true \
--set GDK_PIXBUF_MODULE_FILE /nix/store/7ddlakx6xjczqbfs80xjd14f30fzadws-gdk-pixbuf-2.38.1/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache \
--prefix XDG_DATA_DIRS : /nix/store/0snjc1qg89zqn3v35l9d55xrykh9nj5c-gtk+3-3.24.10/share/gsettings-schemas/gtk+3-3.24.10:/nix/store/b41z51vdv11n6df8ki5vj8dynxw98f9l-gsettings-desktop-schemas-3.32.0/share/gsettings-schemas/gsettings-desktop-schemas-3.32.0:/nix/store/0snjc1qg89zqn3v35l9d55xrykh9nj5c-gtk+3-3.24.10/share/gsettings-schemas/gtk+3-3.24.10 \
- --prefix GST_PLUGIN_SYSTEM_PATH_1_0 : \
+ --prefix GST_PLUGIN_SYSTEM_PATH_1_0 : "" \
--prefix GI_TYPELIB_PATH : /nix/store/0snjc1qg89zqn3v35l9d55xrykh9nj5c-gtk+3-3.24.10/lib/girepository-1.0:/nix/store/z29l5xaaxh1s0697mcldj71ab0zshry1-librsvg-2.44.15/lib/girepository-1.0:/nix/store/pija1xzm7izxfb5m2hvhvlwp1l38ffxa-gobject-introspection-1.60.2/lib/girepository-1.0 \
- --prefix GRL_PLUGIN_PATH :
+ --prefix GRL_PLUGIN_PATH : ""
```
Where the diff is to highlight the problem: we don't have a valid value
for GST_PLUGIN_SYSTEM_PATH_1_0 or GRL_PLUGIN_PATH, and instead of
passing the empy string, the empty string gets unquoted somewhere, so we
end up passing no arguments, thus the parser in wrapProgram takes
--prefix as the argument of GST_PLUGIN_SYSTEM_PATH_1_0, and then
GI_TYPELIB_PATH is missing it's --prefix so wrapProgram complains/dies.
The easiest change is to not add empty arguments to the wrapper
Go beyond the obvious setup hooks now, with a bit of sed, with a skipped case:
- cc-wrapper's `dontlink`, because it already is handled.
Also, in nix files escaping was manually added.
EMP
There is a bug in this feature: It allows extra arguments to leak in
from the environment. For example:
$ export extraFlagsArray=date
$ man ls
Note that you get the man page for date rather than for ls. This happens
because 'man' happens to use a wrapper (to add groff to its PATH).
An attempt to fix this was made in 5ae18574fc in PR #19328 for
issue #2537, but 1. That change didn't actually fix the problem because
it addressed makeWrapper's environment during the build process, not the
constructed wrapper script's environment after installation, and 2. That
change was apparently accidentally lost when merged with 7ff6eec5fd.
Rather than trying to fix the bug again, we remove the extraFlagsArray
feature, since it has never been used in the public repo in the ten
years it has been available.
wrapAclocal continues to use its own, separate flavor of extraFlagsArray
in a more limited context. The analogous bug there was fixed in
4d7d10da6b in 2011.
These should not cause issues in practice but it is good idea to handle them.
* prefix and targetOffset are mandatory, as they are always set by the generic builder.
* wrapPrefixVariables and dontWrapGApps are now defaulting to empty value, as they are not mandatory.
This is a new package that provides a shell hook to make it easy to
declare manpages and shell completions in a manner that doesn't require
remembering where to actually install them. Basic usage looks like
{ stdenv, installShellFiles, ... }:
stdenv.mkDerivation {
# ...
nativeBuildInputs = [ installShellFiles ];
postInstall = ''
installManPage doc/foobar.1
installShellCompletion --bash share/completions/foobar.bash
installShellCompletion --fish share/completions/foobar.fish
installShellCompletion --zsh share/completions/_foobar
'';
# ...
}
See source comments for more details on the functions.
This setup hook modifies a Perl script so that any "-I" flags in its shebang
line are rewritten into a "use lib ..." statement on the next line. This gets
around a limitation in Darwin, which will not properly handle a script whose
shebang line exceeds 511 characters.
Commit "patchShebangs: Allow for multiple arguments" 4a1e51f957
removed the check. We don't want to break existing usages so this
introduces it again with a successful exit code.
It's tempting to think patchShebangs supports multiple arguments.
Without this patch it just silently ignores all but the first. Now it
patches the shebangs in all of its arguments.
Fixes: #57695
Creating the timestamp in the patched script's directory has a few
drawbacks:
* if "foo.timestamp" already exists, it will be overwritten
* it requires the directory to be writable
This rare sitation was caught when building zoom-us package:
```
automatically fixing dependencies for ELF files
/nix/store/71d65fplq44y9yn2fvkpn2d3hrszracd-auto-patchelf-hook/nix-support/setup-hook: line 213: echo: write error: Broken pipe
/nix/store/71d65fplq44y9yn2fvkpn2d3hrszracd-auto-patchelf-hook/nix-support/setup-hook: line 210: echo: write error: Broken pipe
```
The worst is that derivation continued and resulted into broken package:
https://github.com/NixOS/nixpkgs/pull/55566#issuecomment-470065690
I hope, replacing `grep -q` with `grep` will remove this race condition.
When a package provides both executables and gio modules, it is quite
probable the executables will need those modules. wrapGAppsHook wraps
executables with GIO_EXTRA_MODULES picked up from dependencies
but forgets about the package being built. Let’s add to consideration.
Closes: https://github.com/NixOS/nixpkgs/issues/50254
Unless dontWrapGapps is set, the wrap-gapps-hook.sh will currently
wrap all executables (and symbolic links to executables) found under
the target directories: bin and libexec.
As a result, if a symbolic link in a target directory points to an
executable in a target directory, both will get wrapped. This
causes an extra shell/exec when following the symbolic link,
as well as increasing the size of the final executable's environment.
To avoid wrapping a link to an already wrapped executable, this
commit splits the determination of what gets wrapped into two phases:
1. All binaries under the target directories are wrapped and logged
with "Wrapping program ..."
2. All links to executables under the target directories are
identified and checked to see if they reference an executable
under one of the target directories.
If yes, the required wrapping has already been performed on
the associated binary (in phase 1), so no wrapping is done
and "Not wrapping link: ... (already wrapped)" is logged.
If no, the link points at an executable that hasn't been
wrapped, so the link is wrapped and "Wrapping link: ..." is logged.
As an example, the yelp package has a bin directory that contains
an executable "yelp" and a symbolic link "gnome-help" -> "yelp".
Prior to this commit, the bin directory would contain these files
after wrapping:
gnome-help -- wrapper to exec .gnome-help-wrapped
.gnome-help-wrapped -- a symbolic link to yelp
yelp -- wrapper to exec .yelp-wrapped
.yelp-wrapped -- the original yelp binary
After this commit, the bin directory will instead contain:
gnome-help -- a symbolic link to yelp
yelp -- wrapper to exec .yelp-wrapped
.yelp-wrapped -- the original yelp binary
NOTE: The primary motivation for this commit is to avoid obscuring
the fact that two or more paths are simple aliases and expected to
behave identically. It also reduces the likelihood of hitting
limits related to environment variable size.
LIMITATION: The method used above is intended to be conservative
and will still wrap symbolic links to other symbolic links when
the ultimate target is outside of bin or libexec.
This ensures that RPATH entries like "/foo/build/bar" doesn't trigger a
match when TMPDIR is "/build/bar". (I've had this problem with a
prebuilt package.)
If there was no older file than $NIX_BUILD_TOP this would result in a
warning, e.g. with nix-info.
```
/nix/store/15kgcm8hnd99p7plqzx7p4lcr2jni4df-set-source-date-epoch-to-latest.sh: line 13: [: : integer expression expected
```
If the file in question is not a shared object file but an ELF, we
really want to skip the file, because we won't have anything to patch
there.
For example if the file is created via "gcc -c -o foo.o foo.c", we don't
get a segment header and so far autoPatchelf was trying to patch such a
file.
By checking for missing segment headers, we're now no longer going to
attempt patching such a file.
Signed-off-by: aszlig <aszlig@nix.build>
Reported-by: Sander van der Burg <svanderburg@gmail.com>
While declaring it as an array doesn't do any harm in our usage, it
might be a bit confusing when reading the code.
Signed-off-by: aszlig <aszlig@nix.build>
This function is useful if autoPatchelf is invoked during some of the
phases of a build and allows to add arbitrary shared objects to the
search path.
So far the same functionality was in autoPatchelf itself, but not
available as a separate function, so when adding shared objects to the
dependency cache one would have to do so manually.
The function also has the --no-recurse flag, which prevents recursing
into subdirectories.
Signed-off-by: aszlig <aszlig@nix.build>
This is to be used with the autoPatchelf command and allows to only
patch a specific file or directory without recursing into
subdirectories.
Apart from being able to run the command in a standalone way, as
detailled in the previous commit this is also needed for the Android SDK
emulator, because according to @svanderburg there are subdirectories we
don't want to patch.
The reason why I didn't use GNU getopt is that it might not be available
on all operating systems and the getopts bash builtin doesn't support
long arguments. Apart from that, the implementation for recognizing the
flag is pretty trivial and it's also using bash builtins only, so if we
want to do something really fancy someday, we can still change it.
Signed-off-by: aszlig <aszlig@nix.build>
If you want to only run autoPatchelf on a specific path and leave
everything else alone, we now have a $dontAutoPatchelf environment
variable, which causes the postFixup hook to not run at all.
The name "dontAutoPatchelf" probably is a bit weird in conjunction with
putting "autoPatchelfHook" in nativeBuildInputs, but unless someone
comes up with a better name I keep it that way because it's consistent
with all the other dontStrip, dontPatchShebangs, dontPatchELF and
whatnot.
A specific example where this is needed is when building the Android SDK
emulator, which contains a few ARM binaries in subdirectories that
should not be patched. If we were to run autoPatchelf on all outputs
unconditionally we'd run into errors because some ARM libraries couldn't
be found.
Signed-off-by: aszlig <aszlig@nix.build>
The autoPatchelf main function which is run against all of the outputs
was pretty much tailored towards this specific setup-hook and was
relying on $prefix to be set globally.
So if you wanted to run autoPatchelf manually - let's say during
buildPhase - you would have needed to run it like this:
prefix=/some/directory autoPatchelf
This is now more intuitive and all you need to do is run the following:
autoPatchelf /some/directory
Signed-off-by: aszlig <aszlig@nix.build>
On Linux the `$TMPDIR` is `/build`. The TMPDIR audit looks for `$TMPDIR`
in the build output, which will then fail with packages like
/buildkite-agent.
This fixes the heuristic to look for `$TMPDIR/` instead.
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.
This reverts commit bde99096a8.
