1
0
Fork 1
mirror of https://github.com/NixOS/nixpkgs.git synced 2024-11-25 23:20:55 +00:00
Commit graph

3154 commits

Author SHA1 Message Date
Benjamin Hipple 8dbc8e586c fetchurl: refresh and add extra KDE mirrors
* fetchurl: fix and add extra KDE mirrors (#51480)

- The gwdg.de mirror has moved the relative path of its KDE tarballs
- Add new mirrors from Berkeley and MIT, which are on the list of officially supported mirrors

https://download.kde.org/extra/download-mirrors.html

* More KDE mirror touchups

- The official one is a redirect to HTTPS anyways, so go directly to HTTPS
- Add China USTC for Asian users
- Swap Finland university from FTP to HTTP

* fetchurl: keep one ftp KDE mirror
2018-12-08 15:39:05 +01:00
Frederik Rietdijk df304abd27 Merge staging-next into staging 2018-12-08 12:50:38 +01:00
Frederik Rietdijk e0950ae9ad Merge master into staging-next 2018-12-08 12:40:13 +01:00
Dmitry Kalinkin 2da30b1384
Merge pull request #49918 from cocreature/agda-env
agda: use writeShellScriptbin instead of writeScriptBin
2018-12-07 21:48:34 -05:00
Dmitry Kalinkin 3b6de72836
agda: use exec in agdaWrapper 2018-12-07 21:37:19 -05:00
Frederik Rietdijk 747aa4cc93 Merge staging-next into staging 2018-12-07 15:31:52 +01:00
Frederik Rietdijk 5f554279ec Merge master into staging-next 2018-12-07 15:22:35 +01:00
lewo f7e67be1dc
Merge pull request #51528 from grahamc/buildImage-on-layered-image
dockertools buildImage: support new-style image specs
2018-12-07 09:44:58 +01:00
Graham Christensen c88337c9ac
dockerTools.buildImage: support using a layered image in fromImage
Docker images used to be, essentially, a linked list of layers. Each
layer would have a tarball and a json document pointing to its parent,
and the image pointed to the top layer:

    imageA  ----> layerA
                    |
                    v
                  layerB
                    |
                    v
                  layerC

The current image spec changed this format to where the Image defined
the order and set of layers:

    imageA  ---> layerA
            |--> layerB
            `--> layerC

For backwards compatibility, docker produces images which follow both
specs: layers point to parents, and images also point to the entire
list:

    imageA  ---> layerA
            |      |
            |      v
            |--> layerB
            |      |
            |      v
            `--> layerC

This is nice for tooling which supported the older version and never
updated to support the newer format.

Our `buildImage` code only supported the old version, so in order for
`buildImage` to properly generate an image based on another image
with `fromImage`, the parent image's layers must fully support the old
mechanism.

This is not a problem in general, but is a problem with
`buildLayeredImage`.

`buildLayeredImage` creates images with newer image spec, because
individual store paths don't have a guaranteed parent layer. Including
a specific parent ID in the layer's json makes the output less likely
to cache hit when published or pulled.

This means until now, `buildLayeredImage` could not be the input to
`buildImage`.

The changes in this PR change `buildImage` to only use the layer's
manifest when locating parent IDs. This does break buildImage on
extremely old Docker images, though I do wonder how many of these
exist.

This work has been sponsored by Target.
2018-12-05 14:25:54 -05:00
Austin Seipp 3aa9091162 buildRustCreate: export RUSTDOC during cargo config
See https://github.com/NixOS/nixpkgs/pull/50452#issuecomment-443455411

Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-12-04 19:51:51 -06:00
Jörg Thalheim 8c74bea7b8
Merge pull request #51353 from krebs/nix-writers
build-support writers: content can be string or file
2018-12-04 20:13:36 +00:00
lassulus 995defbb9b build-support writers: content can be string or file 2018-12-04 16:40:32 +01:00
Graham Christensen fc459de60e
Merge pull request #50452 from P-E-Meunier/carnix-fix
Rust build-support: fixing a compilation error in some crates
2018-12-03 10:53:25 -05:00
Bjørn Forsman ad28261ad2 audit-tmpdir hook: clean up whitespace 2018-12-02 11:45:34 +01:00
Bjørn Forsman ed0cee645f audit-tmpdir hook: use abspath matching on RPATH entries
This ensures that RPATH entries like "/foo/build/bar" doesn't trigger a
match when TMPDIR is "/build/bar". (I've had this problem with a
prebuilt package.)
2018-12-02 11:45:18 +01:00
Jan Tojnar c5881ec2c9
Merge remote-tracking branch 'upstream/master' into staging 2018-11-30 20:09:45 +01:00
Frederik Rietdijk 1828a5c5ba Merge master into staging-next 2018-11-30 17:46:21 +01:00
Vincent Laporte 66efb76c75 ocamlPackages.buildDunePackage: fix meta.platforms 2018-11-30 16:19:03 +00:00
John Ericson ed71691a81
Merge pull request #51217 from mayflower/source-date-epoch-fix-warning
set-source-date-epoch-to-latest: fix warning
2018-11-29 15:20:22 -05:00
Charles Duffy 8f90b33240
buildMaven: Support metadata-only dependencies
The circumstances which can make this necessary are discussed in NixOS/mvn2nix-maven-plugin#17
2018-11-29 17:29:59 +00:00
Robin Gloster 34faed5bb0
set-source-date-epoch-to-latest: fix warning
If there was no older file than $NIX_BUILD_TOP this would result in a
warning, e.g. with nix-info.

```
/nix/store/15kgcm8hnd99p7plqzx7p4lcr2jni4df-set-source-date-epoch-to-latest.sh: line 13: [: : integer expression expected
```
2018-11-29 09:49:08 +01:00
Frederik Rietdijk 9b81c7e455 Merge staging-next into staging 2018-11-29 09:18:35 +01:00
Frederik Rietdijk 9db2421d1f Merge master into staging-next 2018-11-29 08:12:56 +01:00
Matthew Bauer 7fa8c41c4a
Merge pull request #50235 from illegalprime/more-fhsuserenv-blacklists
fix FHSUserEnv blacklists
2018-11-28 09:32:01 -06:00
Pierre-Étienne Meunier 3083fa2aa1 Carnix 0.9.2 2018-11-27 16:08:11 +00:00
Thomas Tuegel fb7749620b
melpaBuild: Get package-build from melpa/package-build
This commit causes melpaBuild to use package-build from melpa/package-build
instead of melpa/melpa. Development of package-build happens in the former
repository whereas the latter is much larger, containing also the MELPA
recipes. We do not need to fetch the MELPA recipes from melpa/melpa, as we fetch
them one-by-one for Nixpkgs.
2018-11-27 09:36:28 -06:00
Thomas Tuegel 762295a39c
Merge pull request #51095 from bhipple/fix/emacs-builders
emacsPackagesNg.trivialBuild: cleanup and standardize function
2018-11-27 09:34:12 -06:00
Jörg Thalheim afbdeb7b9b
Merge pull request #50802 from aszlig/autopatchelf-improvements
autoPatchelfHook: Fixes/improvements for Android SDK emulator
2018-11-27 10:25:26 +00:00
Jörg Thalheim f12bd000b9
Merge pull request #49290 from krebs/nix-writers
get nix-writers into nixpkgs
2018-11-27 07:17:03 +00:00
lassulus 2d02cd7790 build-support writers: add tests 2018-11-27 07:08:13 +01:00
lassulus abd0efae35 build-support: add writers from krebs/writers
Reference https://github.com/krebs/nix-writers revision 40fde9e
2018-11-27 07:08:12 +01:00
Benjamin Hipple a7d1474023 emacsPackagesNg.trivialBuild: cleanup and standardize function
No real function change here, but this updates the trivial and melpa builders to
be formatted more consistently with the rest of the builders, and swaps
`eval "$preBuild"` for the more standard `runHook preBuild`.
2018-11-27 00:23:04 -05:00
Jörg Thalheim dc8aca448d
Merge pull request #51028 from clefru/tmp-cargo-config
buildRustPackage: write cargo config to temporary file instead of source dir
2018-11-26 15:00:16 +00:00
aszlig 4a6e3e4185
autoPatchelfHook: Skip on missing segment headers
If the file in question is not a shared object file but an ELF, we
really want to skip the file, because we won't have anything to patch
there.

For example if the file is created via "gcc -c -o foo.o foo.c", we don't
get a segment header and so far autoPatchelf was trying to patch such a
file.

By checking for missing segment headers, we're now no longer going to
attempt patching such a file.

Signed-off-by: aszlig <aszlig@nix.build>
Reported-by: Sander van der Burg <svanderburg@gmail.com>
2018-11-26 01:58:36 +01:00
aszlig 9f23a63f79
autoPatchelfHook: Fix type of norecurse variable
While declaring it as an array doesn't do any harm in our usage, it
might be a bit confusing when reading the code.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-26 01:13:59 +01:00
aszlig 2faf905f98
autoPatchelfHook: Add addAutoPatchelfSearchPath
This function is useful if autoPatchelf is invoked during some of the
phases of a build and allows to add arbitrary shared objects to the
search path.

So far the same functionality was in autoPatchelf itself, but not
available as a separate function, so when adding shared objects to the
dependency cache one would have to do so manually.

The function also has the --no-recurse flag, which prevents recursing
into subdirectories.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-25 16:22:32 +01:00
Clemens Fruhwirth 8d4fbc55d8 Write cargo config to temporary file instead of source dir.
... as this fails if the source dir contains a "config" directory.
2018-11-25 15:26:31 +01:00
Frederik Rietdijk c1792242ef Merge staging-next into staging 2018-11-24 10:44:50 +01:00
Frederik Rietdijk e41154d1ab Merge master into staging-next 2018-11-24 10:43:33 +01:00
Jörg Thalheim c424c1161b
Merge pull request #50560 from oxij/pkgs/fetchurl-mirrors
fetchurl: mirrors: http -> https, https before http, http before ftp
2018-11-23 11:56:44 +00:00
Pierre-Étienne Meunier 0e8332ca2b Fixing "include" 2018-11-22 11:40:03 +00:00
Jan Malakhovski 8dcf831978 fetchurl: mirrors: http -> https, https before http, http before ftp
Because HTTP has a higher probability of working behind proxies.
2018-11-22 09:38:22 +00:00
Frederik Rietdijk 2219e2578e Merge staging-next into staging 2018-11-22 10:10:40 +01:00
Michael Raskin ad1abb2824
Merge pull request #46115 from oxij/stdenv/bintools-cc-symlink
cc-wrapper, bintools-wrapper: simply symlink man and info outputs
2018-11-22 08:58:28 +00:00
Frederik Rietdijk c31cb577ae Merge master into staging-next 2018-11-22 09:57:08 +01:00
Matthew Bauer 4e68511bb1 bintools: use i386 on all 32 bit x86 systems 2018-11-21 09:38:28 -06:00
Jörg Thalheim 952f4fda86
makeRustPlatform: refactor to make it easier to understand
It is now clearer what is supposed to be in the rust attribute set
without having studied type theory. The amount of code is identically.
2018-11-21 12:44:58 +00:00
aszlig 3ca35ce0b2
autoPatchelfHook: Add --no-recurse flag
This is to be used with the autoPatchelf command and allows to only
patch a specific file or directory without recursing into
subdirectories.

Apart from being able to run the command in a standalone way, as
detailled in the previous commit this is also needed for the Android SDK
emulator, because according to @svanderburg there are subdirectories we
don't want to patch.

The reason why I didn't use GNU getopt is that it might not be available
on all operating systems and the getopts bash builtin doesn't support
long arguments. Apart from that, the implementation for recognizing the
flag is pretty trivial and it's also using bash builtins only, so if we
want to do something really fancy someday, we can still change it.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-20 00:11:29 +01:00
aszlig e4fbb244ee
autoPatchelfHook: Allow to prevent automatic run
If you want to only run autoPatchelf on a specific path and leave
everything else alone, we now have a $dontAutoPatchelf environment
variable, which causes the postFixup hook to not run at all.

The name "dontAutoPatchelf" probably is a bit weird in conjunction with
putting "autoPatchelfHook" in nativeBuildInputs, but unless someone
comes up with a better name I keep it that way because it's consistent
with all the other dontStrip, dontPatchShebangs, dontPatchELF and
whatnot.

A specific example where this is needed is when building the Android SDK
emulator, which contains a few ARM binaries in subdirectories that
should not be patched. If we were to run autoPatchelf on all outputs
unconditionally we'd run into errors because some ARM libraries couldn't
be found.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-20 00:07:38 +01:00
aszlig d03e4ffdbf
autoPatchelfHook: Make easier to run autoPatchelf
The autoPatchelf main function which is run against all of the outputs
was pretty much tailored towards this specific setup-hook and was
relying on $prefix to be set globally.

So if you wanted to run autoPatchelf manually - let's say during
buildPhase - you would have needed to run it like this:

  prefix=/some/directory autoPatchelf

This is now more intuitive and all you need to do is run the following:

  autoPatchelf /some/directory

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-19 17:18:27 +01:00
Frederik Rietdijk 0d0d7dcd06 Merge staging-next into staging 2018-11-18 10:41:34 +01:00
Austin Seipp 04a543b3a0 defaultCrateOverrides: foundationdb native dependencies
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-11-17 19:28:48 -06:00
zimbatm 551aecfa83
tmpdir audit: only fail with files referenced below (#35068)
On Linux the `$TMPDIR` is `/build`. The TMPDIR audit looks for `$TMPDIR`
in the build output, which will then fail with packages like
/buildkite-agent.

This fixes the heuristic to look for `$TMPDIR/` instead.
2018-11-16 22:35:56 +01:00
Pierre-Étienne Meunier f1de24feb8 Rust build-support: fixing a compilation error in some crates (such as proc-macro2) 2018-11-16 12:12:59 +00:00
Daiderd Jordan bdec3ed049
Revert "Revert "Revert "patch-shebangs: respect cross compilation"""
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.

This reverts commit bde99096a8.
2018-11-14 23:37:32 +01:00
Daiderd Jordan c9223a17bc
Revert "patch-shebangs: use --build for auto patch shebangs"
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.

This reverts commit eb7c50a993.
2018-11-14 23:37:31 +01:00
zimbatm 695a3d4254
Merge pull request #50302 from zimbatm/libredirect-misc
libredirect: misc changes
2018-11-14 00:16:56 +01:00
zimbatm d04a1265a1
libredirect: set install_name on Darwin
fixes https://github.com/NixOS/nixpkgs/pull/50246#issuecomment-437975038
2018-11-14 00:05:26 +01:00
zimbatm 91c130e2f5
libredirect: introduce optional setup-hook
This allows to simplify the usage of libredirect inside of nix build
sandboxes. Add "libredirect.hook" to the build inputs to get everything
linked in automaticall. All that's left is to set NIX_REDIRECTS and call
the target program.
2018-11-14 00:05:23 +01:00
Matthew Bauer 4b8c1d23d0
Merge pull request #42794 from telent/make-closure-x
make-closure: needs build system mkdir and jq
2018-11-13 15:02:08 -06:00
Matthew Bauer f9a6963d9a
Merge pull request #50244 from tathougies/travis/wrap-correctly
make-wrapper should use runtimeShell, not $SHELL, for cross-compilation
2018-11-13 13:55:26 -06:00
zimbatm e62db105c4
libredirect: specify libName
reduces a bit of duplication and can also be used from the outside:

   export LD_PRELOAD=${libredirect}/lib/${libredirect.libName}
2018-11-13 12:26:15 +01:00
Frederik Rietdijk 3b052406ea Merge staging-next into staging 2018-11-12 19:01:36 +01:00
Antoine Eiche c12f75649e dockerTools.buildImageWithNixDb: simplifications and switch to closureInfo
Since Nix 2 is now the stable Nix version, we can use closureInfo
which simplifies the Nix database initialisation (size and hash are
included in the "dump").
2018-11-12 18:30:53 +01:00
aszlig a815f53c60
libredirect: Add preload wrapper for stat()
Pull request #50246 was merged a bit too quickly and it was supposed to
fix libredirect on Darwin. However it still failed on Darwin and this
was missed by the person merging the pull request.

The reason this was failing was that there is no __xstat* on Darwin.

So I'm adding a wrapper for stat() as well as it works on Darwin and it
still doesn't hurt on GNU/Linux.

Signed-off-by: aszlig <aszlig@nix.build>
Cc: @edolstra, @zimbatm
2018-11-12 13:31:43 +01:00
aszlig 34dd1c68f8
libredirect: Add a small test
This is just a sanity check on whether the library correctly wraps the
syscalls and it's using the "true" executable for posix_spawn() and
execv().

The installCheckPhase is not executed if we are cross-compiling, so this
shouldn't break cross-compilation.

One thing I'm not actually sure is whether ${coreutils}/bin/true is
universally available on all the platforms, nor whether all the
functions we use in the test are available, but we can still fix that
after we've found out about that.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-12 11:02:54 +01:00
aszlig ba1fddb315
libredirect: Use extensions.sharedLibrary
This is to make sure we get the correct shared library suffix of the
target platform. While for example on Darwin it would even work with the
hardcoded .so prefix it's IMHO a bit nicer to have the actual native
extension.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-12 10:08:02 +01:00
zimbatm 9ef52352bd
assume that it works on all unix platforms 2018-11-12 00:09:36 +01:00
zimbatm d76ec523bb
use for cross-compilation 2018-11-12 00:08:18 +01:00
aszlig 753743c37b
libredirect: Add support for Darwin
The library can be used also on Darwin using it like this:

  NIX_REDIRECTS='foo=bar' \
  DYLD_INSERT_LIBRARIES=${libredirect}/lib/libredirect.so \
  DYLD_FORCE_FLAT_NAMESPACE=1 \
  some_program

So let's actually not hardcade gcc and add Darwin to meta.platforms.

No other changes seem to be required.

Signed-off-by: aszlig <aszlig@nix.build>
2018-11-11 19:29:12 +01:00
Travis Athougies 9531a32b60 make-wrapper should use runtimeShell, not bash, for cross-compilation 2018-11-11 10:25:05 -08:00
Michael Eden a3488fb9ac fix FHSUserEnv blacklists 2018-11-11 10:32:09 -05:00
Frederik Rietdijk 1d3bff25db Merge staging-next into staging 2018-11-11 14:28:08 +01:00
Moritz Kiefer 0266996a8d agda: use writeShellScriptbin instead of writeScriptBin
This adds the shell shebang to the wrapper script. Without this,
emacs and in particular agda2-mode (but probably other applications as
well) return a format error when trying to execute agda.
2018-11-08 17:53:29 +01:00
Matthew Bauer c8aff96110
Merge pull request #49608 from matthewbauer/cross-patch-shebangs-2
Restore cross-patch-shebangs branch
2018-11-07 13:37:02 -06:00
Théo Zimmermann 742bce7793
buildDunePackage: inline dune.installPhase for easier overriding
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2018-11-07 10:08:08 +01:00
Théo Zimmermann 406405d8bd
buildDunePackage: add support for pre and post phase hooks
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2018-11-07 10:08:08 +01:00
Théo Zimmermann 794158fcd5
buildDunePackage: new support function; use it to refactor some OCaml derivations 2018-11-07 10:08:03 +01:00
Jan Malakhovski d32f51c618 cc-wrapper, bintools-wrapper: simply symlink man and info outputs
With the previous commit `propagateDoc` is now always given the correct value
(i.e. it is never set to `true` when there are no `man` and `info` outputs).
Hence, we can simply symlink the original outputs to the wrapper outputs.

Pros:

- simpler, less indirection compared to `propagated-user-env-packages`,
- uses less inodes (1 symlink, which nix then simply automatically resolves
  and removes, vs. two directories and a file),
- makes direct references like "export MANPATH=${stdenv.cc.man}/share/man"
  simply work.

Cons:

- I'm not aware of any.

This and the previous commit together almost completely revert commits
fde7296a47,
fa41297209, and
c981787db9.
2018-11-07 08:37:51 +00:00
Patrick Hilhorst 0d7c99481b
fetchegg: add version to derivation 2018-11-06 00:17:03 +01:00
Yegor Timoshenko 77dad17ab6
Merge pull request #49725 from pbogdan/chrootenv-strip
chrootenv: strip the binary
2018-11-05 18:57:28 +00:00
Yegor Timoshenko cea0e9226f
chrootenv: use meson 2018-11-04 11:33:34 +00:00
Piotr Bogdan ccb76eeb3c chrootenv: strip the binary 2018-11-04 03:43:22 +00:00
aszlig c64624b843
autoPatchelfHook: Correctly detect PIE binaries
I originally thought it would just be enough to just check for an INTERP
section in isExecutable, however this would mean that we don't detect
statically linked ELF files, which would break our recent improvement to
gracefully handle those.

In theory, we are only interested in ELF files that have an INTERP
section, so checking for INTERP would be enough. Unfortunately the
isExecutable function is already used outside of autoPatchelfHook, so we
can't easily get rid of it now, so let's actually strive for more
correctness and make isExecutable actually match ELF files that are
executable.

So what we're doing instead now is to check whether either the ELF type
is EXEC *or* we have an INTERP section and if one of them is true we
should have an ELF executable, even if it's statically linked.

Along the way I also set LANG=C for the invocations of readelf, just to
be sure we don't get locale-dependent output.

Tested this with the following command (which contains almost[1] all the
packages using autoPatchelfHook), checking whether we run into any
library-related errors:

  nix-build -E 'with import ./. { config.allowUnfree = true; };
    runCommand "test-executables" {
      drvs = [
        anydesk cups-kyodialog3 elasticsearch franz gurobi
        masterpdfeditor oracle-instantclient powershell reaper
        sourcetrail teamviewer unixODBCDrivers.msodbcsql17 virtlyst
        vk-messenger wavebox zoom-us
      ];
    } ("for i in $drvs; do for b in $i/bin/*; do " +
       "[ -x \"$b\" ] && timeout 10 \"$b\" || :; done; done")
  '

Apart from testing against library-related errors I also compared the
resulting store paths against the ones prior to this commit. Only
anydesk and virtlyst had the same as they didn't have self-references,
everything else differed only because of self-references, except
elasticsearch, which had the following PIE binaries:

  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/autoconfig
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/autodetect
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/categorize
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller
  * modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/normalize

These binaries were now patched, which is what this commit is all about.

[1]: I didn't include the "maxx" package (MaXX Interactive Desktop)
     because the upstream URLs are no longer existing and I couldn't
     find them elsewhere on the web.

Signed-off-by: aszlig <aszlig@nix.build>
Fixes: https://github.com/NixOS/nixpkgs/issues/48330
Cc: @gnidorah (for MaXX Interactive Desktop)
2018-11-03 08:07:42 +01:00
Matthew Bauer eb7c50a993 patch-shebangs: use --build for auto patch shebangs
In strictDeps=false, autoPatchshebangs should use
--build (corresponding to PATH) to lookup commands. This restores the
previous behavior of patchshebangs so that we don’t break stuff that
isn’t careful in the buildInputs vs. nativeBuildInputs distinction.
Unfortunately this won’t work under cross compilation.
2018-11-02 00:27:14 -05:00
Matthew Bauer bde99096a8 Revert "Revert "patch-shebangs: respect cross compilation""
This reverts commit 9c4b11e9a0.
2018-11-02 00:27:14 -05:00
Kristoffer Søholm 5e5e57c572 buildFHSUserEnv: use runScript in env (#49077)
This makes its behaviour conform to what is implied in the
documentation.
2018-10-30 22:47:08 +01:00
Michał Janiszewski 3f05186984 Compare to None using identity is operator
This is a trivial change that replaces `==` operator with `is` operator, following PEP 8 guideline:

> Comparisons to singletons like None should always be done with is or is not, never the equality operators.

https://legacy.python.org/dev/peps/pep-0008/#programming-recommendations
2018-10-30 21:30:56 +01:00
Matthew Bauer 412093994b gcc: support avr
- respect libc’s incdir and libdir
- make non-unix systems single threaded
- set LIMITS_H_TEST to false for avr
- misc updates to support new libc’s
- use multilib with avr

For threads we want to use:
- posix on unix systems
- win32 on windows
- single on everything else

For avr:
- add library directories for avrlibc
- to disable relro and bind
- avr5 should have precedence over avr3 - otherwise gcc uses the wrong one
2018-10-29 14:34:09 -05:00
Matthew Bauer d59a9ac7cf avr: use new compilation infrastructure
Gets rid of:
  avrbinutils
  avrgcc

to replace with:
  pkgsCross.avr.buildPackages.binutils
  pkgsCross.avr.buildPackages.gcc
2018-10-29 14:34:09 -05:00
Jörg Thalheim 96c627b3f6
defaultCrateOverrides: add serde_derive 2018-10-28 21:59:19 +00:00
Jörg Thalheim e0a5689528
defaultCrateOverrides: order alphabetically 2018-10-28 21:55:26 +00:00
Pierre-Etienne Meunier ae3b4655a4 Carnix: 0.7.2 -> 0.8.10 (#40587)
Carnix: splits input into two parts: creates from creates.io and local ones
2018-10-28 00:06:29 +01:00
Jörg Thalheim f10b935f84
breakpointHook: add for debugging failing builds
Usuage: Add breakpointHook to your `buildInputs` like this:

  stdenv.mkDerivation rec {
    # ...
    buildInputs = [ breakpointHook ];
  });

When the build fails as show in this example:

  pkgs.hello.overrideAttrs (old: {
    buildInputs = [ breakpointHook ];
    postPatch = ''
      false
    '';
  });

It will halt execution printing the following message:

build failed in patchPhase with exit code 1
To attach to this build run the following command as root:

   cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10

Installing cntr and running the command will provide shell access to the
build sandbox of failed build:

sudo cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10
WARNING: bad ownership on /nix/var/nix/profiles/per-user/root, should be 1000
[nixbld@localhost:/var/lib/cntr]$

At /var/lib/cntr the sandbox filesystem is mounted. All commands and
files of the system are still accessible within the shell.
To execute commands from the sandbox use the `cntr exec` subcommand.
2018-10-25 10:19:41 +01:00
Frederik Rietdijk 821a3beb10
Merge pull request #48306 from NixOS/staging-next
Merge staging-next into master
2018-10-18 11:23:04 +02:00
Renaud 3583fe7586
Merge pull request #26839 from volth/fetchmavenartifact-do-not-leak-hash
fetchMavenArtifact: prevent leaking nix hash to jar name
2018-10-17 09:34:34 +02:00
Timo Kaufmann 1aff3da14e
Merge pull request #48020 from erictapen/47709-fix-regex
buildRustPackage: fix regex for separating lib and bin
2018-10-10 19:29:09 +02:00
Frederik Rietdijk bc9bd012c4 Merge staging-next into staging 2018-10-09 15:37:52 +02:00
Frederik Rietdijk eeaf3a131f Merge master into staging-next 2018-10-09 15:37:22 +02:00
Linus Heckemann 9cc18fa7f9 debian vm tools: use snapshot.debian.org
snapshot.debian.org actually keeps track of all of the updates as they
come in rather than doing arbitrary (?) snapshots.
2018-10-08 18:05:09 +02:00
Justin Humm 64d0676fe1
buildRustPackage: fix regex for separating lib and bin
E.g. exa was wrongly put into /lib, as it matches

  .*.a

but not

  .*\.a
2018-10-07 22:14:19 +02:00
Sarah Brofeldt 2e38f5fc6e
Merge pull request #47448 from kalbasit/nixpkgs_add-bazel-watcher
bazel-watcher: init at 0.5.0
2018-10-04 00:00:47 +02:00
Edward Tate 6ad43a0bce
buildRustPackage now correctly installs binaries to bin and libraries to lib. 2018-10-03 16:27:10 +02:00
Samuel Leathers 024eb9a5a5 trivial builders: adding usage documentation for functions 2018-10-02 22:09:09 +02:00
Frederik Rietdijk 6ce04af137 Merge master into staging 2018-10-02 18:22:37 +02:00
Daiderd Jordan 1383c08f2c
Merge branch 'master' into staging-next 2018-10-01 19:42:07 +02:00
Sarah Brofeldt b256df4937 dockerTools: Use nix instead of nixUnstable 2018-10-01 09:51:52 +02:00
lewo 56b4db9710
Merge pull request #47411 from graham-at-target/multi-layered-images-crafted
Multi-Layered Docker Images
2018-10-01 09:48:24 +02:00
Wael M. Nasreddine 86a5535b2f
bazel-watcher: init at 0.5.0 2018-09-29 13:33:00 -07:00
Wael M. Nasreddine 90b7b4a509
build-bazel-package: remove any .git, .svn and .hg from external 2018-09-29 13:28:15 -07:00
Wael M. Nasreddine 18aa9b0b65
build-bazel-package: prefix bazel with the USER variable
Bazel computes the default value of output_user_root before parsing the
flag[0]. The computation of the default value involves getting the $USER
from the environment. I don't have that variable when building with
sandbox enabled.

[0]: 9323c57607/src/main/cpp/startup_options.cc (L123-L124)
2018-09-29 13:28:12 -07:00
Will Dietz f7db287960 patch-shebangs.sh: use more robust 'for each file' loop, check for dir
The latter is to avoid warnings printed by find if it doesn't exist.
2018-09-28 11:21:51 -05:00
Will Dietz 286381f072 patch-shebangs: simplify a bit per reviewer suggestion 2018-09-28 11:17:33 -05:00
Will Dietz 830f9fabd4 patch-shebangs: use isScript to safely check for shebang start
Fixes commonly encountered errors about broken pipes or null-bytes in
command-substitution.
2018-09-28 11:15:36 -05:00
Graham Christensen fb2d153dac
dockerTools: test buildLayeredImage 2018-09-27 14:19:43 -04:00
Graham Christensen 4fe9006190 dockerTools.buildLayeredImage: init
Create a many-layered Docker Image.

Implements much less than buildImage:

 - Doesn't support specific uids/gids
 - Doesn't support runninng commands after building
 - Doesn't require qemu
 - Doesn't create mutable copies of the files in the path
 - Doesn't support parent images

If you want those feature, I recommend using buildLayeredImage as an
input to buildImage.

Notably, it does support:

 - Caching low level, common paths based on a graph traversial
   algorithm, see referencesByPopularity in
   0a80233487993256e811f566b1c80a40394c03d6
 - Configurable number of layers. If you're not using AUFS or not
   extending the image, you can specify a larger number of layers at
   build time:

       pkgs.dockerTools.buildLayeredImage {
         name = "hello";
         maxLayers = 128;
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

 - Parallelized creation of the layers, improving build speed.
 - The contents of the image includes the closure of the configuration,
   so you don't have to specify paths in contents and config.

   With buildImage, paths referred to by the config were not included
   automatically in the image. Thus, if you wanted to call Git, you
   had to specify it twice:

       pkgs.dockerTools.buildImage {
         name = "hello";
         contents = [ pkgs.gitFull ];
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

   buildLayeredImage on the other hand includes the runtime closure of
   the config when calculating the contents of the image:

       pkgs.dockerTools.buildImage {
         name = "hello";
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

Minor Problems

 - If any of the store paths change, every layer will be rebuilt in
   the nix-build. However, beacuse the layers are bit-for-bit
   reproducable, when these images are loaded in to Docker they will
   match existing layers and not be imported or uploaded twice.

Common Questions

 - Aren't Docker layers ordered?

   No. People who have used a Dockerfile before assume Docker's
   Layers are inherently ordered. However, this is not true -- Docker
   layers are content-addressable and are not explicitly layered until
   they are composed in to an Image.

 - What happens if I have more than maxLayers of store paths?

   The first (maxLayers-2) most "popular" paths will have their own
   individual layers, then layer #(maxLayers-1) will contain all the
   remaining "unpopular" paths, and finally layer #(maxLayers) will
   contain the Image configuration.
2018-09-26 17:54:14 -04:00
Graham Christensen fd045173ce referencesByPopularity: init to sort packages by a cachability heuristic
Using a simple algorithm, convert the references to a path in to a
sorted list of dependent paths based on how often they're referenced
and how deep in the tree they live. Equally-"popular" paths are then
sorted by name.

The existing writeReferencesToFile prints the paths in a simple
ascii-based sorting of the paths.

Sorting the paths by graph improves the chances that the difference
between two builds appear near the end of the list, instead of near
the beginning. This makes a difference for Nix builds which export a
closure for another program to consume, if that program implements its
own level of binary diffing.

For an example, Docker Images. If each store path is a separate layer
then Docker Images can be very efficiently transfered between systems,
and we get very good cache reuse between images built with the same
version of Nixpkgs. However, since Docker only reliably supports a
small number of layers (42) it is important to pick the individual
layers carefully. By storing very popular store paths in the first 40
layers, we improve the chances that the next Docker image will share
many of those layers.*

Given the dependency tree:

    A - B - C - D -\
     \   \   \      \
      \   \   \      \
       \   \ - E ---- F
        \- G

Nodes which have multiple references are duplicated:

    A - B - C - D - F
     \   \   \
      \   \   \- E - F
       \   \
        \   \- E - F
         \
          \- G

Each leaf node is now replaced by a counter defaulted to 1:

    A - B - C - D - (F:1)
     \   \   \
      \   \   \- E - (F:1)
       \   \
        \   \- E - (F:1)
         \
          \- (G:1)

Then each leaf counter is merged with its parent node, replacing the
parent node with a counter of 1, and each existing counter being
incremented by 1. That is to say `- D - (F:1)` becomes `- (D:1, F:2)`:

    A - B - C - (D:1, F:2)
     \   \   \
      \   \   \- (E:1, F:2)
       \   \
        \   \- (E:1, F:2)
         \
          \- (G:1)

Then each leaf counter is merged with its parent node again, merging
any counters, then incrementing each:

    A - B - (C:1, D:2, E:2, F:5)
     \   \
      \   \- (E:1, F:2)
       \
        \- (G:1)

And again:

    A - (B:1, C:2, D:3, E:4, F:8)
     \
      \- (G:1)

And again:

    (A:1, B:2, C:3, D:4, E:5, F:9, G:2)

and then paths have the following "popularity":

    A     1
    B     2
    C     3
    D     4
    E     5
    F     9
    G     2

and the popularity contest would result in the paths being printed as:

    F
    E
    D
    C
    B
    G
    A

* Note: People who have used a Dockerfile before assume Docker's
Layers are inherently ordered. However, this is not true -- Docker
layers are content-addressable and are not explicitly layered until
they are composed in to an Image.
2018-09-26 15:50:10 -04:00
Graham Christensen 2bf0ee3b2b dockertools: tarsum: turn in to a buildInput 2018-09-26 15:50:04 -04:00
Daiderd Jordan 9c4b11e9a0
Revert "patch-shebangs: respect cross compilation"
This causes problems for packages built using a bootstrap stdenv,
resulting in references to /bin/sh or even bootstrap-tools. The darwin
stdenv is much stricter about what requisites/references are allowed but
using /bin/sh on linux is also undesirable.

eg. https://hydra.nixos.org/build/81754896

    $ nix-build -A xz
    $ head -n1 result-bin/bin/xzdiff
    #!/nix/store/yvc7kmw98kq547bnqn1afgyxm8mxdwhp-bootstrap-tools/bin/sh

This reverts commit f06942327a.
2018-09-25 21:11:54 +02:00
John Ericson c701d6cb21 Merge remote-tracking branch 'upstream/master' into staging 2018-09-25 14:20:27 -04:00
Sarah Brofeldt ded8f28c3a Revert "virtualization/qemu-vm: fix and improve virtio/scsi switching"
This reverts commit f777d2b719.
cc #34409
This breaks evaluation of the tested job:
attribute 'diskInterface' missing, at /nix/store/5k9kk52bv6zsvsyyvpxhm8xmwyn2yjvx-source/pkgs/build-support/vm/default.nix:316:24
2018-09-25 11:10:10 +02:00
John Ericson 2b4b7d4ef3
Merge pull request #47233 from oxij/tree/mass-rebuild-noop-cleanups
treewide: mass rebuild noop cleanups
2018-09-25 00:04:52 -04:00
aszlig 19e83bc2ba
Merge autoPatchelfHook improvements (#47222)
This includes the initialy commit was done by @Mic92 plus a few fixes
from my side. So essentially this avoids patching statically linked
executables and also speeds up searching for ELF files altogether.

I've tested this by comparing the outputs of all the derivations which
make use of this hook using the following Nix expression:

  let
    getPackagesForRev = rev: with import (builtins.fetchGit {
      url = ./.;
      inherit rev;
    }) { config.allowUnfree = true; }; [
      cups-kyodialog3 elasticsearch franz gurobi javacard-devkit
      masterpdfeditor maxx oracle-instantclient powershell reaper
      teamviewer unixODBCDrivers.msodbcsql17 virtlyst wavebox zoom-us
    ];

    pkgs = import <nixpkgs> {};
    baseRev = "ef764eb0d8314b81a012dae04642b4766199956d";

  in pkgs.runCommand "diff-contents" {
    chset = pkgs.lib.zipListsWith (old: new: pkgs.runCommand "diff" {
      inherit old new;
      nativeBuildInputs = [ pkgs.nukeReferences ];
    } ''
      mkdir -p "''${NIX_STORE#/}"
      cp --no-preserve=all -r "$old" "''${NIX_STORE#/}"
      cp --no-preserve=all -r "$new" "''${NIX_STORE#/}"
      find "''${old#/}" "''${new#/}" \
        \( -type f -exec nuke-refs {} + \) -o \( -type l -delete \)
      mkdir "$out"
      echo "$old" > "$out/old-path"
      echo "$new" > "$out/new-path"
      diff -Nur "''${old#/}" "''${new#/}" > "$out/diff" || :
    '') (getPackagesForRev baseRev) (getPackagesForRev "");
  } ''
    err=0
    for c in $chset; do
      if [ -s "$c/diff" ]; then
        echo "$(< "$c/old-path") -> $(< "$c/new-path")" \
             "differs, report: $c/diff" >&2
        err=1
      fi
    done
    [ $err -eq 0 ] && touch "$out"
  ''

With these changes there is only one derivation which has altered
contents, which is "franz". However the reason why it has differing
contents is not directly because of the autoPatchelfHook changes, but
because the "env-vars" file from the builder is in
"$out/opt/franz/env-vars" (Cc: @gnidorah) and we now have different
contents for NIX_CFLAGS_COMPILE and other environment variables.

I also tested this against a random static binary and the hook no longer
tries to patch it.

Merges: #47222
2018-09-25 05:21:01 +02:00
aszlig b4526040a2
autoPatchelfHook: Silence errors in isExecutable
The "maxx" package recursively runs isExecutable on a bunch of files and
since the change to use "readelf" instead of "file" a lot of errors like
this one are printed during build:

  readelf: Error: Not an ELF file - it has the wrong magic bytes at the
  start

While the isExecutable was never meant to be used outside of the
autoPatchelfHook, it's still a good idea to silence the errors because
whenever readelf fails, it clearly indicates that the file in question
is not a valid ELF file.

Signed-off-by: aszlig <aszlig@nix.build>
2018-09-25 04:48:12 +02:00
aszlig 9920215d00
autoPatchelfHook: Only check PT_INTERP on execs
If the ELF file is not an executable, we do not get a PT_INTERP section,
because after all, it's a *shared* library.

So instead of checking for PT_INTERP (to avoid statically linked
executables) for all ELF files, we add another check to see if it's an
executable and *only* skip it when it is and there's no PT_INTERP.

Signed-off-by: aszlig <aszlig@nix.build>
2018-09-25 04:42:34 +02:00
John Ericson 22ce614112
Merge pull request #47238 from obsidiansystems/overrideScope-order
lib: Deprecate `overrideScope` in lieu of `overrideScope'` taking arguments in the conventional order
2018-09-24 18:04:18 -04:00
John Ericson b9dce11712 lib: Make overrideScope' which takes arguments in the conventional order
The `overrideScope` bound by `makeScope` (via special `callPackage`)
took an override in the form `super: self { … }`. But this is
dangerously close to the `self: super { … }` form used by *everything*
else, even other definitions of `overrideScope`! Since that
implementation did not even share any code either until I changed it
recently in 3cf43547f4, this inconsistency
is almost certainly an oversight and not intentional.

Unfortunately, just as the inconstency is hard to debug if one just
assumes the conventional order, any sudden fix would break existing
overrides in the same hard-to-debug way. So instead of changing the
definition a new `overrideScope'` with the conventional order is added,
and old `overrideScope` deprecated with a warning saying to use
`overrideScope'` instead. That will hopefully get people to stop using
`overrideScope`, freeing our hand to change or remove it in the future.
2018-09-24 17:50:11 -04:00
Vladimír Čunát 3a09a4b579
Merge branch 'master' into staging
Hydra: ?compare=1480463
2018-09-24 21:18:23 +02:00
Jörg Thalheim 58a97dfb49 autoPatchelfHook: do not patch statically linked files
Also speed up quite significantly due less forking.
2018-09-23 21:33:43 +01:00
Jan Malakhovski b2c7a5a271 bintools-wrapper, cc-wrapper, stdenv: infer propagateDoc automatically
02c09e0171 (NixOS/nixpkgs#44558) was reverted in
c981787db9 but, as it turns out, it fixed an issue
I didn't know about at the time: the values of `propagateDoc` options were
(and now again are) inconsistent with the underlying things those wrappers wrap
(see NixOS/nixpkgs#46119), which was (and now is) likely to produce more instances
of NixOS/nixpkgs#43547, if not now, then eventually as stdenv changes.

This patch (which is a simplified version of the original reverted patch) is the
simplest solution to this whole thing: it forces wrappers to directly inspect the
outputs of the things they are wrapping instead of making stdenv guess the correct
values.
2018-09-23 17:29:56 +00:00
Frederik Rietdijk 14673d61ed Merge master into staging 2018-09-23 09:33:09 +02:00
Stefan Junker f777d2b719 virtualization/qemu-vm: fix and improve virtio/scsi switching 2018-09-22 23:29:19 +02:00
John Ericson aba5f4974a
Merge pull request #47145 from obsidiansystems/fetchzip-cross
fetchzip: Use unzip from buildPackages
2018-09-21 14:52:17 -04:00
John Ericson b97242238d fetchzip: Use unzip from buildPackages
Additionally, the manual path manipulation becomes no longer needed.
2018-09-21 12:55:32 -04:00
Graham Christensen aedc651903
dockerTools.buildImage: test that created=now makes an unstable date 2018-09-20 13:06:14 -04:00
Graham Christensen a32d7e0c74 dockerTools.buildImage: support impure dates
Because dates are an impurity, by default buildImage will use a static
date of one second past the UNIX Epoch. This can be a bit frustrating
when listing docker images in the CLI:

    $ docker image list
    REPOSITORY   TAG      IMAGE ID       CREATED        SIZE
    hello        latest   08c791c7846e   48 years ago   25.2MB

If you want to trade the purity for a better user experience, you can
set created to now.

    pkgs.dockerTools.buildImage {
      name = "hello";
      tag = "latest";
      created = "now";
      contents = pkgs.hello;

      config.Cmd = [ "/bin/hello" ];
    }

and now the Docker CLI will display a reasonable date and sort the
images as expected:

    $ docker image list
    REPOSITORY   TAG      IMAGE ID       CREATED              SIZE
    hello        latest   de2bf4786de6   About a minute ago   25.2MB
2018-09-20 18:26:02 +02:00
Jack Kelly af5eab6ea6 dockerTools.pullImage: correct default arch 2018-09-19 16:13:15 +10:00
John Ericson 7319013ea1 Merge remote-tracking branch 'upstream/master' into staging 2018-09-18 16:55:42 -04:00
Silvan Mosberger 50578abfc5
fetchcargo: Fix cargo-vendor-normalise for darwin 2018-09-17 20:23:50 +02:00
Andreas Rammhold fc5e595003
buildRustCrate: added some edge cases with binaries
This commit adds test based on real-world crates (brotli).
There were a few more edge cases that were missing beforehand. Also it
turned out that we can get rid of the `finalBins` list since that will
now be handled during runtime.
2018-09-13 22:00:29 +02:00
Andreas Rammhold 0f95d05548 buildRustCrate: add test cases 2018-09-13 20:28:39 +02:00
Andreas Rammhold fdc2017f1c buildRustCrate: binary heuristic should be able to treat spaces 2018-09-13 20:28:39 +02:00
Andreas Rammhold 1371815060 buildRustCrate: extracted builder scripts into dedicated files
The build expression got quiet large over time and to make it a bit
easier to grasp the different scripts involved in the build are now
separated from the nix file.
2018-09-13 20:28:39 +02:00
Andreas Rammhold 0c50140da5 buildRustCrate: add heuristic to picking the right source files
Cargo has a few odd (old) ways of picking source files if the `bin.path`
attribute isn't given in the Cargo.toml. This commit adds support for
some of those. The previous behaviour always defaulted to `src/main.rs`
which was not always the right choice.

Since there is  look-ahead into the unpacked sources before running the
actual builder the path selection logic has to be embedded within the
build script.

`buildRustCrate` currently supports two ways of running building
binaries when processing a crate:

- Explicit definition of all the binaries (& optionally the paths to
their respective `main.rs`) and,
- if not binary was explictly configured all files matching the patterns
  `src/main.rs`, `src/bin/*.rs`.

When the explicit list is given without path information paths are now
being picked from a list of candidates. The first match wins. The order
is the same as within the cargo compatibility code.

If the crate does not provide any libraries the path `src/{bin_name}.rs`
is also considered.

All underscores within the binary names are translated into dashes (`-`)
before the lookups are made. This seems to be a common convention.
2018-09-13 20:28:39 +02:00
Andreas Rammhold 87462d6be4
vmTools: update debian repositories to stable Release.xz urls
Previously the Release.xz URL would show up with a new hash whenever
debian releases an update. By using archive.org we should have a stable
source for those. I wasn't able to find the equivalent in the debian
world. Maybe they don't keep all the different Release files around..
2018-09-13 11:10:23 +02:00
Symphorien Gibol a3e1da17cb cargo-vendor-normalise: add a small install check 2018-09-11 23:44:14 +02:00
Jörg Thalheim 7bfa20198a fetchcargo: add type checking to cargo-vendor-normalise.py 2018-09-11 23:44:14 +02:00
Symphorien Gibol f20b229aa1 fectchcargo: don't break old sha256 2018-09-11 23:44:14 +02:00
Symphorien Gibol ccf72b8537 fetchcargo: normalise cargo config to ensure determinism 2018-09-11 23:44:14 +02:00
Justin Humm b66ef28841 buildRustPackage, fetchcargo: optionally use vendor config from cargo-vendor
By setting useRealVendorConfig explicitly to true, the actual (slightly
modified) config generated by cargo-vendor is used.

This solves a problem, where the static vendor config in
pkgs/build-support/rust/default.nix would not sufficiently replace all
crates Cargo is looking for.

As useRealVendorConfig (and writeVendorConfig in fetchcargo) default to
false, there should be no breakage in existing cargoSha256 hashes.

Nethertheless, imho using this new feature should become standard. A
possible deprecation path could be:

- introduce this patch
- set useRealVendorConfig explicitly to false whereever cargoSha256 is
  set but migration is not wanted yet.
- after some time, let writeVendorConfig default to true
- when useRealVendorConfig is true everywhere cargoSha256 is set and
  enough time is passed, `assert cargoVendorDir == null ->
  useRealVendorConfig;`, remove old behaviour
- after some time, remove all appearences of useRealVendorConfig and the
  parameter itself
2018-09-11 23:44:14 +02:00