The hack of using `crossConfig` to enforce stricter handling of
dependencies is replaced with a dedicated `strictDeps` for that purpose.
(Experience has shown that my punning was a terrible idea that made more
difficult and embarrising to teach teach.)
Now that is is clear, a few packages now use `strictDeps`, to fix
various bugs:
- bintools-wrapper and cc-wrapper
Regression introduced in 736848723e.
This commit most certainly hasn't been tested with sandboxing enabled
and breaks not only pullImage but also the docker-tools NixOS VM test
because it doesn't find it's certificate path and also relies on
/var/tmp being there.
Fixing the certificate path is the easiest one because it can be done
via environment variable.
I've used overrideAttrs for changing the hardcoded path to /tmp (which
is available in sandboxed builds and even hardcoded in Nix), so that
whenever someone uses Skopeo from all-packages.nix the path is still
/var/tmp.
The reason why this is hardcoded to /var/tmp can be seen in a comment in
vendor/github.com/containers/image/storage/storage_image.go:
Do not use the system default of os.TempDir(), usually /tmp, because
with systemd it could be a tmpfs.
With sandboxed builds this isn't the case, however for using Nix without
NixOS this could turn into a problem if this indeed is the case.
So in the long term this needs to have a proper solution.
In addition to that, I cleaned up the expression a bit.
Tested by building dockerTools.examples.nixFromDockerHub and the
docker-tools NixOS VM test.
Signed-off-by: aszlig <aszlig@nix.build>
Cc: @nlewo, @Mic92, @Profpatsch, @globin, @LnL7
The script would parse the output of `git submodule status` but
didn't handle paths with spaces in them. This would result in the
following error when trying to determine the URL of the submodule:
error: key does not contain a section: .url
* fetchs3: add configurable name
Change the default from "foo" to the basename of the s3 URL and make it
configurable.
* fetchs3: fix error on missing credentials.session_token
The session token should default to null instead of failing
* fetchs3: make use of the region argument
Set it to null if you don't want to use it
* fetchs3: prefer local build
Fetcher-types spend more time on network than CPU
Skopeo is used to pull images from a Docker registry (instead of a
Docker deamon in a VM).
An image reference is specified with its name and its digest which is
an immutable image identifier (unlike image name and tag).
Skopeo can be used to get the digest of an image, for instance:
$ skopeo inspect docker://docker.io/nixos/nix:1.11 | jq -r '.Digest'
... binutils and gcc add it already anyway.
Without this it's easy to get cross-toolchain paths longer than 256
chars and nix-daemon will then fail to commit them to /nix/store on XFS.
Following legacy packing conventions, `isArm` was defined just for
32-bit ARM instruction set. This is confusing to non packagers though,
because Aarch64 is an ARM instruction set.
The official ARM overview for ARMv8[1] is surprisingly not confusing,
given the overall state of affairs for ARM naming conventions, and
offers us a solution. It divides the nomenclature into three levels:
```
ISA: ARMv8 {-A, -R, -M}
/ \
Mode: Aarch32 Aarch64
| / \
Encoding: A64 A32 T32
```
At the top is the overall v8 instruction set archicture. Second are the
two modes, defined by bitwidth but differing in other semantics too, and
buttom are the encodings, (hopefully?) isomorphic if they encode the
same mode.
The 32 bit encodings are mostly backwards compatible with previous
non-Thumb and Thumb encodings, and if so we can pun the mode names to
instead mean "sets of compatable or isomorphic encodings", and then
voilà we have nice names for 32-bit and 64-bit arm instruction sets
which do not use the word ARM so as to not confused either laymen or
experienced ARM packages.
[1]: https://developer.arm.com/products/architecture/a-profile
(cherry picked from commit ba52ae5048)
Following legacy packing conventions, `isArm` was defined just for
32-bit ARM instruction set. This is confusing to non packagers though,
because Aarch64 is an ARM instruction set.
The official ARM overview for ARMv8[1] is surprisingly not confusing,
given the overall state of affairs for ARM naming conventions, and
offers us a solution. It divides the nomenclature into three levels:
```
ISA: ARMv8 {-A, -R, -M}
/ \
Mode: Aarch32 Aarch64
| / \
Encoding: A64 A32 T32
```
At the top is the overall v8 instruction set archicture. Second are the
two modes, defined by bitwidth but differing in other semantics too, and
buttom are the encodings, (hopefully?) isomorphic if they encode the
same mode.
The 32 bit encodings are mostly backwards compatible with previous
non-Thumb and Thumb encodings, and if so we can pun the mode names to
instead mean "sets of compatable or isomorphic encodings", and then
voilà we have nice names for 32-bit and 64-bit arm instruction sets
which do not use the word ARM so as to not confused either laymen or
experienced ARM packages.
[1]: https://developer.arm.com/products/architecture/a-profile
Adds a couple of useful NetBSD and OpenBSD derivations. Some of these
will be integrated into Nixpkgs later.
Noncomprehensive list:
- netbsd.getent
- netbsd.getconf
- netbsd.fts
- openbsd.mg
- netbsd.compat (can replace libbsd)
Since the script running is a failure condition, we should fail the
build properly, not leaving it up to the missing output to determine
that the build went wrong. This should partly address #38952 — nix
build will print out the build log on non-zero exits.
Also fix numberous bugs, such as:
- Not getting confused on more flags taking file arguments.
- Ensuring children reexport their children, but the original
binary/library doesn't.
- Not spawning children when it turns out we just dynamically link
under the threshold but our total number of inputs exceeeds it.
- Children were always named `libunnamed-*`, when that name was
supposed to be the last resort only.
ld-wrapper's own RPATH check hardcodes `.so`, but darwin uses `.dylib`
*and* (in practice due to lousy build systems) `.so`. We don't care
however because we never inject `--rpath` like that in practice on
Darwin. Hopefully someday we won't on linux either.
Pull request #38470 added support for running/building kernels without
modules. This got merged in 38e04bbf29 but
unfortunately while this works perfectly on kernels without modules it
also makes sure that *every* kernel gets no modules.
So all of our VM tests fail since that merge with something like this:
machine# loading module loop...
machine# modprobe: FATAL: Module loop not found in directory /lib/modules/4.14.33
machine# loading module vfat...
machine# modprobe: FATAL: Module vfat not found in directory /lib/modules/4.14.33
machine# loading module nls_cp437...
machine# modprobe: FATAL: Module nls_cp437 not found in directory /lib/modules/4.14.33
machine# loading module nls_iso8859-1...
machine# modprobe: FATAL: Module nls_iso8859-1 not found in directory /lib/modules/4.14.33
machine# loading module fuse...
machine# modprobe: FATAL: Module fuse not found in directory /lib/modules/4.14.33
machine# loading module dm_mod...
machine# modprobe: FATAL: Module dm_mod not found in directory /lib/modules/4.14.33
I shortly tested this against the "misc" VM test and the test is working
again.
In the long term (and I currently don't have time for this) it would be
better to also have a VM test which tests a kernel without modules.
Signed-off-by: aszlig <aszlig@nix.build>
Cc: @roberth, @7c6f434c
Setting the hash to null is a convenient way to bypass the hash check
while developing. It looks like the ability to do this was inadvertently
removed while adding vendor directory support.
This still checks that the user is explicitly setting the value but
allows null as a valid option.
This is to go to a reproducible image build.
Note without this options image are identical from the Docker point of
view but generated docker archives could have different hashes.
This is to improve image creation reproducibility. Since the nar
format doesn't support hard link, the tar stream of a layer can be
different if a dependency of a layer has been built locally or if it
has been fetched from a binary cache.
If the dependency has been build locally, it can contain hard links
which are encoded in the tar stream. If the dependency has been
fetched from a binary cache, the tar stream doesn't contain any hard
link. So even if the content is the same, tar streams are different.
Resolved the following conflicts (by carefully applying patches from the both
branches since the fork point):
pkgs/development/libraries/epoxy/default.nix
pkgs/development/libraries/gtk+/3.x.nix
pkgs/development/python-modules/asgiref/default.nix
pkgs/development/python-modules/daphne/default.nix
pkgs/os-specific/linux/systemd/default.nix
Per @Ericson2314's suggestion [1], make it more clear that the active
hardenings are decided via whitelist; the blacklist is merely for the
debug messages.
1: 36d5ce41d4 (r133279731)
Existing "mips64el" should be "mipsel".
This is just the barest minimum so that nixpkgs can recognize them as
systems - although required for building individual derivations onto
MIPS boards, it is not sufficient if you want to actually build nixos on
those targets
Previously, cargoUpdateHook was meaningful as it was used
in
[`cargo-fetch-deps`](19d3cf81d3/pkgs/build-support/rust/fetch-cargo-deps (L71)).
However, this entire file was removed in
5f8cf0048e. As far as I can
tell, nothing in the code is using it, but it is still
being passed around:
https://github.com/NixOS/nixpkgs/search?q=cargoUpdateHook&type=Code&utf8=%E2%9C%93
There are, however, legitimate use cases for it. For example,
in some software, some dependencies are not locked in Cargo.toml
and this causes Cargo to try fetching another version of them.
This doesn't work well with vendoring crates.
This hook allows to inject patching or whatever necessary workarounds
in the crate vendoring process. I suppose that's what it was for
in there in the first place.
This patch restores this hook and makes it usable again.
Before the code would fail silently for zero values and with some output for
empties. We now currently handle both via defaulting value to zero and making
`let` return success error code when there's no syntax error.
A separate function for building Bazel-bazed packages. Internally it splits the
build into two phases, fetching and building.
Users are expected to provide `fetchArgs.sha256` -- checksum of fetched
dependencies. Local dependencies should be removed in `fetchArgs.preInstall`.
Overall `fetchArgs` and `buildArgs` can be used to add specific steps to fetch
and build.
`find -executable` finds everything with the executable bit set,
including directories. Thats not harmful in this scenario as `cp` won't
copy those directories, but it does result in a few warning messages.
I originally wrote this for packaging proprietary games in Vuizvui[1]
but I thought it would be generally useful as we have a fair amount of
proprietary software lurking around in nixpkgs, which are a bit tedious
to maintain, especially when the library dependencies change after an
update.
So this setup hook searches for all ELF executables and libraries in the
resulting output paths after install phase and uses patchelf to set the
RPATH and interpreter according to what dependencies are available
inside the builder.
For example consider something like this:
stdenv.mkDerivation {
...
nativeBuildInputs = [ autoPatchelfHook ];
buildInputs = [ mesa zlib ];
...
}
Whenever for example an executable requires mesa or zlib, the RPATH will
automatically be set to the lib dir of the corresponding dependency.
If the library dependency is required at runtime, an attribute called
runtimeDependencies can be used to list dependencies that are added to
all executables that are discovered unconditionally.
Beside this, it also makes initial packaging of proprietary software
easier, because one no longer has to manually figure out the
dependencies in the first place.
[1]: https://github.com/openlab-aux/vuizvui
Signed-off-by: aszlig <aszlig@nix.build>
Closes: #34506
Among other things, this will allow *2nix tools to output plain data
while still being composable with the traditional
callPackage/.override interfaces.
Summary:
According to git-submodule manpage,
"git submodule status" prefixes the hash with a '-' if it is not
initialized, and other chars in other circumstances.
(this is consistent on the various git versions tested)
nix-prefetch-git runs "git submodule init" which does you'd think,
but apparently despite this earlier versions of git before 2.16
would still give the hash the '-' suffix.
In particular this is the behavior when using 2.15 and 2.14.1
from the nixos-17.09 and nixos-17.03 channels respectively.
The script then used awk to drop the first char of the first field
which does the wrong thing when there is no prefix emitted:
while there is a space character before the hash, this is not
part of the field and so we ended up eating the first character
of the hash.
To fix this in a way that also works with the previous behavior,
this commit instead uses awk to grab the hash field
and uses tr to delete any '-' chars should they be present.
This seems to work in my testing, and for example can now
successfully fetch the source for "nginxModules.brotli"
where previously it would generate an error:
fatal: '22564a95d9ab58865a096b8d9f7324ea5f2e03e' is not a commit and a branch 'fetchgit' cannot be created from it
(we dropped a '2' from the beginning of the hash)
We still ensure the old and new ones start, respectfully, with `a/` and
`b/`. Use with `stripLen` to ensure tha the old `a/` and `/b` are gone
if a new prefix is added.
cargo-vendor generates almost the right cargo config. Store it with the
vendored files and patch it on use.
This allows to re-use the generated config when using git dependencies.
- All deps go on the PATH
- CC and Bintools wrappers with their host != depender's host still get their
setup hooks run.
- Environment hooks get applied to all packages
This isn't so elegent, but eases the transition on a very significant
PR.
We now have the information to properly determine the role the
cc-wrapper dependency has, by taking advantage of `offset`. No longer
use the soon-to-be-deprecated crossConfig environment variable, the
temp hack used before this change.
Changes:
* doesn't handle root user separately
* doesn't chdir("/") which makes using it seamless
* only bind mounts, doesn't symlink (i.e. files)
Incidentally, fixes #33106.
It's about two times shorter than the previous version, and much
easier to read/follow through. It uses GLib quite heavily, along with
RAII (available in GCC/Clang).
It would be nice to be able to track Nix requests. It's not trustworthy,
but can be helpful for stats and routing in HTTP logs.
Since `fetchurl` is used so widely, we should "magically" get a UA on
`fetchzip`, `fetchFromGitHub`, and other related fetchers.
Since `fetchurl` is only used for fixed-output derivations, this should
cause no mass rebuild.
User-Agent example: curl/7.57.0 Nixpkgs/18.03
* Wrap LEN macro in parantheses
* Drop env_filter in favor of stateful environ_blacklist_filter,
use execvp instead of execvpe, don't explicitly use environ
* Add argument error logging wherever it makes sense
* Drop strjoin in favor of asprintf
* char* -> const char* where appropriate
* Handle stat errors
* Print user messages with fputs, not errorf
* Abstract away is_str_in (previously bind_blacklisted)
* Cleanup temporary directory on error
* Some minor syntactic and naming changes
Thanks to Jörg Thalheim and Tuomas Tynkkynen for the code review!
Factor a bintools (i.e. binutils / cctools) wrapper out of cc-wrapper. While
only LD is wrapped, the setup hook defines environment variables on behalf of
other utilites.
They aren't meant to be critical (uncatchable) errors.
Tested with nix-env + checkMeta:
[ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ]
This commit adds the CentOS 7.4 base image from the CentOS mirror, for use with
building RPMs or evaluating Nix expressions in a CentOS image.
When CentOS 7.5 comes out, I will swap this URL to the permanently vaulted image.
This change adds granular, non-docker daemon docker image fetchers and
a docker image layer compositor to be used in conjunction with the
`docker2nix` utility provided by the `haskellPackages.hocker` package.
This change includes a hackage package version bump and updated sha256
for recent fixes released to `hocker` resulting from formulating this
patch.
On non-GNU (gcc) compilers, there is no "/lib/gcc/..."
so when this is eventually expanded this is empty
resulting in an incomplete "-idirafter " that
eats the next argument:
-idirafter -B/nix/store/wamjwwdvkmhbf4f2902nhw8jxxzv0hy3-clang-wrapper-4.0.1/bin/
Certain tools, e.g. compilers, are customarily prefixed with the name of
their target platform so that multiple builds can be used at once
without clobbering each other on the PATH. I was using identifiers named
`prefix` for this purpose, but that conflicts with the standard use of
`prefix` to mean the directory where something is installed. To avoid
conflict and confusion, I renamed those to `targetPrefix`.
This continues #23374, which always kept around both attributes, by
always including both propagated files: `propgated-native-build-inputs`
and `propagated-build-inputs`. `nativePkgs` and `crossPkgs` are still
defined as before, however, so this change should only barely
observable.
This is an incremental step to fully keeping the dependencies separate
in all cases.
After #31497 starter quoting all values, there arouse the need to left some
values evaluated.
`--set-default var value` expands to `export var=${var-value}`, where value is
not evaluated and literally assigned to var unless it is already set.
`--set-eval var value` expands to `export var=$(eval echo value)`, where value
is evaluated by `eval`.
Currently we wrap ssh so it can find the config file passed in by
<ssh-config-file>. If one however uses ProxyCommand ssh, then ssh that
is on PATH is taken (which is also unavailable when using nix-shell
--pure), which is the plain ${openssh}/bin/ssh.
This commit makes sure our wrapped ssh is available on PATH.
For a while now, the only thing the 'uboot' attribute does is to tell
whether to add ubootTools to kernel/initrd builds. That can be
determined with platform.kernelTarget == "uImage" just as well.
If a dynamic linker for target is not found the generated script fails
due to unbound variable error (due to "set -u"). Correct by specifying
default value with dynamicLinker:- and not generating ldflagsBefore if
no linker is found.
This problem was found when cross compiling to mingw32 targets
The previous one was very bad and worsened the situation.
But even running with some nix-1.12 I'm unable to reproduce
the original failure. Let's unblock channels for now.
Unlike pathsFromGraph, on Nix 1.12, this function produces a
registration file containing correct NAR hash/size information.
https://hydra.nixos.org/build/62832723
The biggest benefit is that we no longer have to update the registry
package. This means that just about any cargo package can be built by
nix. No longer does `cargo update` need to be feared because it will
update to packages newer then what is available in nixpkgs.
Instead of fetching the cargo registry this bundles all the source code
into a "vendor/" folder.
This also uses the new --frozen and --locked flags which is nice.
Currently cargo-vendor only provides binaries for Linux and
macOS 64-bit. This can be solved by building it for the other
architectures and uploading it somewhere (like the NixOS cache).
This also has the downside that it requires a change to everyone's deps
hash. And if the old one is used because it was cached it will fail to
build as it will attempt to use the old version. For this reason the
attribute has been renamed to `cargoSha256`.
Authors:
* Kevin Cox <kevincox@kevincox.ca>
* Jörg Thalheim <Mic92@users.noreply.github.com>
* zimbatm <zimbatm@zimbatm.com>
This requires some small changes in the stdenv, then working around the
weird choice LLVM made to hardcode @rpath in its install name, and then
lets us remove a ton of annoying workaround hacks in many of our Go
packages. With any luck this will mean less hackery going forward.
Before this fix, it seemed to be trying to merge our postFetch with the
patch normalization logic, but accidentally clobbering the whole thing
with the passed-in value.
cc-wrapper may wrap a cc-compiler, but it doesn't need one to build
itself. (c.f. expand-response-params is a separate derivation.) This
helps avoid cycles on the cross stuff, in addition to removing a
useless dependency edge.
I could have been super careful with overrides in the stdenv to avoid
the mass rebuild, but I don't think it's worth it.
We were using 'Combined Image JSON + Filesystem Changeset Format' [1] to
unpack and pack image and this patch switches to the format used by the registry.
We used the 'repository' file which is not generated by Skopeo when it
pulls an image. Moreover, all information of this file are also in the
manifest.json file.
We then use the manifest.json file instead of 'repository' file. Note
also the manifest.json file is required to push an image with Skopeo.
Fix #29636
[1] 749d90e10f/image/spec/v1.1.md (combined-image-json--filesystem-changeset-format)
The database dump doesn't contain sha and size. This leads to invalid
path in the container. We have to fix the database by using
nix-store.
Note a better way to do this is available in Nix 1.12 (since the
database dump contains all required information).
We also add content output paths in the gcroots since they ca be used
by the container.
Currently, the contents closure is copied to the layer but there is no
nix database initialization. If pkgs.nix is added in the contents,
nix-store doesn't work because there is no nix database.
From the contents of the layer, this commit generates and loads the
database in the nix store of the container. This only works if there
is no parent layer that already have a nix store (to support several
nix layers, we would have to merge nix databases of parent layers).
We also add an example to play with the nix store inside the
container. Note it seems `more` is a missing dependency of the nix
package!
1. `crossDrv` is now the default so we don't need to worry about that in
build != host builds.
2. shell is the build time shell, so `wrapCCCross` doesn't need to
worry, as build == host.
3. `shell.shellPath` will always be appended where useful.
4. Complicated `shell == ""` logic served no purpose.
Fetch into $out and remove all version control files to make it
deterministic (.repo and all .git subdirectories - e.g. the .git/index
files change every time).
Additionally I've changed the default of "useArchive" to false because
fetching with "--archive" will fail for some projects (e.g.
"platform/external/iosched" from the AOSP).
Now, this function should hopefully work for every tag of the AOSP.
Before this patch, a VM was used to spawn docker that pulled the
VM. Now, the tool Skopeo does this job well so we can simplify our
dockerTools since we doesn't need Docker anymore:)
This also fixe the regression described in
https://github.com/NixOS/nixpkgs/issues/29271 : cntlm proxy doesn't
work in 17.09 while it worked in 17.03.
Note Skopeo doesn't produce the same output than docker pull so, we
have to update sha.
This was a problem when run inside a sandbox, e.g. via
"fetchRepoProject". The error message from repo seems unrelated:
fatal: Cannot get https://gerrit.googlesource.com/git-repo/clone.bundle
fatal: error no host given
But the exception is actually thrown due to missing certificates
(/etc/ssl/certs). It should be possible to provide another location via
environment variables (e.g. SSL_CERT_FILE, REQUESTS_CA_BUNDLE or
CURL_CA_BUNDLE) but apparently that doesn't actually work for some
reason (would have to study our Python packaging).
Now "fetchRepoProject" works without the "--no-clone-bundle" option.
The verification was failing with the following error:
gpg: keyblock resource '/tmp/nix-build-XYZ.drv-0/.repo/repo/./.repoconfig/gnupg/pubring.kbx': No such file or directory
Using an absolute path for $HOME fixes this.
And since 175ecbab91 the dependencies on
"git" and "gnupg" aren't required anymore as "gitRepo" already covers
them.
