Also begin to start work on cross compilation, though that will have to
be finished later.
The patches are based on the first version of
https://reviews.llvm.org/D99484. It's very annoying to do the
back-porting but the review has uncovered nothing super major so I'm
fine sticking with what I've got.
Beyond making the outputs work, I also strove to re-sync the packages,
as they have been drifting pointlessly apart for some time.
----
Other misc notes, highly incomplete
- lvm-config-native and llvm-config are put in `dev` because they are
tools just for build time.
- Clang no longer has an lld dep. That was introduced in
db29857eb3, but if clang needs help
finding lld when it is used we should just pass it flags / put in the
resource dir. Providing it at build time increases critical path
length for no good reason.
----
A note on `nativeCC`:
`stdenv` takes tools from the previous stage, so:
1. `pkgsBuildBuild`: `(?1, x, x)`
2. `pkgsBuildBuild.stdenv.cc`: `(?0, ?1, x)`
while:
1. `pkgsBuildBuild`: `(?1, x, x)`
2. `pkgsBuildBuild.targetPackages`: `(x, x, ?2)`
3. `pkgsBuildBuild.targetPackages.stdenv.cc`: `(?1, x, x)`
The `checkType` argument of buildRustPackage was not used anymore
since the refactoring of `buildRustPackage` into hooks. This was
an oversight that is fixed by this change.
The check type can also be passed directly to cargoCheckHook using the
`cargoCheckType` environment variable.
API change:
`cargoParallelTestThreads` suggests that this attribute sets the
number of threads used during tests, while it is actually a boolean
option (use 1 thread or NIX_BUILD_CORES threads). In the hook, this
is replaced by a more canonical name `dontUseCargoParallelTests`.
The directory in the tarball of vendored dependencies contains `name`,
which is by default set to `${pname}-${version}`. This adds an
additional attribute to permit setting the name to something of the
user's choosing.
Since `cargoSha256`/`cargoHash` depend on the name of the directory of
vendored dependencies, `cargoDepsName` can be used to e.g. make the
hash invariant to the package version by setting `cargoDepsName =
pname`.
- API change: remove the `target` argument of `buildRustPackage`, the
target should always be in sync with the C/C++ compiler that is used.
- Gathering of binaries has moved from `buildPhase` to `installPhase`,
this simplifies the hook and orders this functionality logically
with the installation logic.
`buildRustPackage` currently accepts `cargoSha256` as a hash for
vendored dependencies. This change adds `cargoHash` which accepts SRI
hashes, setting `outputHashAlgo` to `null`.
The hash mismatch message still uses `cargoSha256` as an example,
which it probably should until we completely switch to SRI hashes.
As @lopsided98 points out in #105305, since the hashes are now target
sensative, and until we find reason to actually care to test what they
are exactly, we are best just normalizing them away in the tests.
Bofore this change, NUM_JOBS was set to 1. Some crates for building
C/C++ code (e.g. the cc and cmake crates), rely on this variable to
set the number of jobs. As a consequence, we were compiling embedded
libraries serially. Change this to NIX_BUILD_CORES to permit parallel
builds.
Prior discussion:
https://github.com/NixOS/nixpkgs/pull/50452#issuecomment-439407547
This enables short argument attrsets similar to fetchPypi:
src = fetchCrate {
inherit pname version;
sha256 = "02h8pikmk19ziqw9jgxxf7kjhnb3792vz9is446p1xfvlh4mzmyx";
};
While the artifacts from `buildPhase` should be used for testing as
well, it should be avoided that those are modified during `checkPhase`.
This can happen if a package is built e.g. with special
`cargoBuildFlags` that don't apply to the `checkPhase`. In that case, a
binary would be installed into `$out` without those flags since
`checkPhase` overrides the binary in the `target`-directory.
This patch copies the state of `target/release` into a temporary
location at the end of the `buildPhase` and installs the results from
that temporary directory into `$out` while `checkPhase` can continue
using the configured build-dir.
cc #91689Closes#93119Closes#91191
When features were supplied in cargoBuildFlags, the binaries were built
with these features enabled. Unless checking was disabled, `cargo test`
was executed without these build flags, meaning the binaries were
rebuilt and overwritten without the specified features.
Fix this bug by running tests after the installation phase.
Cargo sets `CARGO_FEATURE_*` for all features when running a build
script:
https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts
Some crates have build scripts (e.g. openblas-src) that rely on the
feature variables being properly set.
Since we now need several representations of features, this change
also updates `createFeatures` to be a list of features, rather than
`rustc` feature arguments. `configureCrate` and `buildCrate` then
build the required representations as-needed.
Fixes#68978
There are several tarballs (such as the `rust-lang/rust`-source) with a
`Cargo.toml` at root and several sub-packages (with their own Cargo.toml)
without using workspaces[1].
In such a case it's needed to move into a subdir to only build the
specified sub-package (e.g. `rustfmt` or `rsl`), however the artifacts
are at `/target` in the root-dir of the build environment. This breaks
the build since `buildRustPackage` searches for executables in `target`
(which is at the build-env's root) at the end of the `buildPhase`.
With the optional `buildAndTestSubdir`-argument, the builder moves into
the specified subdir using `pushd`/`popd` during `buildPhase` and
`checkPhase`.
Also moved the logic to find executables and libs to the end of the `buildPhase`
from a custom `postBuild`-hook to fix packages with custom `build`/`install`-procedures
such as `uutils-coreutils`.
[1] https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html
If a user provides `nativeBuildInputs = [ llvmPackages.bintools ]` or any other
package containing a `${prefix}/bin/diff`, the builder could use it instead
of the standard unix `diff`, causing a build failure.
This updates the call to specify an abspath to `diff` and avoid reliance on `PATH`.
Resolves#87081
When running `cargo test --release`, the artifacts from `buildPhase`
will be reused here. Previously, most of the stuff had to be recompiled
without optimizations.
I know, heretic, but...
I also know that this is not perfect but it is a good start, I think. It
would be nice if this were part of the automatic "nixdoc" function
reference. I'd like guidance if this should be part of the rust section
or something else.
As it turns out Darwin does most of the things differently then "normal"
systems. They are using a different shared library extension and require
an obscure commandline parameter that has to be added to every build
system out there. That issue seems to be with clang on Darwin as on
Linux that flag isn't required to build the very same tests (when using
clang).
After adjusting these two details the tests are running fine on the
darwin box that I was able to obtain.
According to the Cargo documentation:
> The build script does not have access to the dependencies listed in
> the dependencies or dev-dependencies section (they’re not built
> yet!). Also, build dependencies are not available to the package
> itself unless also explicitly added in the [dependencies] table.
https://doc.rust-lang.org/cargo/reference/build-scripts.html
This change separates linkage of regular dependencies and build
dependencies.
* Make errors include the crate name and make them much more prominent.
* Move more code into lib.sh
* Already source generated logging code and lib.sh in configure
The inlined readme that we were iterating on has been moved to GitHub
issue #79975, and the default is now the new cargo fetcher, so this
doc comment is out of date.
Previously, we would asssert that the lockfiles are consistent during the
unpackPhase, but if the pkg has a patch for the lockfile itself then we must
wait until the patchPhase is complete to check.
This also removes an implicity dependency on the src attribute coming from
`fetchzip` / `fetchFromGitHub`, which happens to name the source directory
"source". Now we glob for it, so different fetchers will work consistently.
Changes the default fetcher in the Rust Platform to be the newer
`fetchCargoTarball`, and changes every application using the current default to
instead opt out.
This commit does not change any hashes or cause any rebuilds. Once integrated,
we will start deleting the opt-outs and recomputing hashes.
See #79975 for details.
The readme was nice to discuss in the implementation PR, but now that this is
merged it's better to have an issue that can be linked against in PRs and
doesn't require further merges to update status.
Ported with a status update in #79975
By overriding each dependency on every level of the dependency tree we
are creating a lot of unnecessary instances of the same derivation
Looking at the output size of `nix-instantiate --trace-function-calls
-vvvv …` and the execution time I got about a 10x improvement after
applying this change.
It was probably good intentions that lead to these overrides but in
practice no tooling (that I know of) really needs this. `carnix` and
`crate2nix` are fine without those overrides. Furthermore I believe that
it is the job of the tooling around `buildRustCrate` to provide a
coherent set of overrides. By not enforcing all of the overrides, debug
flags, verbosity, … to be the same throughout the closure we also allow
consumers to override specific aspects of the crates. Some (older?)
crates might need different `crateOverrides` then newer crates with the
same name. Currently such situations can not (easily) be implemented
with the override in-place.
This has several advantages:
1. It takes up less space on disk in-between builds in the nix store.
2. It uses less space in the binary cache for vendor derivation packages.
3. It uses less network traffic downloading from the binary cache.
4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only
substitute --flat hashes on single files (not recursive directory hashes).
5. It's consistent with how simple `fetchurl` src derivations work.
6. It provides a stronger abstraction between input src-package and output
package, e.g., it's harder to accidentally depend on the src derivation at
runtime by referencing something like `${src}/etc/index.html`. Likewise, in
the store it's harder to get confused with something that is just there as a
build-time dependency vs. a runtime dependency, since the build-time
src dependencies are tarred up.
Disadvantages are:
1. It takes slightly longer to untar at the start of a build.
As currently implemented, this attaches the compacted vendor.tar.gz feature as a
rider on `verifyCargoDeps`, since both of them are relatively newly implemented
behavior that change the `cargoSha256`.
If this PR is accepted, I will push forward the remaining rust packages with a
series of treewide PRs to update the `cargoSha256`s.
Previously I did use `runCommand` to do the same. Using
releaseTools.aggregate seems a lot saner and we might get nicer hydra
output of the tests that are failing.
