the fix to extendDerivation in #140051 unwittingly worsened eval performance by
quite a bit. set elements alone needed over 1GB extra after the change, which
seems disproportionate to how small it was. if we flip the logic used to
determine which outputs to install around and keep a "this one exactly" flag in
the specific outputs instead of a "all of them" in the root we can avoid most
of that cost.
if extendDerivation is called on something that already had extendDerivation
called on it (eg a mkDerivation result) the second call will set
outputUnspecified=true on every output by way of propagating attributes of the
full derivation to the individual outputs. this in turn causes buildEnv--and
thus nix-shell and environment.systemPackages--to install every output of such a
derivation even when only a specific output was requested, which renders the
point of multiple outputs moot. this happens in python modules (see #139756),
though it seems that tcl and possibly others should also be affected.
I am taking the non-invasive parts of #110914 to hopefully help out with #111988.
In particular:
- Use `lib.makeScopeWithSplicing` to make the `darwin` package set have
a proper `callPackage`.
- Adjust Darwin `stdenv`'s overlays keeping things from the previous
stage to not stick around too much.
- Expose `binutilsNoLibc` / `darwin.binutilsNoLibc` to hopefully get us
closer to a unified LLVM and GCC bootstrap.
This allows querying function arguments of things like fetchFromGitHub:
nix-repl> lib.functionArgs pkgs.fetchFromGitHub
{ fetchSubmodules = true; githubBase = true; ... }
The `overrideScope` bound by `makeScope` (via special `callPackage`)
took an override in the form `super: self { … }`. But this is
dangerously close to the `self: super { … }` form used by *everything*
else, even other definitions of `overrideScope`! Since that
implementation did not even share any code either until I changed it
recently in 3cf43547f4, this inconsistency
is almost certainly an oversight and not intentional.
Unfortunately, just as the inconstency is hard to debug if one just
assumes the conventional order, any sudden fix would break existing
overrides in the same hard-to-debug way. So instead of changing the
definition a new `overrideScope'` with the conventional order is added,
and old `overrideScope` deprecated with a warning saying to use
`overrideScope'` instead. That will hopefully get people to stop using
`overrideScope`, freeing our hand to change or remove it in the future.
Another attempt after my sloppy 48ccdf322d.
@Infinisil, thanks again, reverted in 4794aa5de2 and explained my mistakes in 48ccdf322d (commitcomment-29678643). I start with their work and provide this proof of this commit's correctness:
```nix
(lib.fixedPoints.extends (lib.flip g) f) # now
((f: rattrs: self: let super = rattrs self; in super // f self super) (lib.flip g) f) # inline extends
(self: let super = f self; in super // (lib.flip g) self super) # beta reduce
(self: let super = f self; in super // g super self) # beta reduce
(self_: let super = f self_; in super // g super self_) # alpha rename
(self_: let super = f self_; in super // g super self_) # original, same
```
Eventually we might harmonize `overrideScope`'s `g` parameter with the general pattern, but I leave that breaking change as a separate step. Best not to refactor and break at once, and at least the abstractions make the oddity clearer.
Among other things, this will allow *2nix tools to output plain data
while still being composable with the traditional
callPackage/.override interfaces.
addPassthru became unused in #33057, but its signature was changed at the same
time. This commit restores the original signature and updates the warning and
the changelog.
This does break the API of being able to import any lib file and get
its libs, however I'm not sure people did this.
I made this while exploring being able to swap out docFn with a stub
in #2305, to avoid functor performance problems. I don't know if that
is going to move forward (or if it is a problem or not,) but after
doing all this work figured I'd put it up anyway :)
Two notable advantages to this approach:
1. when a lib inherits another lib's functions, it doesn't
automatically get put in to the scope of lib
2. when a lib implements a new obscure functions, it doesn't
automatically get put in to the scope of lib
Using the test script (later in this commit) I got the following diff
on the API:
+ diff master fixed-lib
11764a11765,11766
> .types.defaultFunctor
> .types.defaultTypeMerge
11774a11777,11778
> .types.isOptionType
> .types.isType
11781a11786
> .types.mkOptionType
11788a11794
> .types.setType
11795a11802
> .types.types
This means that this commit _adds_ to the API, however I can't find a
way to fix these last remaining discrepancies. At least none are
_removed_.
Test script (run with nix-repl in the PATH):
#!/bin/sh
set -eux
repl() {
suff=${1:-}
echo "(import ./lib)$suff" \
| nix-repl 2>&1
}
attrs_to_check() {
repl "${1:-}" \
| tr ';' $'\n' \
| grep "\.\.\." \
| cut -d' ' -f2 \
| sed -e "s/^/${1:-}./" \
| sort
}
summ() {
repl "${1:-}" \
| tr ' ' $'\n' \
| sort \
| uniq
}
deep_summ() {
suff="${1:-}"
depth="${2:-4}"
depth=$((depth - 1))
summ "$suff"
for attr in $(attrs_to_check "$suff" | grep -v "types.types"); do
if [ $depth -eq 0 ]; then
summ "$attr" | sed -e "s/^/$attr./"
else
deep_summ "$attr" "$depth" | sed -e "s/^/$attr./"
fi
done
}
(
cd nixpkgs
#git add .
#git commit -m "Auto-commit, sorry" || true
git checkout fixed-lib
deep_summ > ../fixed-lib
git checkout master
deep_summ > ../master
)
if diff master fixed-lib; then
echo "SHALLOW MATCH!"
fi
(
cd nixpkgs
git checkout fixed-lib
repl .types
)
nix/nixUnstable, tomcatN and postgresqlNN use `callPackages` pattern, they have .override
attribute, but lack .overrideDerivation and recent .overrideAttrs.
Packages created with `callPackage` have all of those. Because .overrideDerivation function
is used in public, without this we can break code when refactoring callPackage -> callPackages.
This is similar to `overrideDerivation`, but overrides the arguments to
`mkDerivation` instead of the underlying `derivation` call.
Also update `makeOverridable` so that uses of `overrideAttrs` can be
followed by `override` and `overrideDerivation`, i.e. they can be
mix-and-matched.
- Now `pkg.outputUnspecified = true` but this attribute is missing in
every output, so we can recognize whether the user chose or not.
If (s)he didn't choose, we put `pkg.bin or pkg.out or pkg` into
`systemPackages`.
- `outputsToLink` is replaced by `extraOutputsToLink`.
We add extra outputs *regardless* of whether the user chose anything.
It's mainly meant for outputs with docs and debug symbols.
- Note that as a result, some libraries will disappear from system path.
This is like callPackageWith, except that it expects the supplied
function to return a *set* of packages. It will then make the
individual packages overridable.
It's unused, and also a bad idea: because it recursively recomputes
every function argument and there is no sharing, you can get an
exponential (?) blowup in evaluation time. For example, evaluating
‘linuxPackages.kernel’ takes 0.09s and ~13 MiB, but evaluating
‘linuxPackages.kernel.deepOverride {}’ takes 3.6s and ~305 MiB.
