2017-12-25 17:38:45 +00:00
|
|
|
|
# Checks derivation meta and attrs for problems (like brokenness,
|
|
|
|
|
# licenses, etc).
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
check-env: don't execute check-meta.nix 15,000 times
Generated from https://github.com/NixOS/nix/pull/2761:
```
ns calls ns/call
- /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 591200 15026 39.3451
+ /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 8744 308 28.3896
```
more, generated by:
```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > before 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' before | sort > before.list
```
applying this patch, then:
```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > after 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' after | sort > after.list
```
and then diffing before.list and after.list to get:
```
calls
- :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1 7513
+ :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1 154
- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23 7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23 154
- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26 7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26 154
- onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 15026
+ onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 308
```
The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-env -f ./outpaths.nix -qaP --no-name --out-path --arg checkMeta true`:
| stat | before | after | Δ | Δ% |
|:---------------------------|---------------:|---------------:|:----------------|--------:|
| **cpuTime** | 179.915 | 145.543 | 🡖 34.372 | -19.10% |
| **envs-bytes** | 3,900,878,824 | 3,599,483,208 | 🡖 301,395,616 | -7.73% |
| **envs-elements** | 214,426,071 | 185,881,709 | 🡖 28,544,362 | -13.31% |
| **envs-number** | 136,591,891 | 132,026,846 | 🡖 4,565,045 | -3.34% |
| **gc-heapSize** | 11,400,048,640 | 12,314,890,240 | 🡕 914,841,600 | 8.02% |
| **gc-totalBytes** | 25,976,902,560 | 24,510,740,176 | 🡖 1,466,162,384 | -5.64% |
| **list-bytes** | 1,665,290,080 | 1,665,290,080 | 0 | |
| **list-concats** | 7,264,417 | 7,264,417 | 0 | |
| **list-elements** | 208,161,260 | 208,161,260 | 0 | |
| **nrAvoided** | 191,359,386 | 179,693,661 | 🡖 11,665,725 | -6.10% |
| **nrFunctionCalls** | 119,665,062 | 116,348,547 | 🡖 3,316,515 | -2.77% |
| **nrLookups** | 80,996,257 | 76,069,825 | 🡖 4,926,432 | -6.08% |
| **nrOpUpdateValuesCopied** | 213,930,649 | 213,930,649 | 0 | |
| **nrOpUpdates** | 12,025,937 | 12,025,937 | 0 | |
| **nrPrimOpCalls** | 88,105,604 | 86,451,598 | 🡖 1,654,006 | -1.88% |
| **nrThunks** | 196,842,044 | 175,126,701 | 🡖 21,715,343 | -11.03% |
| **sets-bytes** | 7,678,425,776 | 7,285,767,928 | 🡖 392,657,848 | -5.11% |
| **sets-elements** | 310,241,340 | 294,373,227 | 🡖 15,868,113 | -5.11% |
| **sets-number** | 29,079,202 | 27,601,310 | 🡖 1,477,892 | -5.08% |
| **sizes-Attr** | 24 | 24 | 0 | |
| **sizes-Bindings** | 8 | 8 | 0 | |
| **sizes-Env** | 16 | 16 | 0 | |
| **sizes-Value** | 24 | 24 | 0 | |
| **symbols-bytes** | 16,474,666 | 16,474,676 | 🡕 10 | 0.00% |
| **symbols-number** | 376,426 | 376,427 | 🡕 1 | 0.00% |
| **values-bytes** | 6,856,506,288 | 6,316,585,560 | 🡖 539,920,728 | -7.87% |
| **values-number** | 285,687,762 | 263,191,065 | 🡖 22,496,697 | -7.87% |
The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-instantiate ./nixos/release-combined.nix -A tested`:
| stat | before | after | Δ | Δ% |
|:---------------------------|---------------:|---------------:|:----------------|-------:|
| **cpuTime** | 256.071 | 237.531 | 🡖 18.54 | -7.24% |
| **envs-bytes** | 7,111,004,192 | 7,041,478,520 | 🡖 69,525,672 | -0.98% |
| **envs-elements** | 346,236,940 | 339,588,487 | 🡖 6,648,453 | -1.92% |
| **envs-number** | 271,319,292 | 270,298,164 | 🡖 1,021,128 | -0.38% |
| **gc-heapSize** | 8,995,291,136 | 10,110,009,344 | 🡕 1,114,718,208 | 12.39% |
| **gc-totalBytes** | 37,172,737,408 | 36,878,391,888 | 🡖 294,345,520 | -0.79% |
| **list-bytes** | 1,886,162,656 | 1,886,163,472 | 🡕 816 | 0.00% |
| **list-concats** | 6,898,114 | 6,898,114 | 0 | |
| **list-elements** | 235,770,332 | 235,770,434 | 🡕 102 | 0.00% |
| **nrAvoided** | 328,829,821 | 326,618,157 | 🡖 2,211,664 | -0.67% |
| **nrFunctionCalls** | 240,850,845 | 239,998,495 | 🡖 852,350 | -0.35% |
| **nrLookups** | 144,849,632 | 142,126,339 | 🡖 2,723,293 | -1.88% |
| **nrOpUpdateValuesCopied** | 251,032,504 | 251,032,504 | 0 | |
| **nrOpUpdates** | 17,903,110 | 17,903,110 | 0 | |
| **nrPrimOpCalls** | 140,674,913 | 139,485,975 | 🡖 1,188,938 | -0.85% |
| **nrThunks** | 294,643,131 | 288,678,022 | 🡖 5,965,109 | -2.02% |
| **sets-bytes** | 9,464,322,192 | 9,456,172,048 | 🡖 8,150,144 | -0.09% |
| **sets-elements** | 377,474,889 | 377,134,877 | 🡖 340,012 | -0.09% |
| **sets-number** | 50,615,607 | 50,616,875 | 🡕 1,268 | 0.00% |
| **sizes-Attr** | 24 | 24 | 0 | |
| **sizes-Bindings** | 8 | 8 | 0 | |
| **sizes-Env** | 16 | 16 | 0 | |
| **sizes-Value** | 24 | 24 | 0 | |
| **symbols-bytes** | 3,147,102 | 3,147,064 | 🡖 38 | -0.00% |
| **symbols-number** | 82,819 | 82,819 | 0 | |
| **values-bytes** | 11,147,448,768 | 10,996,111,512 | 🡖 151,337,256 | -1.36% |
| **values-number** | 464,477,032 | 458,171,313 | 🡖 6,305,719 | -1.36% |
2019-04-11 17:35:35 +01:00
|
|
|
|
{ lib, config, hostPlatform }:
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
|
|
|
|
let
|
2018-12-18 15:50:53 +00:00
|
|
|
|
# If we're in hydra, we can dispense with the more verbose error
|
|
|
|
|
# messages and make problems easier to spot.
|
|
|
|
|
inHydra = config.inHydra or false;
|
2019-08-15 00:23:52 +01:00
|
|
|
|
getName = attrs: attrs.name or ("${attrs.pname or "«name-missing»"}-${attrs.version or "«version-missing»"}");
|
2018-12-18 15:50:53 +00:00
|
|
|
|
|
2017-07-02 12:52:14 +01:00
|
|
|
|
# See discussion at https://github.com/NixOS/nixpkgs/pull/25304#issuecomment-298385426
|
|
|
|
|
# for why this defaults to false, but I (@copumpkin) want to default it to true soon.
|
|
|
|
|
shouldCheckMeta = config.checkMeta or false;
|
|
|
|
|
|
2019-11-04 01:38:35 +00:00
|
|
|
|
allowUnfree = config.allowUnfree or false
|
|
|
|
|
|| builtins.getEnv "NIXPKGS_ALLOW_UNFREE" == "1";
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
2021-02-23 15:25:18 +00:00
|
|
|
|
allowlist = config.allowlistedLicenses or config.whitelistedLicenses or [];
|
|
|
|
|
blocklist = config.blocklistedLicenses or config.blacklistedLicenses or [];
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
|
|
|
|
areLicenseListsValid =
|
2021-02-23 15:25:18 +00:00
|
|
|
|
if lib.mutuallyExclusive allowlist blocklist then
|
2022-02-17 12:09:23 +00:00
|
|
|
|
true
|
2017-07-02 12:52:14 +01:00
|
|
|
|
else
|
2021-02-23 15:25:18 +00:00
|
|
|
|
throw "allowlistedLicenses and blocklistedLicenses are not mutually exclusive.";
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
|
|
|
|
hasLicense = attrs:
|
|
|
|
|
attrs ? meta.license;
|
|
|
|
|
|
2021-02-23 15:25:18 +00:00
|
|
|
|
hasAllowlistedLicense = assert areLicenseListsValid; attrs:
|
|
|
|
|
hasLicense attrs && lib.lists.any (l: builtins.elem l allowlist) (lib.lists.toList attrs.meta.license);
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
2021-02-23 15:25:18 +00:00
|
|
|
|
hasBlocklistedLicense = assert areLicenseListsValid; attrs:
|
|
|
|
|
hasLicense attrs && lib.lists.any (l: builtins.elem l blocklist) (lib.lists.toList attrs.meta.license);
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
2019-11-04 01:38:35 +00:00
|
|
|
|
allowBroken = config.allowBroken or false
|
|
|
|
|
|| builtins.getEnv "NIXPKGS_ALLOW_BROKEN" == "1";
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
2019-11-04 01:38:35 +00:00
|
|
|
|
allowUnsupportedSystem = config.allowUnsupportedSystem or false
|
|
|
|
|
|| builtins.getEnv "NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM" == "1";
|
2017-05-20 17:44:20 +01:00
|
|
|
|
|
2018-09-23 17:45:03 +01:00
|
|
|
|
isUnfree = licenses: lib.lists.any (l: !l.free or true) licenses;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
2020-04-19 11:04:58 +01:00
|
|
|
|
hasUnfreeLicense = attrs:
|
|
|
|
|
hasLicense attrs &&
|
|
|
|
|
isUnfree (lib.lists.toList attrs.meta.license);
|
|
|
|
|
|
|
|
|
|
isMarkedBroken = attrs: attrs.meta.broken or false;
|
|
|
|
|
|
|
|
|
|
hasUnsupportedPlatform = attrs:
|
|
|
|
|
(!lib.lists.elem hostPlatform.system (attrs.meta.platforms or lib.platforms.all) ||
|
|
|
|
|
lib.lists.elem hostPlatform.system (attrs.meta.badPlatforms or []));
|
|
|
|
|
|
|
|
|
|
isMarkedInsecure = attrs: (attrs.meta.knownVulnerabilities or []) != [];
|
|
|
|
|
|
2017-07-02 12:52:14 +01:00
|
|
|
|
# Alow granular checks to allow only some unfree packages
|
|
|
|
|
# Example:
|
|
|
|
|
# {pkgs, ...}:
|
|
|
|
|
# {
|
|
|
|
|
# allowUnfree = false;
|
2021-02-07 22:05:25 +00:00
|
|
|
|
# allowUnfreePredicate = (x: pkgs.lib.hasPrefix "vscode" x.name);
|
2017-07-02 12:52:14 +01:00
|
|
|
|
# }
|
|
|
|
|
allowUnfreePredicate = config.allowUnfreePredicate or (x: false);
|
|
|
|
|
|
|
|
|
|
# Check whether unfree packages are allowed and if not, whether the
|
|
|
|
|
# package has an unfree license and is not explicitely allowed by the
|
2018-09-23 17:45:03 +01:00
|
|
|
|
# `allowUnfreePredicate` function.
|
2017-07-02 12:52:14 +01:00
|
|
|
|
hasDeniedUnfreeLicense = attrs:
|
2020-04-19 11:04:58 +01:00
|
|
|
|
hasUnfreeLicense attrs &&
|
2017-07-02 12:52:14 +01:00
|
|
|
|
!allowUnfree &&
|
|
|
|
|
!allowUnfreePredicate attrs;
|
|
|
|
|
|
2019-12-15 17:59:43 +00:00
|
|
|
|
allowInsecureDefaultPredicate = x: builtins.elem (getName x) (config.permittedInsecurePackages or []);
|
2017-12-09 20:36:48 +00:00
|
|
|
|
allowInsecurePredicate = x: (config.allowInsecurePredicate or allowInsecureDefaultPredicate) x;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
|
|
|
|
hasAllowedInsecure = attrs:
|
2020-04-19 11:04:58 +01:00
|
|
|
|
!(isMarkedInsecure attrs) ||
|
2017-07-02 12:52:14 +01:00
|
|
|
|
allowInsecurePredicate attrs ||
|
2019-11-04 01:38:35 +00:00
|
|
|
|
builtins.getEnv "NIXPKGS_ALLOW_INSECURE" == "1";
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
2019-10-27 10:15:48 +00:00
|
|
|
|
showLicense = license: toString (map (l: l.shortName or "unknown") (lib.lists.toList license));
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
check-env: don't execute check-meta.nix 15,000 times
Generated from https://github.com/NixOS/nix/pull/2761:
```
ns calls ns/call
- /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 591200 15026 39.3451
+ /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 8744 308 28.3896
```
more, generated by:
```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > before 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' before | sort > before.list
```
applying this patch, then:
```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > after 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' after | sort > after.list
```
and then diffing before.list and after.list to get:
```
calls
- :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1 7513
+ :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1 154
- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23 7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23 154
- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26 7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26 154
- onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 15026
+ onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 308
```
The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-env -f ./outpaths.nix -qaP --no-name --out-path --arg checkMeta true`:
| stat | before | after | Δ | Δ% |
|:---------------------------|---------------:|---------------:|:----------------|--------:|
| **cpuTime** | 179.915 | 145.543 | 🡖 34.372 | -19.10% |
| **envs-bytes** | 3,900,878,824 | 3,599,483,208 | 🡖 301,395,616 | -7.73% |
| **envs-elements** | 214,426,071 | 185,881,709 | 🡖 28,544,362 | -13.31% |
| **envs-number** | 136,591,891 | 132,026,846 | 🡖 4,565,045 | -3.34% |
| **gc-heapSize** | 11,400,048,640 | 12,314,890,240 | 🡕 914,841,600 | 8.02% |
| **gc-totalBytes** | 25,976,902,560 | 24,510,740,176 | 🡖 1,466,162,384 | -5.64% |
| **list-bytes** | 1,665,290,080 | 1,665,290,080 | 0 | |
| **list-concats** | 7,264,417 | 7,264,417 | 0 | |
| **list-elements** | 208,161,260 | 208,161,260 | 0 | |
| **nrAvoided** | 191,359,386 | 179,693,661 | 🡖 11,665,725 | -6.10% |
| **nrFunctionCalls** | 119,665,062 | 116,348,547 | 🡖 3,316,515 | -2.77% |
| **nrLookups** | 80,996,257 | 76,069,825 | 🡖 4,926,432 | -6.08% |
| **nrOpUpdateValuesCopied** | 213,930,649 | 213,930,649 | 0 | |
| **nrOpUpdates** | 12,025,937 | 12,025,937 | 0 | |
| **nrPrimOpCalls** | 88,105,604 | 86,451,598 | 🡖 1,654,006 | -1.88% |
| **nrThunks** | 196,842,044 | 175,126,701 | 🡖 21,715,343 | -11.03% |
| **sets-bytes** | 7,678,425,776 | 7,285,767,928 | 🡖 392,657,848 | -5.11% |
| **sets-elements** | 310,241,340 | 294,373,227 | 🡖 15,868,113 | -5.11% |
| **sets-number** | 29,079,202 | 27,601,310 | 🡖 1,477,892 | -5.08% |
| **sizes-Attr** | 24 | 24 | 0 | |
| **sizes-Bindings** | 8 | 8 | 0 | |
| **sizes-Env** | 16 | 16 | 0 | |
| **sizes-Value** | 24 | 24 | 0 | |
| **symbols-bytes** | 16,474,666 | 16,474,676 | 🡕 10 | 0.00% |
| **symbols-number** | 376,426 | 376,427 | 🡕 1 | 0.00% |
| **values-bytes** | 6,856,506,288 | 6,316,585,560 | 🡖 539,920,728 | -7.87% |
| **values-number** | 285,687,762 | 263,191,065 | 🡖 22,496,697 | -7.87% |
The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-instantiate ./nixos/release-combined.nix -A tested`:
| stat | before | after | Δ | Δ% |
|:---------------------------|---------------:|---------------:|:----------------|-------:|
| **cpuTime** | 256.071 | 237.531 | 🡖 18.54 | -7.24% |
| **envs-bytes** | 7,111,004,192 | 7,041,478,520 | 🡖 69,525,672 | -0.98% |
| **envs-elements** | 346,236,940 | 339,588,487 | 🡖 6,648,453 | -1.92% |
| **envs-number** | 271,319,292 | 270,298,164 | 🡖 1,021,128 | -0.38% |
| **gc-heapSize** | 8,995,291,136 | 10,110,009,344 | 🡕 1,114,718,208 | 12.39% |
| **gc-totalBytes** | 37,172,737,408 | 36,878,391,888 | 🡖 294,345,520 | -0.79% |
| **list-bytes** | 1,886,162,656 | 1,886,163,472 | 🡕 816 | 0.00% |
| **list-concats** | 6,898,114 | 6,898,114 | 0 | |
| **list-elements** | 235,770,332 | 235,770,434 | 🡕 102 | 0.00% |
| **nrAvoided** | 328,829,821 | 326,618,157 | 🡖 2,211,664 | -0.67% |
| **nrFunctionCalls** | 240,850,845 | 239,998,495 | 🡖 852,350 | -0.35% |
| **nrLookups** | 144,849,632 | 142,126,339 | 🡖 2,723,293 | -1.88% |
| **nrOpUpdateValuesCopied** | 251,032,504 | 251,032,504 | 0 | |
| **nrOpUpdates** | 17,903,110 | 17,903,110 | 0 | |
| **nrPrimOpCalls** | 140,674,913 | 139,485,975 | 🡖 1,188,938 | -0.85% |
| **nrThunks** | 294,643,131 | 288,678,022 | 🡖 5,965,109 | -2.02% |
| **sets-bytes** | 9,464,322,192 | 9,456,172,048 | 🡖 8,150,144 | -0.09% |
| **sets-elements** | 377,474,889 | 377,134,877 | 🡖 340,012 | -0.09% |
| **sets-number** | 50,615,607 | 50,616,875 | 🡕 1,268 | 0.00% |
| **sizes-Attr** | 24 | 24 | 0 | |
| **sizes-Bindings** | 8 | 8 | 0 | |
| **sizes-Env** | 16 | 16 | 0 | |
| **sizes-Value** | 24 | 24 | 0 | |
| **symbols-bytes** | 3,147,102 | 3,147,064 | 🡖 38 | -0.00% |
| **symbols-number** | 82,819 | 82,819 | 0 | |
| **values-bytes** | 11,147,448,768 | 10,996,111,512 | 🡖 151,337,256 | -1.36% |
| **values-number** | 464,477,032 | 458,171,313 | 🡖 6,305,719 | -1.36% |
2019-04-11 17:35:35 +01:00
|
|
|
|
pos_str = meta: meta.position or "«unknown-file»";
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
|
|
|
|
remediation = {
|
2021-02-23 15:25:18 +00:00
|
|
|
|
unfree = remediate_allowlist "Unfree" remediate_unfree_predicate;
|
|
|
|
|
broken = remediate_allowlist "Broken" (x: "");
|
|
|
|
|
unsupported = remediate_allowlist "UnsupportedSystem" (x: "");
|
|
|
|
|
blocklisted = x: "";
|
2017-07-02 12:52:14 +01:00
|
|
|
|
insecure = remediate_insecure;
|
2018-09-18 14:30:59 +01:00
|
|
|
|
broken-outputs = remediateOutputsToInstall;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
unknown-meta = x: "";
|
|
|
|
|
};
|
2020-10-27 09:01:32 +00:00
|
|
|
|
remediation_env_var = allow_attr: {
|
|
|
|
|
Unfree = "NIXPKGS_ALLOW_UNFREE";
|
|
|
|
|
Broken = "NIXPKGS_ALLOW_BROKEN";
|
|
|
|
|
UnsupportedSystem = "NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM";
|
|
|
|
|
}.${allow_attr};
|
|
|
|
|
remediation_phrase = allow_attr: {
|
|
|
|
|
Unfree = "unfree packages";
|
|
|
|
|
Broken = "broken packages";
|
|
|
|
|
UnsupportedSystem = "packages that are unsupported for this system";
|
|
|
|
|
}.${allow_attr};
|
2020-06-05 20:39:08 +01:00
|
|
|
|
remediate_unfree_predicate = attrs:
|
|
|
|
|
''
|
|
|
|
|
|
2021-02-23 15:25:18 +00:00
|
|
|
|
Alternatively you can configure a predicate to allow specific packages:
|
2020-06-05 20:39:08 +01:00
|
|
|
|
{ nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
|
|
|
|
|
"${lib.getName attrs}"
|
|
|
|
|
];
|
|
|
|
|
}
|
|
|
|
|
'';
|
|
|
|
|
|
2021-11-13 04:47:23 +00:00
|
|
|
|
# flakeNote will be printed in the remediation messages below.
|
|
|
|
|
flakeNote = "
|
|
|
|
|
Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
|
|
|
|
|
(Flake) command, `--impure` must be passed in order to read this
|
|
|
|
|
environment variable.
|
|
|
|
|
";
|
|
|
|
|
|
2021-02-23 15:25:18 +00:00
|
|
|
|
remediate_allowlist = allow_attr: rebuild_amendment: attrs:
|
2017-07-02 12:52:14 +01:00
|
|
|
|
''
|
2020-10-27 09:01:32 +00:00
|
|
|
|
a) To temporarily allow ${remediation_phrase allow_attr}, you can use an environment variable
|
|
|
|
|
for a single invocation of the nix tools.
|
|
|
|
|
|
|
|
|
|
$ export ${remediation_env_var allow_attr}=1
|
2021-11-13 04:47:23 +00:00
|
|
|
|
${flakeNote}
|
2020-10-27 09:01:32 +00:00
|
|
|
|
b) For `nixos-rebuild` you can set
|
2017-07-02 12:52:14 +01:00
|
|
|
|
{ nixpkgs.config.allow${allow_attr} = true; }
|
|
|
|
|
in configuration.nix to override this.
|
2020-06-05 20:39:08 +01:00
|
|
|
|
${rebuild_amendment attrs}
|
2020-10-27 09:01:32 +00:00
|
|
|
|
c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
|
2017-07-02 12:52:14 +01:00
|
|
|
|
{ allow${allow_attr} = true; }
|
|
|
|
|
to ~/.config/nixpkgs/config.nix.
|
|
|
|
|
'';
|
|
|
|
|
|
|
|
|
|
remediate_insecure = attrs:
|
|
|
|
|
''
|
|
|
|
|
|
|
|
|
|
Known issues:
|
2018-02-10 21:47:05 +00:00
|
|
|
|
'' + (lib.concatStrings (map (issue: " - ${issue}\n") attrs.meta.knownVulnerabilities)) + ''
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
2021-02-23 15:25:18 +00:00
|
|
|
|
You can install it anyway by allowing this package, using the
|
2017-07-02 12:52:14 +01:00
|
|
|
|
following methods:
|
2020-06-11 03:25:07 +01:00
|
|
|
|
|
|
|
|
|
a) To temporarily allow all insecure packages, you can use an environment
|
|
|
|
|
variable for a single invocation of the nix tools:
|
|
|
|
|
|
|
|
|
|
$ export NIXPKGS_ALLOW_INSECURE=1
|
2021-11-13 04:47:23 +00:00
|
|
|
|
${flakeNote}
|
2020-06-10 09:16:31 +01:00
|
|
|
|
b) for `nixos-rebuild` you can add ‘${getName attrs}’ to
|
2017-07-02 12:52:14 +01:00
|
|
|
|
`nixpkgs.config.permittedInsecurePackages` in the configuration.nix,
|
|
|
|
|
like so:
|
|
|
|
|
|
|
|
|
|
{
|
|
|
|
|
nixpkgs.config.permittedInsecurePackages = [
|
2019-08-15 00:23:52 +01:00
|
|
|
|
"${getName attrs}"
|
2017-07-02 12:52:14 +01:00
|
|
|
|
];
|
|
|
|
|
}
|
|
|
|
|
|
2020-06-10 09:16:31 +01:00
|
|
|
|
c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
|
2020-06-11 03:25:07 +01:00
|
|
|
|
‘${getName attrs}’ to `permittedInsecurePackages` in
|
|
|
|
|
~/.config/nixpkgs/config.nix, like so:
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
|
|
|
|
{
|
|
|
|
|
permittedInsecurePackages = [
|
2019-08-15 00:23:52 +01:00
|
|
|
|
"${getName attrs}"
|
2017-07-02 12:52:14 +01:00
|
|
|
|
];
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
'';
|
|
|
|
|
|
2018-09-18 14:30:59 +01:00
|
|
|
|
remediateOutputsToInstall = attrs: let
|
|
|
|
|
expectedOutputs = attrs.meta.outputsToInstall or [];
|
|
|
|
|
actualOutputs = attrs.outputs or [ "out" ];
|
|
|
|
|
missingOutputs = builtins.filter (output: ! builtins.elem output actualOutputs) expectedOutputs;
|
|
|
|
|
in ''
|
2019-08-15 00:23:52 +01:00
|
|
|
|
The package ${getName attrs} has set meta.outputsToInstall to: ${builtins.concatStringsSep ", " expectedOutputs}
|
2018-09-18 14:30:59 +01:00
|
|
|
|
|
2019-08-15 00:23:52 +01:00
|
|
|
|
however ${getName attrs} only has the outputs: ${builtins.concatStringsSep ", " actualOutputs}
|
2018-09-18 14:30:59 +01:00
|
|
|
|
|
|
|
|
|
and is missing the following ouputs:
|
|
|
|
|
|
|
|
|
|
${lib.concatStrings (builtins.map (output: " - ${output}\n") missingOutputs)}
|
|
|
|
|
'';
|
|
|
|
|
|
check-env: don't execute check-meta.nix 15,000 times
Generated from https://github.com/NixOS/nix/pull/2761:
```
ns calls ns/call
- /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 591200 15026 39.3451
+ /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 8744 308 28.3896
```
more, generated by:
```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > before 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' before | sort > before.list
```
applying this patch, then:
```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > after 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' after | sort > after.list
```
and then diffing before.list and after.list to get:
```
calls
- :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1 7513
+ :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1 154
- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23 7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23 154
- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26 7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26 154
- onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 15026
+ onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 308
```
The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-env -f ./outpaths.nix -qaP --no-name --out-path --arg checkMeta true`:
| stat | before | after | Δ | Δ% |
|:---------------------------|---------------:|---------------:|:----------------|--------:|
| **cpuTime** | 179.915 | 145.543 | 🡖 34.372 | -19.10% |
| **envs-bytes** | 3,900,878,824 | 3,599,483,208 | 🡖 301,395,616 | -7.73% |
| **envs-elements** | 214,426,071 | 185,881,709 | 🡖 28,544,362 | -13.31% |
| **envs-number** | 136,591,891 | 132,026,846 | 🡖 4,565,045 | -3.34% |
| **gc-heapSize** | 11,400,048,640 | 12,314,890,240 | 🡕 914,841,600 | 8.02% |
| **gc-totalBytes** | 25,976,902,560 | 24,510,740,176 | 🡖 1,466,162,384 | -5.64% |
| **list-bytes** | 1,665,290,080 | 1,665,290,080 | 0 | |
| **list-concats** | 7,264,417 | 7,264,417 | 0 | |
| **list-elements** | 208,161,260 | 208,161,260 | 0 | |
| **nrAvoided** | 191,359,386 | 179,693,661 | 🡖 11,665,725 | -6.10% |
| **nrFunctionCalls** | 119,665,062 | 116,348,547 | 🡖 3,316,515 | -2.77% |
| **nrLookups** | 80,996,257 | 76,069,825 | 🡖 4,926,432 | -6.08% |
| **nrOpUpdateValuesCopied** | 213,930,649 | 213,930,649 | 0 | |
| **nrOpUpdates** | 12,025,937 | 12,025,937 | 0 | |
| **nrPrimOpCalls** | 88,105,604 | 86,451,598 | 🡖 1,654,006 | -1.88% |
| **nrThunks** | 196,842,044 | 175,126,701 | 🡖 21,715,343 | -11.03% |
| **sets-bytes** | 7,678,425,776 | 7,285,767,928 | 🡖 392,657,848 | -5.11% |
| **sets-elements** | 310,241,340 | 294,373,227 | 🡖 15,868,113 | -5.11% |
| **sets-number** | 29,079,202 | 27,601,310 | 🡖 1,477,892 | -5.08% |
| **sizes-Attr** | 24 | 24 | 0 | |
| **sizes-Bindings** | 8 | 8 | 0 | |
| **sizes-Env** | 16 | 16 | 0 | |
| **sizes-Value** | 24 | 24 | 0 | |
| **symbols-bytes** | 16,474,666 | 16,474,676 | 🡕 10 | 0.00% |
| **symbols-number** | 376,426 | 376,427 | 🡕 1 | 0.00% |
| **values-bytes** | 6,856,506,288 | 6,316,585,560 | 🡖 539,920,728 | -7.87% |
| **values-number** | 285,687,762 | 263,191,065 | 🡖 22,496,697 | -7.87% |
The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-instantiate ./nixos/release-combined.nix -A tested`:
| stat | before | after | Δ | Δ% |
|:---------------------------|---------------:|---------------:|:----------------|-------:|
| **cpuTime** | 256.071 | 237.531 | 🡖 18.54 | -7.24% |
| **envs-bytes** | 7,111,004,192 | 7,041,478,520 | 🡖 69,525,672 | -0.98% |
| **envs-elements** | 346,236,940 | 339,588,487 | 🡖 6,648,453 | -1.92% |
| **envs-number** | 271,319,292 | 270,298,164 | 🡖 1,021,128 | -0.38% |
| **gc-heapSize** | 8,995,291,136 | 10,110,009,344 | 🡕 1,114,718,208 | 12.39% |
| **gc-totalBytes** | 37,172,737,408 | 36,878,391,888 | 🡖 294,345,520 | -0.79% |
| **list-bytes** | 1,886,162,656 | 1,886,163,472 | 🡕 816 | 0.00% |
| **list-concats** | 6,898,114 | 6,898,114 | 0 | |
| **list-elements** | 235,770,332 | 235,770,434 | 🡕 102 | 0.00% |
| **nrAvoided** | 328,829,821 | 326,618,157 | 🡖 2,211,664 | -0.67% |
| **nrFunctionCalls** | 240,850,845 | 239,998,495 | 🡖 852,350 | -0.35% |
| **nrLookups** | 144,849,632 | 142,126,339 | 🡖 2,723,293 | -1.88% |
| **nrOpUpdateValuesCopied** | 251,032,504 | 251,032,504 | 0 | |
| **nrOpUpdates** | 17,903,110 | 17,903,110 | 0 | |
| **nrPrimOpCalls** | 140,674,913 | 139,485,975 | 🡖 1,188,938 | -0.85% |
| **nrThunks** | 294,643,131 | 288,678,022 | 🡖 5,965,109 | -2.02% |
| **sets-bytes** | 9,464,322,192 | 9,456,172,048 | 🡖 8,150,144 | -0.09% |
| **sets-elements** | 377,474,889 | 377,134,877 | 🡖 340,012 | -0.09% |
| **sets-number** | 50,615,607 | 50,616,875 | 🡕 1,268 | 0.00% |
| **sizes-Attr** | 24 | 24 | 0 | |
| **sizes-Bindings** | 8 | 8 | 0 | |
| **sizes-Env** | 16 | 16 | 0 | |
| **sizes-Value** | 24 | 24 | 0 | |
| **symbols-bytes** | 3,147,102 | 3,147,064 | 🡖 38 | -0.00% |
| **symbols-number** | 82,819 | 82,819 | 0 | |
| **values-bytes** | 11,147,448,768 | 10,996,111,512 | 🡖 151,337,256 | -1.36% |
| **values-number** | 464,477,032 | 458,171,313 | 🡖 6,305,719 | -1.36% |
2019-04-11 17:35:35 +01:00
|
|
|
|
handleEvalIssue = { meta, attrs }: { reason , errormsg ? "" }:
|
2017-12-12 23:06:55 +00:00
|
|
|
|
let
|
2018-12-18 15:50:53 +00:00
|
|
|
|
msg = if inHydra
|
2019-08-15 00:23:52 +01:00
|
|
|
|
then "Failed to evaluate ${getName attrs}: «${reason}»: ${errormsg}"
|
2018-12-18 15:50:53 +00:00
|
|
|
|
else ''
|
2019-08-15 00:23:52 +01:00
|
|
|
|
Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
|
2017-12-12 23:06:55 +00:00
|
|
|
|
|
2018-12-18 15:50:53 +00:00
|
|
|
|
'' + (builtins.getAttr reason remediation) attrs;
|
2017-12-12 23:06:55 +00:00
|
|
|
|
|
2019-08-13 22:52:01 +01:00
|
|
|
|
handler = if config ? handleEvalIssue
|
2017-12-12 23:06:55 +00:00
|
|
|
|
then config.handleEvalIssue reason
|
|
|
|
|
else throw;
|
|
|
|
|
in handler msg;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
metaTypes = with lib.types; rec {
|
|
|
|
|
# These keys are documented
|
|
|
|
|
description = str;
|
2021-04-26 05:19:49 +01:00
|
|
|
|
mainProgram = str;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
longDescription = str;
|
|
|
|
|
branch = str;
|
2017-12-05 11:57:11 +00:00
|
|
|
|
homepage = either (listOf str) str;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
downloadPage = str;
|
2019-05-10 17:55:29 +01:00
|
|
|
|
changelog = either (listOf str) str;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
license = either (listOf lib.types.attrs) (either lib.types.attrs str);
|
2018-03-04 03:18:28 +00:00
|
|
|
|
maintainers = listOf (attrsOf str);
|
2017-07-02 12:52:14 +01:00
|
|
|
|
priority = int;
|
2019-04-28 01:37:42 +01:00
|
|
|
|
platforms = listOf str;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
hydraPlatforms = listOf str;
|
|
|
|
|
broken = bool;
|
2020-04-19 11:04:58 +01:00
|
|
|
|
unfree = bool;
|
|
|
|
|
unsupported = bool;
|
|
|
|
|
insecure = bool;
|
dovecot, opensmtpd: add link to test in `meta.tests`
Rationale
---------
Currently, tests are hard to discover. For instance, someone updating
`dovecot` might not notice that the interaction of `dovecot` with
`opensmtpd` is handled in the `opensmtpd.nix` test.
And even for someone updating `opensmtpd`, it requires manual work to go
check in `nixos/tests` whether there is actually a test, especially
given not so many packages in `nixpkgs` have tests and this is thus most
of the time useless.
Finally, for the reviewer, it is much easier to check that the “Tested
via one or more NixOS test(s)” has been checked if the file modified
already includes the list of relevant tests.
Implementation
--------------
Currently, this commit only adds the metadata in the package. Each
element of the `meta.tests` attribute is a derivation that, when it
builds successfully, means the test has passed (ie. following the same
convention as NixOS tests).
Future Work
-----------
In the future, the tools could be made aware of this `meta.tests`
attribute, and for instance a `--with-tests` could be added to
`nix-build` so that it also builds all the tests. Or a `--without-tests`
to build without all the tests. @Profpatsch described in his NixCon talk
such systems.
Another thing that would help in the future would be the possibility to
reasonably easily have cross-derivation nix tests without the whole
NixOS VM stack. @7c6f434c already proposed such a system.
This RFC currently handles none of these concerns. Only the addition of
`meta.tests` as metadata to be used by maintainers to remember to run
relevant tests.
2018-08-04 04:18:28 +01:00
|
|
|
|
# TODO: refactor once something like Profpatsch's types-simple will land
|
2018-11-11 13:55:23 +00:00
|
|
|
|
# This is currently dead code due to https://github.com/NixOS/nix/issues/2532
|
dovecot, opensmtpd: add link to test in `meta.tests`
Rationale
---------
Currently, tests are hard to discover. For instance, someone updating
`dovecot` might not notice that the interaction of `dovecot` with
`opensmtpd` is handled in the `opensmtpd.nix` test.
And even for someone updating `opensmtpd`, it requires manual work to go
check in `nixos/tests` whether there is actually a test, especially
given not so many packages in `nixpkgs` have tests and this is thus most
of the time useless.
Finally, for the reviewer, it is much easier to check that the “Tested
via one or more NixOS test(s)” has been checked if the file modified
already includes the list of relevant tests.
Implementation
--------------
Currently, this commit only adds the metadata in the package. Each
element of the `meta.tests` attribute is a derivation that, when it
builds successfully, means the test has passed (ie. following the same
convention as NixOS tests).
Future Work
-----------
In the future, the tools could be made aware of this `meta.tests`
attribute, and for instance a `--with-tests` could be added to
`nix-build` so that it also builds all the tests. Or a `--without-tests`
to build without all the tests. @Profpatsch described in his NixCon talk
such systems.
Another thing that would help in the future would be the possibility to
reasonably easily have cross-derivation nix tests without the whole
NixOS VM stack. @7c6f434c already proposed such a system.
This RFC currently handles none of these concerns. Only the addition of
`meta.tests` as metadata to be used by maintainers to remember to run
relevant tests.
2018-08-04 04:18:28 +01:00
|
|
|
|
tests = attrsOf (mkOptionType {
|
|
|
|
|
name = "test";
|
2018-11-11 11:30:07 +00:00
|
|
|
|
check = x: x == {} || ( # Accept {} for tests that are unsupported
|
|
|
|
|
isDerivation x &&
|
2018-11-11 13:47:49 +00:00
|
|
|
|
x ? meta.timeout
|
2018-11-11 11:30:07 +00:00
|
|
|
|
);
|
dovecot, opensmtpd: add link to test in `meta.tests`
Rationale
---------
Currently, tests are hard to discover. For instance, someone updating
`dovecot` might not notice that the interaction of `dovecot` with
`opensmtpd` is handled in the `opensmtpd.nix` test.
And even for someone updating `opensmtpd`, it requires manual work to go
check in `nixos/tests` whether there is actually a test, especially
given not so many packages in `nixpkgs` have tests and this is thus most
of the time useless.
Finally, for the reviewer, it is much easier to check that the “Tested
via one or more NixOS test(s)” has been checked if the file modified
already includes the list of relevant tests.
Implementation
--------------
Currently, this commit only adds the metadata in the package. Each
element of the `meta.tests` attribute is a derivation that, when it
builds successfully, means the test has passed (ie. following the same
convention as NixOS tests).
Future Work
-----------
In the future, the tools could be made aware of this `meta.tests`
attribute, and for instance a `--with-tests` could be added to
`nix-build` so that it also builds all the tests. Or a `--without-tests`
to build without all the tests. @Profpatsch described in his NixCon talk
such systems.
Another thing that would help in the future would be the possibility to
reasonably easily have cross-derivation nix tests without the whole
NixOS VM stack. @7c6f434c already proposed such a system.
This RFC currently handles none of these concerns. Only the addition of
`meta.tests` as metadata to be used by maintainers to remember to run
relevant tests.
2018-08-04 04:18:28 +01:00
|
|
|
|
merge = lib.options.mergeOneOption;
|
|
|
|
|
});
|
|
|
|
|
timeout = int;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
|
|
|
|
# Weirder stuff that doesn't appear in the documentation?
|
2021-08-07 00:15:04 +01:00
|
|
|
|
maxSilent = int;
|
2017-12-14 12:36:26 +00:00
|
|
|
|
knownVulnerabilities = listOf str;
|
2018-01-03 15:22:35 +00:00
|
|
|
|
name = str;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
version = str;
|
|
|
|
|
tag = str;
|
|
|
|
|
executables = listOf str;
|
|
|
|
|
outputsToInstall = listOf str;
|
|
|
|
|
position = str;
|
2018-02-17 18:44:23 +00:00
|
|
|
|
available = bool;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
isBuildPythonPackage = platforms;
|
2018-01-09 13:25:24 +00:00
|
|
|
|
schedulingPriority = int;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
isFcitxEngine = bool;
|
|
|
|
|
isIbusEngine = bool;
|
2017-12-05 11:57:11 +00:00
|
|
|
|
isGutenprint = bool;
|
2018-03-27 13:12:14 +01:00
|
|
|
|
badPlatforms = platforms;
|
2017-07-02 12:52:14 +01:00
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
checkMetaAttr = k: v:
|
|
|
|
|
if metaTypes?${k} then
|
2017-12-05 11:55:48 +00:00
|
|
|
|
if metaTypes.${k}.check v then null else "key '${k}' has a value ${toString v} of an invalid type ${builtins.typeOf v}; expected ${metaTypes.${k}.description}"
|
2017-07-02 12:52:14 +01:00
|
|
|
|
else "key '${k}' is unrecognized; expected one of: \n\t [${lib.concatMapStringsSep ", " (x: "'${x}'") (lib.attrNames metaTypes)}]";
|
|
|
|
|
checkMeta = meta: if shouldCheckMeta then lib.remove null (lib.mapAttrsToList checkMetaAttr meta) else [];
|
|
|
|
|
|
2018-09-18 14:30:59 +01:00
|
|
|
|
checkOutputsToInstall = attrs: let
|
|
|
|
|
expectedOutputs = attrs.meta.outputsToInstall or [];
|
|
|
|
|
actualOutputs = attrs.outputs or [ "out" ];
|
|
|
|
|
missingOutputs = builtins.filter (output: ! builtins.elem output actualOutputs) expectedOutputs;
|
|
|
|
|
in if shouldCheckMeta
|
|
|
|
|
then builtins.length missingOutputs > 0
|
|
|
|
|
else false;
|
|
|
|
|
|
2017-07-02 12:52:14 +01:00
|
|
|
|
# Check if a derivation is valid, that is whether it passes checks for
|
|
|
|
|
# e.g brokenness or license.
|
|
|
|
|
#
|
|
|
|
|
# Return { valid: Bool } and additionally
|
|
|
|
|
# { reason: String; errormsg: String } if it is not valid, where
|
2021-02-23 15:25:18 +00:00
|
|
|
|
# reason is one of "unfree", "blocklisted", "broken", "insecure", ...
|
2020-04-19 11:04:58 +01:00
|
|
|
|
# Along with a boolean flag for each reason
|
2017-07-02 12:52:14 +01:00
|
|
|
|
checkValidity = attrs:
|
2020-04-19 11:04:58 +01:00
|
|
|
|
{
|
|
|
|
|
unfree = hasUnfreeLicense attrs;
|
|
|
|
|
broken = isMarkedBroken attrs;
|
|
|
|
|
unsupported = hasUnsupportedPlatform attrs;
|
|
|
|
|
insecure = isMarkedInsecure attrs;
|
|
|
|
|
}
|
2021-02-23 15:25:18 +00:00
|
|
|
|
// (if hasDeniedUnfreeLicense attrs && !(hasAllowlistedLicense attrs) then
|
2017-07-02 12:52:14 +01:00
|
|
|
|
{ valid = false; reason = "unfree"; errormsg = "has an unfree license (‘${showLicense attrs.meta.license}’)"; }
|
2021-02-23 15:25:18 +00:00
|
|
|
|
else if hasBlocklistedLicense attrs then
|
|
|
|
|
{ valid = false; reason = "blocklisted"; errormsg = "has a blocklisted license (‘${showLicense attrs.meta.license}’)"; }
|
2017-07-02 12:52:14 +01:00
|
|
|
|
else if !allowBroken && attrs.meta.broken or false then
|
|
|
|
|
{ valid = false; reason = "broken"; errormsg = "is marked as broken"; }
|
2020-04-19 11:04:58 +01:00
|
|
|
|
else if !allowUnsupportedSystem && hasUnsupportedPlatform attrs then
|
2019-04-28 01:37:42 +01:00
|
|
|
|
{ valid = false; reason = "unsupported"; errormsg = "is not supported on ‘${hostPlatform.system}’"; }
|
2017-07-02 12:52:14 +01:00
|
|
|
|
else if !(hasAllowedInsecure attrs) then
|
|
|
|
|
{ valid = false; reason = "insecure"; errormsg = "is marked as insecure"; }
|
2018-09-18 14:30:59 +01:00
|
|
|
|
else if checkOutputsToInstall attrs then
|
|
|
|
|
{ valid = false; reason = "broken-outputs"; errormsg = "has invalid meta.outputsToInstall"; }
|
2017-07-02 12:52:14 +01:00
|
|
|
|
else let res = checkMeta (attrs.meta or {}); in if res != [] then
|
|
|
|
|
{ valid = false; reason = "unknown-meta"; errormsg = "has an invalid meta attrset:${lib.concatMapStrings (x: "\n\t - " + x) res}"; }
|
2020-04-19 11:04:58 +01:00
|
|
|
|
else { valid = true; });
|
2017-07-02 12:52:14 +01:00
|
|
|
|
|
check-env: don't execute check-meta.nix 15,000 times
Generated from https://github.com/NixOS/nix/pull/2761:
```
ns calls ns/call
- /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 591200 15026 39.3451
+ /home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:22:5 8744 308 28.3896
```
more, generated by:
```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > before 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' before | sort > before.list
```
applying this patch, then:
```
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate ./pkgs/top-level/release.nix -A unstable > after 2>&1
$ jq -r '.functions | map((.name + ":" + .file + ":" + (.line|tostring) + ":" + (.column|tostring) + " " + (.count|tostring))) | .[]' after | sort > after.list
```
and then diffing before.list and after.list to get:
```
calls
- :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1 7513
+ :/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:4:1 154
- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23 7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:23 154
- mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26 7513
+ mutuallyExclusive:/home/grahamc/projects/github.com/NixOS/nixpkgs/lib/lists.nix:658:26 154
- onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 15026
+ onlyLicenses:/home/grahamc/projects/github.com/NixOS/nixpkgs/pkgs/stdenv/generic/check-meta.nix:21:18 308
```
The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-env -f ./outpaths.nix -qaP --no-name --out-path --arg checkMeta true`:
| stat | before | after | Δ | Δ% |
|:---------------------------|---------------:|---------------:|:----------------|--------:|
| **cpuTime** | 179.915 | 145.543 | 🡖 34.372 | -19.10% |
| **envs-bytes** | 3,900,878,824 | 3,599,483,208 | 🡖 301,395,616 | -7.73% |
| **envs-elements** | 214,426,071 | 185,881,709 | 🡖 28,544,362 | -13.31% |
| **envs-number** | 136,591,891 | 132,026,846 | 🡖 4,565,045 | -3.34% |
| **gc-heapSize** | 11,400,048,640 | 12,314,890,240 | 🡕 914,841,600 | 8.02% |
| **gc-totalBytes** | 25,976,902,560 | 24,510,740,176 | 🡖 1,466,162,384 | -5.64% |
| **list-bytes** | 1,665,290,080 | 1,665,290,080 | 0 | |
| **list-concats** | 7,264,417 | 7,264,417 | 0 | |
| **list-elements** | 208,161,260 | 208,161,260 | 0 | |
| **nrAvoided** | 191,359,386 | 179,693,661 | 🡖 11,665,725 | -6.10% |
| **nrFunctionCalls** | 119,665,062 | 116,348,547 | 🡖 3,316,515 | -2.77% |
| **nrLookups** | 80,996,257 | 76,069,825 | 🡖 4,926,432 | -6.08% |
| **nrOpUpdateValuesCopied** | 213,930,649 | 213,930,649 | 0 | |
| **nrOpUpdates** | 12,025,937 | 12,025,937 | 0 | |
| **nrPrimOpCalls** | 88,105,604 | 86,451,598 | 🡖 1,654,006 | -1.88% |
| **nrThunks** | 196,842,044 | 175,126,701 | 🡖 21,715,343 | -11.03% |
| **sets-bytes** | 7,678,425,776 | 7,285,767,928 | 🡖 392,657,848 | -5.11% |
| **sets-elements** | 310,241,340 | 294,373,227 | 🡖 15,868,113 | -5.11% |
| **sets-number** | 29,079,202 | 27,601,310 | 🡖 1,477,892 | -5.08% |
| **sizes-Attr** | 24 | 24 | 0 | |
| **sizes-Bindings** | 8 | 8 | 0 | |
| **sizes-Env** | 16 | 16 | 0 | |
| **sizes-Value** | 24 | 24 | 0 | |
| **symbols-bytes** | 16,474,666 | 16,474,676 | 🡕 10 | 0.00% |
| **symbols-number** | 376,426 | 376,427 | 🡕 1 | 0.00% |
| **values-bytes** | 6,856,506,288 | 6,316,585,560 | 🡖 539,920,728 | -7.87% |
| **values-number** | 285,687,762 | 263,191,065 | 🡖 22,496,697 | -7.87% |
The following information is from `NIX_SHOW_STATS=1 GC_INITIAL_HEAP_SIZE=4g nix-instantiate ./nixos/release-combined.nix -A tested`:
| stat | before | after | Δ | Δ% |
|:---------------------------|---------------:|---------------:|:----------------|-------:|
| **cpuTime** | 256.071 | 237.531 | 🡖 18.54 | -7.24% |
| **envs-bytes** | 7,111,004,192 | 7,041,478,520 | 🡖 69,525,672 | -0.98% |
| **envs-elements** | 346,236,940 | 339,588,487 | 🡖 6,648,453 | -1.92% |
| **envs-number** | 271,319,292 | 270,298,164 | 🡖 1,021,128 | -0.38% |
| **gc-heapSize** | 8,995,291,136 | 10,110,009,344 | 🡕 1,114,718,208 | 12.39% |
| **gc-totalBytes** | 37,172,737,408 | 36,878,391,888 | 🡖 294,345,520 | -0.79% |
| **list-bytes** | 1,886,162,656 | 1,886,163,472 | 🡕 816 | 0.00% |
| **list-concats** | 6,898,114 | 6,898,114 | 0 | |
| **list-elements** | 235,770,332 | 235,770,434 | 🡕 102 | 0.00% |
| **nrAvoided** | 328,829,821 | 326,618,157 | 🡖 2,211,664 | -0.67% |
| **nrFunctionCalls** | 240,850,845 | 239,998,495 | 🡖 852,350 | -0.35% |
| **nrLookups** | 144,849,632 | 142,126,339 | 🡖 2,723,293 | -1.88% |
| **nrOpUpdateValuesCopied** | 251,032,504 | 251,032,504 | 0 | |
| **nrOpUpdates** | 17,903,110 | 17,903,110 | 0 | |
| **nrPrimOpCalls** | 140,674,913 | 139,485,975 | 🡖 1,188,938 | -0.85% |
| **nrThunks** | 294,643,131 | 288,678,022 | 🡖 5,965,109 | -2.02% |
| **sets-bytes** | 9,464,322,192 | 9,456,172,048 | 🡖 8,150,144 | -0.09% |
| **sets-elements** | 377,474,889 | 377,134,877 | 🡖 340,012 | -0.09% |
| **sets-number** | 50,615,607 | 50,616,875 | 🡕 1,268 | 0.00% |
| **sizes-Attr** | 24 | 24 | 0 | |
| **sizes-Bindings** | 8 | 8 | 0 | |
| **sizes-Env** | 16 | 16 | 0 | |
| **sizes-Value** | 24 | 24 | 0 | |
| **symbols-bytes** | 3,147,102 | 3,147,064 | 🡖 38 | -0.00% |
| **symbols-number** | 82,819 | 82,819 | 0 | |
| **values-bytes** | 11,147,448,768 | 10,996,111,512 | 🡖 151,337,256 | -1.36% |
| **values-number** | 464,477,032 | 458,171,313 | 🡖 6,305,719 | -1.36% |
2019-04-11 17:35:35 +01:00
|
|
|
|
assertValidity = { meta, attrs }: let
|
2018-01-31 03:10:34 +00:00
|
|
|
|
validity = checkValidity attrs;
|
|
|
|
|
in validity // {
|
|
|
|
|
# Throw an error if trying to evaluate an non-valid derivation
|
|
|
|
|
handled = if !validity.valid
|
2020-04-19 11:04:58 +01:00
|
|
|
|
then handleEvalIssue { inherit meta attrs; } { inherit (validity) reason errormsg; }
|
2018-01-31 03:10:34 +00:00
|
|
|
|
else true;
|
|
|
|
|
};
|
2017-12-25 17:38:45 +00:00
|
|
|
|
|
2018-01-31 03:10:34 +00:00
|
|
|
|
in assertValidity
|