Merge pull request #119942 from hercules-ci/mkDerivation-overridable-recursive-attributes

doc/stdenv/meta.chapter.md

@@ -175,6 +175,40 @@ The NixOS tests are available as `nixosTests` in parameters of derivations. For
 
 NixOS tests run in a VM, so they are slower than regular package tests. For more information see [NixOS module tests](https://nixos.org/manual/nixos/stable/#sec-nixos-tests).
 
+Alternatively, you can specify other derivations as tests. You can make use of
+the optional parameter to inject the correct package without
+relying on non-local definitions, even in the presence of `overrideAttrs`.
+Here that's `finalAttrs.finalPackage`, but you could choose a different name if
+`finalAttrs` already exists in your scope.
+
+`(mypkg.overrideAttrs f).passthru.tests` will be as expected, as long as the
+definition of `tests` does not rely on the original `mypkg` or overrides it in
+all places.
+
+```nix
+# my-package/default.nix
+{ stdenv, callPackage }:
+stdenv.mkDerivation (finalAttrs: {
+  # ...
+  passthru.tests.example = callPackage ./example.nix { my-package = finalAttrs.finalPackage; };
+})
+```
+
+```nix
+# my-package/example.nix
+{ runCommand, lib, my-package, ... }:
+runCommand "my-package-test" {
+  nativeBuildInputs = [ my-package ];
+  src = lib.sources.sourcesByRegex ./. [ ".*.in" ".*.expected" ];
+} ''
+  my-package --help
+  my-package <example.in >example.actual
+  diff -U3 --color=auto example.expected example.actual
+  mkdir $out
+''
+```
+
+
 ### `timeout` {#var-meta-timeout}
 
 A timeout (in seconds) for building the derivation. If the derivation takes longer than this time to build, it can fail due to breaking the timeout. However, all computers do not have the same computing power, hence some builders may decide to apply a multiplicative factor to this value. When filling this value in, try to keep it approximately consistent with other values already present in `nixpkgs`.

doc/stdenv/stdenv.chapter.md

@@ -317,6 +317,60 @@ The script will be usually run from the root of the Nixpkgs repository but you s
 
 For information about how to run the updates, execute `nix-shell maintainers/scripts/update.nix`.
 
+### Recursive attributes in `mkDerivation`
+
+If you pass a function to `mkDerivation`, it will receive as its argument the final arguments, including the overrides when reinvoked via `overrideAttrs`. For example:
+
+```nix
+mkDerivation (finalAttrs: {
+  pname = "hello";
+  withFeature = true;
+  configureFlags =
+    lib.optionals finalAttrs.withFeature ["--with-feature"];
+})
+```
+
+Note that this does not use the `rec` keyword to reuse `withFeature` in `configureFlags`.
+The `rec` keyword works at the syntax level and is unaware of overriding.
+
+Instead, the definition references `finalAttrs`, allowing users to change `withFeature`
+consistently with `overrideAttrs`.
+
+`finalAttrs` also contains the attribute `finalPackage`, which includes the output paths, etc.
+
+Let's look at a more elaborate example to understand the differences between
+various bindings:
+
+```nix
+# `pkg` is the _original_ definition (for illustration purposes)
+let pkg =
+  mkDerivation (finalAttrs: {
+    # ...
+
+    # An example attribute
+    packages = [];
+
+    # `passthru.tests` is a commonly defined attribute.
+    passthru.tests.simple = f finalAttrs.finalPackage;
+
+    # An example of an attribute containing a function
+    passthru.appendPackages = packages':
+      finalAttrs.finalPackage.overrideAttrs (newSelf: super: {
+        packages = super.packages ++ packages';
+      });
+
+    # For illustration purposes; referenced as
+    # `(pkg.overrideAttrs(x)).finalAttrs` etc in the text below.
+    passthru.finalAttrs = finalAttrs;
+    passthru.original = pkg;
+  });
+in pkg
+```
+
+Unlike the `pkg` binding in the above example, the `finalAttrs` parameter always references the final attributes. For instance `(pkg.overrideAttrs(x)).finalAttrs.finalPackage` is identical to `pkg.overrideAttrs(x)`, whereas `(pkg.overrideAttrs(x)).original` is the same as the original `pkg`.
+
+See also the section about [`passthru.tests`](#var-meta-tests).
+
 ## Phases {#sec-stdenv-phases}
 
 `stdenv.mkDerivation` sets the Nix [derivation](https://nixos.org/manual/nix/stable/expressions/derivations.html#derivations)'s builder to a script that loads the stdenv `setup.sh` bash library and calls `genericBuild`. Most packaging functions rely on this default builder.

doc/using/overrides.chapter.md

@@ -39,14 +39,18 @@ The function `overrideAttrs` allows overriding the attribute set passed to a `st
 Example usage:
 
 ```nix
-helloWithDebug = pkgs.hello.overrideAttrs (oldAttrs: rec {
+helloWithDebug = pkgs.hello.overrideAttrs (finalAttrs: previousAttrs: {
   separateDebugInfo = true;
 });
 ```
 
 In the above example, the `separateDebugInfo` attribute is overridden to be true, thus building debug info for `helloWithDebug`, while all other attributes will be retained from the original `hello` package.
 
-The argument `oldAttrs` is conventionally used to refer to the attr set originally passed to `stdenv.mkDerivation`.
+The argument `previousAttrs` is conventionally used to refer to the attr set originally passed to `stdenv.mkDerivation`.
+
+The argument `finalAttrs` refers to the final attributes passed to `mkDerivation`, plus the `finalPackage` attribute which is equal to the result of `mkDerivation` or subsequent `overrideAttrs` calls.
+
+If only a one-argument function is written, the argument has the meaning of `previousAttrs`.
 
 ::: {.note}
 Note that `separateDebugInfo` is processed only by the `stdenv.mkDerivation` function, not the generated, raw Nix derivation. Thus, using `overrideDerivation` will not work in this case, as it overrides only the attributes of the final derivation. It is for this reason that `overrideAttrs` should be preferred in (almost) all cases to `overrideDerivation`, i.e. to allow using `stdenv.mkDerivation` to process input arguments, as well as the fact that it is easier to use (you can use the same attribute names you see in your Nix code, instead of the ones generated (e.g. `buildInputs` vs `nativeBuildInputs`), and it involves less typing).

nixos/doc/manual/from_md/release-notes/rl-2205.section.xml

@@ -43,6 +43,33 @@
           Shell.
         </para>
       </listitem>
+      <listitem>
+        <para>
+          <literal>stdenv.mkDerivation</literal> now supports a
+          self-referencing <literal>finalAttrs:</literal> parameter
+          containing the final <literal>mkDerivation</literal> arguments
+          including overrides. <literal>drv.overrideAttrs</literal> now
+          supports two parameters
+          <literal>finalAttrs: previousAttrs:</literal>. This allows
+          packaging configuration to be overridden in a consistent
+          manner by providing an alternative to
+          <literal>rec {}</literal> syntax.
+        </para>
+        <para>
+          Additionally, <literal>passthru</literal> can now reference
+          <literal>finalAttrs.finalPackage</literal> containing the
+          final package, including attributes such as the output paths
+          and <literal>overrideAttrs</literal>.
+        </para>
+        <para>
+          New language integrations can be simplified by overriding a
+          <quote>prototype</quote> package containing the
+          language-specific logic. This removes the need for a extra
+          layer of overriding for the <quote>generic builder</quote>
+          arguments, thus removing a usability problem and source of
+          error.
+        </para>
+      </listitem>
       <listitem>
         <para>
           PHP 8.1 is now available

nixos/doc/manual/release-notes/rl-2205.section.md

@@ -17,6 +17,21 @@ In addition to numerous new and upgraded packages, this release has the followin
 
 - GNOME has been upgraded to 42. Please take a look at their [Release Notes](https://release.gnome.org/42/) for details. Notably, it replaces gedit with GNOME Text Editor, GNOME Terminal with GNOME Console (formerly King’s Cross), and GNOME Screenshot with a tool built into the Shell.
 
+- `stdenv.mkDerivation` now supports a self-referencing `finalAttrs:` parameter
+  containing the final `mkDerivation` arguments including overrides.
+  `drv.overrideAttrs` now supports two parameters `finalAttrs: previousAttrs:`.
+  This allows packaging configuration to be overridden in a consistent manner by
+  providing an alternative to `rec {}` syntax.
+
+  Additionally, `passthru` can now reference `finalAttrs.finalPackage` containing
+  the final package, including attributes such as the output paths and
+  `overrideAttrs`.
+
+  New language integrations can be simplified by overriding a "prototype"
+  package containing the language-specific logic. This removes the need for a
+  extra layer of overriding for the "generic builder" arguments, thus removing a
+  usability problem and source of error.
+
 - PHP 8.1 is now available
 
 - Mattermost has been updated to extended support release 6.3, as the previously packaged extended support release 5.37 is [reaching its end of life](https://docs.mattermost.com/upgrade/extended-support-release.html).

pkgs/applications/misc/hello/default.nix

@@ -1,4 +1,5 @@
-{ lib
+{ callPackage
+, lib
 , stdenv
 , fetchurl
 , nixos
@@ -6,12 +7,12 @@
 , hello
 }:
 
-stdenv.mkDerivation rec {
+stdenv.mkDerivation (finalAttrs: {
   pname = "hello";
   version = "2.12";
 
   src = fetchurl {
-    url = "mirror://gnu/hello/${pname}-${version}.tar.gz";
+    url = "mirror://gnu/hello/hello-${finalAttrs.version}.tar.gz";
     sha256 = "1ayhp9v4m4rdhjmnl2bq3cibrbqqkgjbl3s7yk2nhlh8vj3ay16g";
   };
 
@@ -27,6 +28,8 @@ stdenv.mkDerivation rec {
         (nixos { environment.noXlibs = true; }).pkgs.hello;
   };
 
+  passthru.tests.run = callPackage ./test.nix { hello = finalAttrs.finalPackage; };
+
   meta = with lib; {
     description = "A program that produces a familiar, friendly greeting";
     longDescription = ''
@@ -34,9 +37,9 @@ stdenv.mkDerivation rec {
       It is fully customizable.
     '';
     homepage = "https://www.gnu.org/software/hello/manual/";
-    changelog = "https://git.savannah.gnu.org/cgit/hello.git/plain/NEWS?h=v${version}";
+    changelog = "https://git.savannah.gnu.org/cgit/hello.git/plain/NEWS?h=v${finalAttrs.version}";
     license = licenses.gpl3Plus;
     maintainers = [ maintainers.eelco ];
     platforms = platforms.all;
   };
-}
+})

pkgs/applications/misc/hello/test.nix

@@ -0,0 +1,8 @@
+{ runCommand, hello }:
+
+runCommand "hello-test-run" {
+  nativeBuildInputs = [ hello ];
+} ''
+  diff -U3 --color=auto <(hello) <(echo 'Hello, world!')
+  touch $out
+''

pkgs/stdenv/generic/make-derivation.nix

@@ -9,8 +9,72 @@ let
     # to build it. This is a bit confusing for cross compilation.
     inherit (stdenv) hostPlatform;
   };
+
+  makeOverlayable = mkDerivationSimple:
+    fnOrAttrs:
+      if builtins.isFunction fnOrAttrs
+      then makeDerivationExtensible mkDerivationSimple fnOrAttrs
+      else makeDerivationExtensibleConst mkDerivationSimple fnOrAttrs;
+
+  # Based off lib.makeExtensible, with modifications:
+  makeDerivationExtensible = mkDerivationSimple: rattrs:
+    let
+      # NOTE: The following is a hint that will be printed by the Nix cli when
+      # encountering an infinite recursion. It must not be formatted into
+      # separate lines, because Nix would only show the last line of the comment.
+
+      # An infinite recursion here can be caused by having the attribute names of expression `e` in `.overrideAttrs(finalAttrs: previousAttrs: e)` depend on `finalAttrs`. Only the attribute values of `e` can depend on `finalAttrs`.
+      args = rattrs (args // { inherit finalPackage; });
+      #              ^^^^
+
+      finalPackage =
+        mkDerivationSimple
+          (f0:
+            let
+              f = self: super:
+                # Convert f0 to an overlay. Legacy is:
+                #   overrideAttrs (super: {})
+                # We want to introduce self. We follow the convention of overlays:
+                #   overrideAttrs (self: super: {})
+                # Which means the first parameter can be either self or super.
+                # This is surprising, but far better than the confusion that would
+                # arise from flipping an overlay's parameters in some cases.
+                let x = f0 super;
+                in
+                  if builtins.isFunction x
+                  then
+                    # Can't reuse `x`, because `self` comes first.
+                    # Looks inefficient, but `f0 super` was a cheap thunk.
+                    f0 self super
+                  else x;
+            in
+              makeDerivationExtensible mkDerivationSimple
+                (self: let super = rattrs self; in super // f self super))
+          args;
+    in finalPackage;
+
+  # makeDerivationExtensibleConst == makeDerivationExtensible (_: attrs),
+  # but pre-evaluated for a slight improvement in performance.
+  makeDerivationExtensibleConst = mkDerivationSimple: attrs:
+    mkDerivationSimple
+      (f0:
+        let
+          f = self: super:
+            let x = f0 super;
+            in
+              if builtins.isFunction x
+              then
+                f0 self super
+              else x;
+        in
+          makeDerivationExtensible mkDerivationSimple (self: attrs // f self attrs))
+      attrs;
+
 in
 
+makeOverlayable (overrideAttrs:
+
+
 # `mkDerivation` wraps the builtin `derivation` function to
 # produce derivations that use this stdenv and its shell.
 #
@@ -70,6 +134,7 @@ in
 
 , # TODO(@Ericson2314): Make always true and remove
   strictDeps ? if config.strictDepsByDefault then true else stdenv.hostPlatform != stdenv.buildPlatform
+
 , meta ? {}
 , passthru ? {}
 , pos ? # position used in error messages and for meta.position
@@ -381,8 +446,6 @@ in
 lib.extendDerivation
   validity.handled
   ({
-     overrideAttrs = f: stdenv.mkDerivation (attrs // (f attrs));
-
      # A derivation that always builds successfully and whose runtime
      # dependencies are the original derivations build time dependencies
      # This allows easy building and distributing of all derivations
@@ -408,10 +471,12 @@ lib.extendDerivation
        args = [ "-c" "export > $out" ];
      });
 
-     inherit meta passthru;
+     inherit meta passthru overrideAttrs;
    } //
    # Pass through extra attributes that are not inputs, but
    # should be made available to Nix expressions using the
    # derivation (e.g., in assertions).
    passthru)
   (derivation derivationArg)
+
+)