diff --git a/doc/stdenv.xml b/doc/stdenv.xml index e38ff403e748..10d58f38399a 100644 --- a/doc/stdenv.xml +++ b/doc/stdenv.xml @@ -228,18 +228,19 @@ genericBuild - The extension of PATH with dependencies, alluded to above, - proceeds according to the relative platforms alone. The process is carried - out only for dependencies whose host platform matches the new derivation's - build platform–i.e. which run on the platform where the new derivation - will be built. + The extension of PATH with dependencies, alluded to + above, proceeds according to the relative platforms alone. The + process is carried out only for dependencies whose host platform + matches the new derivation's build platform i.e. dependencies which + run on the platform where the new derivation will be built. - Currently, that means for native builds all dependencies are put on the - PATH. But in the future that may not be the case for sake - of matching cross: the platforms would be assumed to be unique for native - and cross builds alike, so only the depsBuild* and - nativeBuildDependencies dependencies would affect the + Currently, this means for native builds all dependencies are put + on the PATH. But in the future that may not be the + case for sake of matching cross: the platforms would be assumed + to be unique for native and cross builds alike, so only the + depsBuild* and + nativeBuildInputs would be added to the PATH. @@ -251,28 +252,27 @@ genericBuild The dependency is propagated when it forces some of its other-transitive (non-immediate) downstream dependencies to also take it on as an immediate - dependency. Nix itself already takes a package's transitive dependencies - into account, but this propagation ensures nixpkgs-specific infrastructure - like setup hooks (mentioned above) also are run as if the propagated - dependency. + dependency. Nix itself already takes a package's transitive dependencies into + account, but this propagation ensures nixpkgs-specific infrastructure like + setup hooks (mentioned above) also are run as if the propagated dependency. - It is important to note dependencies are not necessary propagated as the + It is important to note dependencies are not necessarily propagated as the same sort of dependency that they were before, but rather as the corresponding sort so that the platform rules still line up. The exact rules - for dependency propagation can be given by assigning each sort of dependency - two integers based one how it's host and target platforms are offset from - the depending derivation's platforms. Those offsets are given - below in the descriptions of each dependency list attribute. - Algorithmically, we traverse propagated inputs, accumulating every - propagated dep's propagated deps and adjusting them to account for the - "shift in perspective" described by the current dep's platform offsets. This - results in sort a transitive closure of the dependency relation, with the - offsets being approximately summed when two dependency links are combined. - We also prune transitive deps whose combined offsets go out-of-bounds, which - can be viewed as a filter over that transitive closure removing dependencies - that are blatantly absurd. + for dependency propagation can be given by assigning to each dependency two + integers based one how its host and target platforms are offset from the + depending derivation's platforms. Those offsets are given below in the + descriptions of each dependency list attribute. Algorithmically, we traverse + propagated inputs, accumulating every propagated dependency's propagated + dependenciess and adjusting them to account for the "shift in perspective" + described by the current dependency's platform offsets. This results in sort + a transitive closure of the dependency relation, with the offsets being + approximately summed when two dependency links are combined. We also prune + transitive dependencies whose combined offsets go out-of-bounds, which can be + viewed as a filter over that transitive closure removing dependencies that + are blatantly absurd. @@ -287,8 +287,8 @@ genericBuild propagation logic. - They're confusing in very different ways so...hopefully if something doesn't - make sense in one presentation, it does in the other! + They're confusing in very different ways so... hopefully if something doesn't + make sense in one presentation, it will in the other! let mapOffset(h, t, i) = i + (if i <= 0 then h else t - 1) @@ -307,13 +307,13 @@ dep(h0, _, A, B) propagated-dep(h1, t1, B, C) h0 + h1 in {-1, 0, 1} h0 + t1 in {-1, 0, -1} --------------------------------------- Take immediate deps' propagated deps +----------------------------- Take immediate dependencies' propagated dependencies propagated-dep(mapOffset(h0, t0, h1), mapOffset(h0, t0, t1), A, C) propagated-dep(h, t, A, B) --------------------------------------- Propagated deps count as deps +----------------------------- Propagated dependencies count as dependencies dep(h, t, A, B) Some explanation of this monstrosity is in order. In the common case, the target offset of a dependency is the successor to the target offset: @@ -324,31 +324,31 @@ let f(h, h + 1, i) = i + (if i <= 0 then h else (h + 1) - 1) let f(h, h + 1, i) = i + (if i <= 0 then h else h) let f(h, h + 1, i) = i + h - This is where the "sum-like" comes from above: We can just sum all the host - offset to get the host offset of the transitive dependency. The target - offset is the transitive dep is simply the host offset + 1, just as it was - with the dependencies composed to make this transitive one; it can be + This is where "sum-like" comes in from above: We can just sum all of the host + offsets to get the host offset of the transitive dependency. The target + offset is the transitive dependency is simply the host offset + 1, just as it + was with the dependencies composed to make this transitive one; it can be ignored as it doesn't add any new information. - Because of the bounds checks, the uncommon cases are h = - t and h + 2 = t. In the former case, the - motivation for mapOffset is that since its host and - target platforms are the same, no transitive dep of it should be able to - "discover" an offset greater than its reduced target offsets. + Because of the bounds checks, the uncommon cases are h = t + and h + 2 = t. In the former case, the motivation for + mapOffset is that since its host and target platforms + are the same, no transitive dependency of it should be able to "discover" an + offset greater than its reduced target offsets. mapOffset effectively "squashes" all its transitive dependencies' offsets so that none will ever be greater than the target offset of the original h = t package. In the other case, - h + 1 is skipped over between the host and target - offsets. Instead of squashing the offsets, we need to "rip" them apart so no + h + 1 is skipped over between the host and target offsets. + Instead of squashing the offsets, we need to "rip" them apart so no transitive dependencies' offset is that one. - Overall, the unifying theme here is that propagation shouldn't be - introducing transitive dependencies involving platforms the needing package - is unaware of. The offset bounds checking and definition of + Overall, the unifying theme here is that propagation shouldn't be introducing + transitive dependencies involving platforms the depending package is unaware + of. The offset bounds checking and definition of mapOffset together ensure that this is the case. Discovering a new offset is discovering a new platform, and since those platforms weren't in the derivation "spec" of the needing package, they @@ -369,20 +369,20 @@ let f(h, h + 1, i) = i + h A list of dependencies whose host and target platforms are the new derivation's build platform. This means a -1 host and -1 target offset from the new derivation's platforms. - They are programs/libraries used at build time that furthermore produce - programs/libraries also used at build time. If the dependency doesn't - care about the target platform (i.e. isn't a compiler or similar tool), - put it in nativeBuildInputs instead. The most common - use for this buildPackages.stdenv.cc, the default C - compiler for this role. That example crops up more than one might think - in old commonly used C libraries. + These are programs and libraries used at build time that produce programs + and libraries also used at build time. If the dependency doesn't care + about the target platform (i.e. isn't a compiler or similar tool), put it + in nativeBuildInputs instead. The most common use of + this buildPackages.stdenv.cc, the default C compiler + for this role. That example crops up more than one might think in old + commonly used C libraries. - Since these packages are able to be run at build time, that are always + Since these packages are able to be run at build-time, they are always added to the PATH, as described above. But since these packages are only guaranteed to be able to run then, they shouldn't - persist as run-time dependencies. This isn't currently enforced, but - could be in the future. + persist as run-time dependencies. This isn't currently enforced, but could + be in the future. @@ -395,21 +395,20 @@ let f(h, h + 1, i) = i + h A list of dependencies whose host platform is the new derivation's build platform, and target platform is the new derivation's host platform. This means a -1 host offset and 0 target - offset from the new derivation's platforms. They are programs/libraries - used at build time that, if they are a compiler or similar tool, produce - code to run at run time—i.e. tools used to build the new derivation. If - the dependency doesn't care about the target platform (i.e. isn't a - compiler or similar tool), put it here, rather than in + offset from the new derivation's platforms. These are programs and + libraries used at build-time that, if they are a compiler or similar tool, + produce code to run at run-time—i.e. tools used to build the new + derivation. If the dependency doesn't care about the target platform (i.e. + isn't a compiler or similar tool), put it here, rather than in depsBuildBuild or depsBuildTarget. - This would be called depsBuildHost but for historical - continuity. + This could be called depsBuildHost but + nativeBuildInputs is used for historical continuity. - Since these packages are able to be run at build time, that are added to - the PATH, as described above. But since these packages - only are guaranteed to be able to run then, they shouldn't persist as - run-time dependencies. This isn't currently enforced, but could be in the - future. + Since these packages are able to be run at build-time, they are added to + the PATH, as described above. But since these packages are + only guaranteed to be able to run then, they shouldn't persist as run-time + dependencies. This isn't currently enforced, but could be in the future. @@ -422,34 +421,33 @@ let f(h, h + 1, i) = i + h A list of dependencies whose host platform is the new derivation's build platform, and target platform is the new derivation's target platform. This means a -1 host offset and 1 - target offset from the new derivation's platforms. They are programs used - at build time that produce code to run at run with code produced by the - depending package. Most commonly, these would tools used to build the - runtime or standard library the currently-being-built compiler will - inject into any code it compiles. In many cases, the currently-being - built compiler is itself employed for that task, but when that compiler - won't run (i.e. its build and host platform differ) this is not possible. - Other times, the compiler relies on some other tool, like binutils, that - is always built separately so the dependency is unconditional. + target offset from the new derivation's platforms. These are programs used + at build time that produce code to run with code produced by the depending + package. Most commonly, these are tools used to build the runtime or + standard library taht the currently-being-built compiler will inject into + any code it compiles. In many cases, the currently-being-built-compiler is + itself employed for that task, but when that compiler won't run (i.e. its + build and host platform differ) this is not possible. Other times, the + compiler relies on some other tool, like binutils, that is always built + separately so that the dependency is unconditional. - This is a somewhat confusing dependency to wrap ones head around, and for - good reason. As the only one where the platform offsets are not adjacent - integers, it requires thinking of a bootstrapping stage - two away from the current one. It and it's use-case - go hand in hand and are both considered poor form: try not to need this - sort dependency, and try not avoid building standard libraries / runtimes + This is a somewhat confusing concept to wrap one’s head around, and for + good reason. As the only dependency type where the platform offsets are + not adjacent integers, it requires thinking of a bootstrapping stage + two away from the current one. It and its use-case go + hand in hand and are both considered poor form: try to not need this sort + of dependency, and try to avoid building standard libraries and runtimes in the same derivation as the compiler produces code using them. Instead strive to build those like a normal library, using the newly-built compiler just as a normal library would. In short, do not use this attribute unless you are packaging a compiler and are sure it is needed. - Since these packages are able to be run at build time, that are added to - the PATH, as described above. But since these packages - only are guaranteed to be able to run then, they shouldn't persist as - run-time dependencies. This isn't currently enforced, but could be in the - future. + Since these packages are able to run at build time, they are added to the + PATH, as described above. But since these packages are only + guaranteed to be able to run then, they shouldn't persist as run-time + dependencies. This isn't currently enforced, but could be in the future. @@ -460,15 +458,15 @@ let f(h, h + 1, i) = i + h A list of dependencies whose host and target platforms match the new - derivation's host platform. This means a both 0 host - offset and 0 target offset from the new derivation's - host platform. These are packages used at run-time to generate code also - used at run-time. In practice, that would usually be tools used by - compilers for metaprogramming/macro systems, or libraries used by the - macros/metaprogramming code itself. It's always preferable to use a - depsBuildBuild dependency in the derivation being - built than a depsHostHost on the tool doing the - building for this purpose. + derivation's host platform. This means a 0 host offset + and 0 target offset from the new derivation's host + platform. These are packages used at run-time to generate code also used + at run-time. In practice, this would usually be tools used by compilers + for macros or a metaprogramming system, or libraries used by the macros or + metaprogramming code itself. It's always preferable to use a + depsBuildBuild dependency in the derivation being built + over a depsHostHost on the tool doing the building for + this purpose. @@ -479,20 +477,20 @@ let f(h, h + 1, i) = i + h A list of dependencies whose host platform and target platform match the - new derivation's. This means a 0 host offset and + new derivation's. This means a 0 host offset and a 1 target offset from the new derivation's host platform. This would be called depsHostTarget but for historical continuity. If the dependency doesn't care about the target - platform (i.e. isn't a compiler or similar tool), put it here, rather - than in depsBuildBuild. + platform (i.e. isn't a compiler or similar tool), put it here, rather than + in depsBuildBuild. - These often are programs/libraries used by the new derivation at + These are often programs and libraries used by the new derivation at run-time, but that isn't always the case. For - example, the machine code in a statically linked library is only used at - run time, but the derivation containing the library is only needed at - build time. Even in the dynamic case, the library may also be needed at - build time to appease the linker. + example, the machine code in a statically-linked library is only used at + run-time, but the derivation containing the library is only needed at + build-time. Even in the dynamic case, the library may also be needed at + build-time to appease the linker. @@ -604,10 +602,10 @@ let f(h, h + 1, i) = i + h A natural number indicating how much information to log. If set to 1 or - higher, stdenv will print moderate debug information - during the build. In particular, the gcc and - ld wrapper scripts will print out the complete command - line passed to the wrapped tools. If set to 6 or higher, the + higher, stdenv will print moderate debugging + information during the build. In particular, the gcc + and ld wrapper scripts will print out the complete + command line passed to the wrapped tools. If set to 6 or higher, the stdenv setup script will be run with set -x tracing. If set to 7 or higher, the gcc and ld wrapper scripts will also be run with @@ -666,11 +664,10 @@ passthru = { hello.baz.value1. We don't specify any usage or schema of passthru - it is meant for values that would be useful outside the derivation in other parts of a Nix expression (e.g. in - other derivations). An example would be to convey some specific - dependency of your derivation which contains a program with plugins - support. Later, others who make derivations with plugins can use - passed-through dependency to ensure that their plugin would be - binary-compatible with built program. + other derivations). An example would be to convey some specific dependency + of your derivation which contains a program with plugins support. Later, + others who make derivations with plugins can use passed-through dependency + to ensure that their plugin would be binary-compatible with built program. @@ -1144,12 +1141,11 @@ passthru = { By default, when cross compiling, the configure script has and passed. Packages can instead pass [ "build" "host" "target" ] - or a subset to control exactly which platform flags are passed. - Compilers and other tools should use this to also pass the target - platform, for example. + or a subset to control exactly which platform flags are passed. Compilers + and other tools can use this to also pass the target platform. - Eventually these will be passed when in native builds too, to improve + Eventually these will be passed building natively as well, to improve determinism: build-time guessing, as is done today, is a risk of impurity. @@ -1647,13 +1643,11 @@ installTargets = "install-bin install-doc"; - A package can export a setup - hook by setting this variable. The setup hook, if defined, is - copied to $out/nix-support/setup-hook. Environment - variables are then substituted in it using - substituteAll. + A package can export a setup hook + by setting this variable. The setup hook, if defined, is copied to + $out/nix-support/setup-hook. Environment variables + are then substituted in it using substituteAll. @@ -2086,12 +2080,12 @@ someVar=$(stripHash $name) Package setup hooks - Nix itself considers a build-time dependency merely something that should + Nix itself considers a build-time dependency as merely something that should previously be built and accessible at build time—packages themselves are on their own to perform any additional setup. In most cases, that is fine, and the downstream derivation can deal with its own dependencies. But for a few common tasks, that would result in almost every package doing the same - sort of setup work---depending not on the package itself, but entirely on + sort of setup work—depending not on the package itself, but entirely on which dependencies were used. @@ -2106,20 +2100,19 @@ someVar=$(stripHash $name) - The Setup hook mechanism is a bit of a sledgehammer though: a powerful + The setup hook mechanism is a bit of a sledgehammer though: a powerful feature with a broad and indiscriminate area of effect. The combination of its power and implicit use may be expedient, but isn't without costs. Nix - itself is unchanged, but the spirit of adding dependencies being effect-free + itself is unchanged, but the spirit of added dependencies being effect-free is violated even if the letter isn't. For example, if a derivation path is mentioned more than once, Nix itself doesn't care and simply makes sure the dependency derivation is already built just the same—depending is just needing something to exist, and needing is idempotent. However, a dependency specified twice will have its setup hook run twice, and that could easily - change the build environment (though a well-written setup hook will - therefore strive to be idempotent so this is in fact not observable). More - broadly, setup hooks are anti-modular in that multiple dependencies, whether - the same or different, should not interfere and yet their setup hooks may - well do so. + change the build environment (though a well-written setup hook will therefore + strive to be idempotent so this is in fact not observable). More broadly, + setup hooks are anti-modular in that multiple dependencies, whether the same + or different, should not interfere and yet their setup hooks may well do so. @@ -2138,15 +2131,14 @@ someVar=$(stripHash $name) Packages adding a hook should not hard code a specific hook, but rather choose a variable relative to how they are included. - Returning to the C compiler wrapper example, if it itself is an + Returning to the C compiler wrapper example, if the wrapper itself is an n dependency, then it only wants to accumulate flags from n + 1 dependencies, as only those ones match the - compiler's target platform. The hostOffset variable is - defined with the current dependency's host offset - targetOffset with its target offset, before its setup hook is - sourced. Additionally, since most environment hooks don't care about the - target platform, That means the setup hook can append to the right bash array - by doing something like + compiler's target platform. The hostOffset variable is defined + with the current dependency's host offset targetOffset with + its target offset, before its setup hook is sourced. Additionally, since most + environment hooks don't care about the target platform, that means the setup + hook can append to the right bash array by doing something like addEnvHooks "$hostOffset" myBashFunction @@ -2171,19 +2163,19 @@ addEnvHooks "$hostOffset" myBashFunction - Bintools Wrapper wraps the binary utilities for a bunch of miscellaneous - purposes. These are GNU Binutils when targetting Linux, and a mix of - cctools and GNU binutils for Darwin. [The "Bintools" name is supposed to - be a compromise between "Binutils" and "cctools" not denoting any - specific implementation.] Specifically, the underlying bintools package, - and a C standard library (glibc or Darwin's libSystem, just for the - dynamic loader) are all fed in, and dependency finding, hardening (see - below), and purity checks for each are handled by Bintools Wrapper. - Packages typically depend on CC Wrapper, which in turn (at run time) - depends on Bintools Wrapper. + The Bintools Wrapper wraps the binary utilities for a bunch of + miscellaneous purposes. These are GNU Binutils when targetting Linux, and + a mix of cctools and GNU binutils for Darwin. [The "Bintools" name is + supposed to be a compromise between "Binutils" and "cctools" not denoting + any specific implementation.] Specifically, the underlying bintools + package, and a C standard library (glibc or Darwin's libSystem, just for + the dynamic loader) are all fed in, and dependency finding, hardening + (see below), and purity checks for each are handled by the Bintools + Wrapper. Packages typically depend on CC Wrapper, which in turn (at run + time) depends on the Bintools Wrapper. - Bintools Wrapper was only just recently split off from CC Wrapper, so + The Bintools Wrapper was only just recently split off from CC Wrapper, so the division of labor is still being worked out. For example, it shouldn't care about about the C standard library, but just take a derivation with the dynamic loader (which happens to be the glibc on @@ -2191,24 +2183,24 @@ addEnvHooks "$hostOffset" myBashFunction to need to share, and probably the most important to understand. It is currently accomplished by collecting directories of host-platform dependencies (i.e. buildInputs and - nativeBuildInputs) in environment variables. Bintools - Wrapper's setup hook causes any lib and + nativeBuildInputs) in environment variables. The + Bintools Wrapper's setup hook causes any lib and lib64 subdirectories to be added to - NIX_LDFLAGS. Since CC Wrapper and Bintools Wrapper use - the same strategy, most of the Bintools Wrapper code is sparsely - commented and refers to CC Wrapper. But CC Wrapper's code, by contrast, - has quite lengthy comments. Bintools Wrapper merely cites those, rather - than repeating them, to avoid falling out of sync. + NIX_LDFLAGS. Since the CC Wrapper and the Bintools Wrapper + use the same strategy, most of the Bintools Wrapper code is sparsely + commented and refers to the CC Wrapper. But the CC Wrapper's code, by + contrast, has quite lengthy comments. The Bintools Wrapper merely cites + those, rather than repeating them, to avoid falling out of sync. A final task of the setup hook is defining a number of standard - environment variables to tell build systems which executables full-fill + environment variables to tell build systems which executables fulfill which purpose. They are defined to just be the base name of the tools, - under the assumption that Bintools Wrapper's binaries will be on the + under the assumption that the Bintools Wrapper's binaries will be on the path. Firstly, this helps poorly-written packages, e.g. ones that look for just gcc when CC isn't defined yet - clang is to be used. Secondly, this helps packages - not get confused when cross-compiling, in which case multiple Bintools + clang is to be used. Secondly, this helps packages not + get confused when cross-compiling, in which case multiple Bintools Wrappers may simultaneously be in use. @@ -2220,20 +2212,20 @@ addEnvHooks "$hostOffset" myBashFunction BUILD_- and TARGET_-prefixed versions of - the normal environment variable are defined for the additional Bintools + the normal environment variable are defined for additional Bintools Wrappers, properly disambiguating them. - A problem with this final task is that Bintools Wrapper is honest and + A problem with this final task is that the Bintools Wrapper is honest and defines LD as ld. Most packages, however, firstly use the C compiler for linking, secondly use LD anyways, defining it as the C compiler, and thirdly, - only so define LD when it is undefined as a fallback. - This triple-threat means Bintools Wrapper will break those packages, as - LD is already defined as the actual linker which the package won't - override yet doesn't want to use. The workaround is to define, just for - the problematic package, LD as the C compiler. A good way - to do this would be preConfigure = "LD=$CC". + only so define LD when it is undefined as a fallback. This + triple-threat means Bintools Wrapper will break those packages, as LD is + already defined as the actual linker which the package won't override yet + doesn't want to use. The workaround is to define, just for the + problematic package, LD as the C compiler. A good way to + do this would be preConfigure = "LD=$CC". @@ -2243,30 +2235,31 @@ addEnvHooks "$hostOffset" myBashFunction - CC Wrapper wraps a C toolchain for a bunch of miscellaneous purposes. + The CC Wrapper wraps a C toolchain for a bunch of miscellaneous purposes. Specifically, a C compiler (GCC or Clang), wrapped binary tools, and a C standard library (glibc or Darwin's libSystem, just for the dynamic loader) are all fed in, and dependency finding, hardening (see below), - and purity checks for each are handled by CC Wrapper. Packages typically - depend on CC Wrapper, which in turn (at run time) depends on Bintools - Wrapper. + and purity checks for each are handled by the CC Wrapper. Packages + typically depend on the CC Wrapper, which in turn (at run-time) depends + on the Bintools Wrapper. - Dependency finding is undoubtedly the main task of CC Wrapper. This - works just like Bintools Wrapper, except that any + Dependency finding is undoubtedly the main task of the CC Wrapper. This + works just like the Bintools Wrapper, except that any include subdirectory of any relevant dependency is added to NIX_CFLAGS_COMPILE. The setup hook itself contains some lengthy comments describing the exact convoluted mechanism by which this is accomplished. - CC Wrapper also like Bintools Wrapper defines standard environment - variables with the names of the tools it wraps, for the same reasons - described above. Importantly, while it includes a cc - symlink to the c compiler for portability, the CC will be - defined using the compiler's "real name" (i.e. gcc or - clang). This helps lousy build systems that inspect - on the name of the compiler rather than run it. + Similarly, the CC Wrapper follows the Bintools Wrapper in defining + standard environment variables with the names of the tools it wraps, for + the same reasons described above. Importantly, while it includes a + cc symlink to the c compiler for portability, the + CC will be defined using the compiler's "real name" (i.e. + gcc or clang). This helps lousy + build systems that inspect on the name of the compiler rather than run + it. @@ -2326,9 +2319,11 @@ addEnvHooks "$hostOffset" myBashFunction The autoreconfHook derivation adds - autoreconfPhase, which runs autoreconf, libtoolize - and automake, essentially preparing the configure script in - autotools-based builds. + autoreconfPhase, which runs autoreconf, libtoolize and + automake, essentially preparing the configure script in autotools-based + builds. Most autotools-based packages come with the configure script + pre-generated, but this hook is necessary for a few packages and when you + need to patch the package’s configure scripts. @@ -2372,9 +2367,9 @@ addEnvHooks "$hostOffset" myBashFunction - Exports GDK_PIXBUF_MODULE_FILE environment variable the - the builder. Add librsvg package to buildInputs to - get svg support. + Exports GDK_PIXBUF_MODULE_FILE environment variable to the + builder. Add librsvg package to buildInputs to get svg + support. @@ -2411,7 +2406,7 @@ addEnvHooks "$hostOffset" myBashFunction PaX flags on Linux (where it is available by default; on all other platforms, paxmark is a no-op). For example, to disable secure memory protections on the executable - foo: + foo postFixup = '' paxmark m $out/bin/foo