Nix can perform static scope checking, but whenever code is inside
a `with` expression, the analysis breaks down, because it can't
know statically what's in the attribute set whose attributes were
brought into scope. In those cases, Nix has to assume that
everything works out.
Except it doesnt. Removing `with` from lib/ revealed an undefined
variable in an error message.
If that doesn't convince you that we're better off without `with`,
I can tell you that this PR results in a 3% evaluation performance
improvement because Nix can look up local variables by index.
This adds up with applications like the module system.
Furthermore, removing `with` makes the binding site of each
variable obvious, which helps with comprehension.
Previously if `_file` was specified by a module:
trace: warning: The type `types.string' of option `foo' defined in `/nix/store/yxhm2il5yrb92fldgriw0wyqh2kk9qyc-bug.nix' is deprecated. See https://github.com/NixOS/nixpkgs/pull/66346 for better alternative types.
With this change:
trace: warning: The type `types.string' of option `foo' defined in `/home/infinisil/src/nixpkgs/bug.nix' is deprecated. See https://github.com/NixOS/nixpkgs/pull/66346 for better alternative types.
If multiple definitions are passed, this evaluates them all as if they
were the only one, for a better error message. In particular this won't
show module-internal properties like `_type = "override"` and co.
The vision here is that configuration tools can generate .json or .toml
files, which can be plugged into an existing configuration.
Eg:
{ lib, ... }:
{
imports = [
(lib.modules.importJSON ./hardware-configuration.json)
];
}
Previously the only way to deprecate a type was using
theType = lib.warn "deprecated" (mkOptionType ...)
This caused the warning to be emitted when the type was evaluated, but
the error didn't include which option actually used that type.
With this commit, types can specify a deprecationMessage, which when
non-null, is printed along with the option that uses the type
An easy-to-make mistake when declaring e.g. a submodule is the accidental
confusion of `options` and `config`:
types.submodule {
config = {
foo = mkOption { /* ... */ };
};
}
However the error-message
The option `[definition 1-entry 1].foo' defined in `<expr.nix>' does not exist.
is fairly unhelpful because it seems as the options are declared at the
first sight. In fact, it took a colleague and me a while to track down such
a mistake a few days ago and we both agreed that this should be somehow caught
to save the time we spent debugging the module in question.
At first I decided to catch this error in the `submodules`-type directly
by checking whether `options` is undeclared, however this becomes fairly
complicated as soon as a submodule-declaration e.g. depends on existing
`config`-values which would've lead to some ugly `builtins.tryExec`-heuristic.
This patch now simply checks if the option's prefix has any options
defined if a point in evaluation is reached where it's clear that the
option in question doesn't exist. This means that this patch doesn't
change the logic of the module system, it only provides a more detailed
error in certain cases:
The option `[definition 1-entry 1].foo' defined in `<expr.nix>' does not exist.
However it seems as there are no options defined in [definition 1-entry 1]. Are you sure you've
declared your options properly? This happens if you e.g. declared your options in `types.submodule'
under `config' rather than `options'.
The refactoring in fd75dc8765
introduced a mistake in the error message that doesn't show the full
context anymore. E.g. with this module:
options.foo.bar = lib.mkOption {
type = lib.types.submodule {
baz = 10;
};
default = {};
};
You'd get the error
The option `baz' defined in `/home/infinisil/src/nixpkgs/config.nix' does not exist.
instead of the previous
The option `foo.bar.baz' defined in `/home/infinisil/src/nixpkgs/config.nix' does not exist.
This commit undoes this regression
For programs that have a lot of (Nix-representable) configuration options,
a simple way to represent this in a NixOS module is to declare an
option of a type like `attrsOf str`, representing a key-value mapping
which then gets generated into a config file. However with such a type,
there's no way to add type checking for only some key values.
On the other end of the spectrum, one can declare a single separate
option for every key value that the program supports, ending up with a module
with potentially 100s of options. This has the benefit that every value
gets type checked, catching mistakes at evaluation time already. However
the disadvantage is that the module becomes big, becomes coupled to the
program version and takes a lot of effort to write and maintain.
Previously there was a middle ground between these two
extremes: Declare an option of a type like `attrsOf str`, but declare
additional separate options for the values you wish to have type
checked, and assign their values to the `attrsOf str` option. While this
works decently, it has the problem of duplicated options, since now both
the additional options and the `attrsOf str` option can be used to set a
key value. This leads to confusion about what should happen if both are
set, which defaults should apply, and more.
Now with this change, a middle ground becomes available that solves above
problems: The module system now supports setting a freeform type, which
gets used for all definitions that don't have an associated option. This
means that you can now declare all options you wish to have type
checked, while for the rest a freeform type like `attrsOf str` can be
used.
This fundamentally changes how the module evaluation internally
handles definitions without an associated option.
Previously the values of these definitions were discarded and only
the names were propagated. This was fine because that's all that's
needed for optionally checking whether all definitions have an
associated option with _module.check.
However with the upcoming change of supporting freeform modules,
we *do* need the values of these.
With this change, the module evaluation cleanly separates definitions
that match an option, and ones that do not.
The _module option is added as an internal option set, and it messes up
the results of module evaluations, requiring people to manually filter
_modules out.
If people depend on this, they can still use config._module from inside
the modules, exposing _module as an explicitly declared user option. Or
alternatively with the _module attribute now returned by evalModules.
This helps with troubleshooting exceptions in config values, which were hard
to track down for options with many definitions.
The trace will look like:
error: while evaluating the attribute 'config.foo' at undefined position:
[...]
while evaluating the option `foo':
[...]
while evaluating definitions from `/home/user/mymod.nix':
while evaluating 'dischargeProperties' at /home/user/nixpkgs/lib/modules.nix:464:25, called from /home/user/nixpkgs/lib/modules.nix:392:137:
while evaluating the attribute 'value' at /home/user/nixpkgs/lib/modules.nix:277:44:
Value error!
where the `/home/user/mymod.nix` module is
{ lib, ... }: {
options.foo = lib.mkOption {
type = lib.types.lines;
};
config.foo = builtins.throw "Value error!";
}
Without this change, accessing `mergedValue` from `mergeDefinitions` in
case there are no definitions will throw an error like
error: evaluation aborted with the following error message: 'This case should never happen.'
This change makes it throw the appropriate error
error: The option `foo' is used but not defined.
This is fully backwards compatible.
This fixes imports from the store not being possible, which was caused by
https://github.com/NixOS/nixpkgs/pull/76857
E.g. such a case:
imports = [ "${home-manager}/nixos" ];
With this change, disabledModules applies recursively, meaning if you
have a module "foo.nix" with
imports = [ ./bar.nix ];
then setting
disabledModules = [ "foo.nix" ];
will disable both "foo.nix" and "bar.nix", whereas previously only
"foo.nix" would be disabled.
This change along with https://github.com/NixOS/nixpkgs/pull/61570 allows
modules to be fully disabled even when they have some `mkRenamedOption`
imports.
If I understand correctly, the problem isn't so much that you're assigning to
that top-level attribute, but that the assignment to the attribute (or any
child of the attribute) introduces the 'config' object and prevents 'lifting'
all settings to a generated 'config' object.
We don't want to ignore config that can mess up machines. In general
this should always fail evaluation, as you think you are changing
behaviour and don't, which can easily create run-time errors we can
catch early.
Previously mkRemovedOptionModule would only show the replacement
instructions when the removed option was *defined*. With this change, it
also does so when an option is *used*.
This is essential for options that are only intended to be used such as
`security.acme.directory`, whose replacement instructions would never
trigger without this change because almost everybody only uses the
option and isn't defining it.
This allows `apply` functions to return a valid value if they completely
ignore their argument, which is the case for the option renaming
functions like `mkAliasOptionModule`. Therefore this solves issue #63693
The explicit remove helped to uncover some hidden uses of `optionSet`
in NixOps. However it makes life harder for end-users of NixOps - it will
be impossible to deploy 19.03 systems with old NixOps, but there is no
new release of NixOps with `optionSet` fixes.
Also, "deprecation" process isn't well defined. Even that `optionSet` was
declared "deprecated" for many years, it was never announced. Hence, I
leave "deprecation" announce. Then, 3 releases after announce,
we can announce removal of this feature.
This type has to be removed, not `throw`-ed in runtime, because it makes
some perfectly fine code to fail. For example:
```
$ nix-instantiate --eval -E '(import <nixpkgs/lib>).types' --strict
trace: `types.list` is deprecated; use `types.listOf` instead
error: types.optionSet is deprecated; use types.submodule instead
(use '--show-trace' to show detailed location information)
```
This commit changes the `mkAliasOptionModule` function to make sure that
the priority for the aliased option is propagated to the non-aliased
option.
This also affects the `mkRenamedOptionModule` function in a similar
fashion.
This also removes the `mkAliasOptionModuleWithPriority` function, since
its functionality is now subsumed by `mkAliasOptionModule`.
This change was recommended by @nbp:
https://github.com/NixOS/nixpkgs/pull/53397#discussion_r245487432
This commit adds a function `mkAliasOptionModuleWithPriority`. This
function will make an alias to an existing option and copy over the
priority.
This functionality is needed for PRs like #53041. In that case
`nixos-generate-config` added an option to `hardware-configuration.nix`
with `mkDefault`. That option was then changed and an alias created for
the old name.
The end user should be able to set the non-alias option in their
`configuration.nix` and have everything work correctly. Without this
function, the priority for the option won't be copied over correctly
and the end-user will get a message saying they have the same option
set to two different values.
Before this change `mkRenamedOptionModule` would override option defaults
even when the old option name is left unused. For instance
```nix
{
optios = {
services.name.new = mkOption {
default = { one = {}; };
};
};
imports = [
(mkRenamedOptionModule [ "services" "name" "old" ] [ "services" "name" "new" "two" ])
];
config = {};
}
```
would evaluate to
`{ config.services.name.new = { two = {}; }; }`
when you'd expect it to evaluate to
`{ config.services.name.new = { one = {}; }; }`.
The function isn’t used anywhere and `addErrorContext` is an undocumented
builtin.
The builtin is explicitely qualified at its two uses in the module system.
This allows the lib fixed point to be extended with
myLib = lib.extend (self: super: {
foo = "foo";
})
With this it's possible to have the new modified lib attrset available to all
modules when using evalModules
myLib.evalModules {
modules = [ ({ lib, ... }: {
options.bar = lib.mkOption {
default = lib.foo;
};
}) ];
}
=> { config = { bar = "foo"; ... }; options = ...; }
Among other things, this will allow *2nix tools to output plain data
while still being composable with the traditional
callPackage/.override interfaces.
Before:
<x> is not a integer between 0 and 100 (inclusively).
(notice that “a” is wrong, it should be “an”)
Now:
<x> is not of type `integer between 0 and 100 (inclusively)'.
This sounds a bit more formal, but circumvents the grammatical problems.
Multi-word type descriptions are also easier to see.
This does break the API of being able to import any lib file and get
its libs, however I'm not sure people did this.
I made this while exploring being able to swap out docFn with a stub
in #2305, to avoid functor performance problems. I don't know if that
is going to move forward (or if it is a problem or not,) but after
doing all this work figured I'd put it up anyway :)
Two notable advantages to this approach:
1. when a lib inherits another lib's functions, it doesn't
automatically get put in to the scope of lib
2. when a lib implements a new obscure functions, it doesn't
automatically get put in to the scope of lib
Using the test script (later in this commit) I got the following diff
on the API:
+ diff master fixed-lib
11764a11765,11766
> .types.defaultFunctor
> .types.defaultTypeMerge
11774a11777,11778
> .types.isOptionType
> .types.isType
11781a11786
> .types.mkOptionType
11788a11794
> .types.setType
11795a11802
> .types.types
This means that this commit _adds_ to the API, however I can't find a
way to fix these last remaining discrepancies. At least none are
_removed_.
Test script (run with nix-repl in the PATH):
#!/bin/sh
set -eux
repl() {
suff=${1:-}
echo "(import ./lib)$suff" \
| nix-repl 2>&1
}
attrs_to_check() {
repl "${1:-}" \
| tr ';' $'\n' \
| grep "\.\.\." \
| cut -d' ' -f2 \
| sed -e "s/^/${1:-}./" \
| sort
}
summ() {
repl "${1:-}" \
| tr ' ' $'\n' \
| sort \
| uniq
}
deep_summ() {
suff="${1:-}"
depth="${2:-4}"
depth=$((depth - 1))
summ "$suff"
for attr in $(attrs_to_check "$suff" | grep -v "types.types"); do
if [ $depth -eq 0 ]; then
summ "$attr" | sed -e "s/^/$attr./"
else
deep_summ "$attr" "$depth" | sed -e "s/^/$attr./"
fi
done
}
(
cd nixpkgs
#git add .
#git commit -m "Auto-commit, sorry" || true
git checkout fixed-lib
deep_summ > ../fixed-lib
git checkout master
deep_summ > ../master
)
if diff master fixed-lib; then
echo "SHALLOW MATCH!"
fi
(
cd nixpkgs
git checkout fixed-lib
repl .types
)
* lib: introduce imap0, imap1
For historical reasons, imap starts counting at 1 and it's not
consistent with the rest of the lib.
So for now we split imap into imap0 that starts counting at zero and
imap1 that starts counting at 1. And imap is marked as deprecated.
See c71e2d4235 (commitcomment-21873221)
* replace uses of lib.imap
* lib: move imap to deprecated.nix
Nix style seems to have settled on not using spaces between bound
variable names and the lambda : so I also tried to make those somewhat
more consistent throughout.
This is based on a prototype Nicolas B. Pierron worked on during a
discussion we had at FOSDEM.
A new version with a workaround for problems of the reverted original.
Discussion: https://github.com/NixOS/nixpkgs/commit/3f2566689
This gives a nicer error message than (say)
while evaluating the option `fileSystems':
while evaluating the attribute ‘isDefined’ at /nix/store/r8z4vvl2qzg31zm4ra6awlx5b22k7gf9-nixos-16.09.tar.gz/lib/modules.nix:323:5:
while evaluating ‘filterOverrides’ at /nix/store/r8z4vvl2qzg31zm4ra6awlx5b22k7gf9-nixos-16.09.tar.gz/lib/modules.nix:395:21, called from /nix/store/r8z4vvl2qzg31zm4ra6awlx5b22k7gf9-nixos-16.09.tar.gz/lib/modules.nix:307:18:
while evaluating ‘concatMap’ at /nix/store/r8z4vvl2qzg31zm4ra6awlx5b22k7gf9-nixos-16.09.tar.gz/lib/lists.nix:79:18, called from /nix/store/r8z4vvl2qzg31zm4ra6awlx5b22k7gf9-nixos-16.09.tar.gz/lib/modules.nix:401:8:
while evaluating ‘concatMap’ at /nix/store/r8z4vvl2qzg31zm4ra6awlx5b22k7gf9-nixos-16.09.tar.gz/lib/lists.nix:79:18, called from /nix/store/r8z4vvl2qzg31zm4ra6awlx5b22k7gf9-nixos-16.09.tar.gz/lib/modules.nix:302:17:
while evaluating anonymous function at /nix/store/r8z4vvl2qzg31zm4ra6awlx5b22k7gf9-nixos-16.09.tar.gz/lib/modules.nix:302:28, called from undefined position:
while evaluating ‘dischargeProperties’ at /nix/store/r8z4vvl2qzg31zm4ra6awlx5b22k7gf9-nixos-16.09.tar.gz/lib/modules.nix:365:25, called from /nix/store/r8z4vvl2qzg31zm4ra6awlx5b22k7gf9-nixos-16.09.tar.gz/lib/modules.nix:303:62:
value is a list while a Boolean was expected
When displaying a warning about a removed Option we should always
include reasoning why it was removed and how to get the same
functionality without it.
Introduces such a description argument and patches occurences (mostly
with an empty string).
startGnuPGAgent: further notes on replacement
- Enforce that an option declaration has a "defaultText" if and only if the
type of the option derives from "package", "packageSet" or "nixpkgsConfig"
and if a "default" attribute is defined.
- Enforce that the value of the "example" attribute is wrapped with "literalExample"
if the type of the option derives from "package", "packageSet" or "nixpkgsConfig".
- Warn if a "defaultText" is defined in an option declaration if the type of
the option does not derive from "package", "packageSet" or "nixpkgsConfig".
- Warn if no "type" is defined in an option declaration.
Option aliases/deprecations can now be declared in any NixOS module,
not just in nixos/modules/rename.nix. This is more modular (since it
allows for example grub-related aliases to be declared in the grub
module), and allows aliases outside of NixOS (e.g. in NixOps modules).
The syntax is a bit funky. Ideally we'd have something like:
options = {
foo.bar.newOption = mkOption { ... };
foo.bar.oldOption = mkAliasOption [ "foo" "bar" "newOption" ];
};
but that's not possible because options cannot define values in
*other* options - you need to have a "config" for that. So instead we
have functions that return a *module*: mkRemovedOptionModule,
mkRenamedOptionModule and mkAliasOptionModule. These can be used via
"imports", e.g.
imports = [
(mkAliasOptionModule [ "foo" "bar" "oldOption" ] [ "foo" "bar" "newOption" ]);
];
As an added bonus, deprecation warnings now show the file name of the
offending module.
Fixes #10385.
This move idioms which were used in `evalOptionValue` and in the `merge`
functions of `listOf` and `attrsOf` types, such that we can use a names such
as `isDefined` and `optionalValue` instead or repeating identical
comparisons of `defsFinal == []`.
The current implementation of the ApplyIfFunction is looking at the
arguments of a module to decide which arguments should be given to each
module. This patch make sure that we do not wrap a submodule function in
order to keep functionArgs working as expected.
Ideally the module system could be configured pretty much completely by
the contents of the modules themselves, so add comments about avoiding
complicating it further and possibly removing now-redundant
configurability from the existing interface.
This allows for module arguments to be handled modularly, in particular
allowing the nixpkgs module to handle the nixpkgs import internally.
This creates the __internal option namespace, which should only be added
to by the module system itself.
This makes the relationship between property types clearer, and more
importantly will let option types parameterized by other option types
reuse the code for delegated type checking and merging.
Now we should be able to have multiple declaration of the same option as
long as all declarations have the same type. If the type has a sub module,
then it is merged with the submodules of other declarations, as done with
option sets.
In addition, the file of the option declaration is passed into the
submodule, such as the documentation can display it correctly.
This modification improves NixOS manual by listing in which file, each
submodule option is declared. This solve the issue that files are not
reported when looking at options such as fileSystems.<name?>.neededForBoot
E.g.
The unique option `fileSystems./.device' is defined multiple times, in `/etc/nixos/configuration.nix' and `/etc/nixos/foo.nix'.
This requires passing file/value tuples to the merge functions.
Now that overriding fileSystems in qemu-vm.nix works again, it's
important that the VM tests that add additional file systems use the
same override priority. Instead of using the same magic constant
everywhere, they can now use mkVMOverride.
http://hydra.nixos.org/build/6695561
For instance, if time.timeZone is defined multiple times, you now get
the error message:
error: user-thrown exception: The unique option `time.timeZone' is defined multiple times, in `/etc/nixos/configurations/misc/eelco/x11vnc.nix' and `/etc/nixos/configuration.nix'.
while previously you got:
error: user-thrown exception: Multiple definitions of string. Only one is allowed for this option.
and only an inspection of the stack trace gave a clue as to what
option caused the problem.
Also, when an option definition fails to type-check, print the file
name of the module in which the offending definition occurs, e.g.
error: user-thrown exception: The option value `boot.loader.grub.version' in `/etc/nixos/configuration.nix' is not a integer.
The major changes are:
* The evaluation is now driven by the declared options. In
particular, this fixes the long-standing problem with lack of
laziness of disabled option definitions. Thus, a configuration like
config = mkIf false {
environment.systemPackages = throw "bla";
};
will now evaluate without throwing an error. This also improves
performance since we're not evaluating unused option definitions.
* The implementation of properties is greatly simplified.
* There is a new type constructor "submodule" that replaces
"optionSet". Unlike "optionSet", "submodule" gets its option
declarations as an argument, making it more like "listOf" and other
type constructors. A typical use is:
foo = mkOption {
type = type.attrsOf (type.submodule (
{ config, ... }:
{ bar = mkOption { ... };
xyzzy = mkOption { ... };
}));
};
Existing uses of "optionSet" are automatically mapped to
"submodule".
* Modules are now checked for unsupported attributes: you get an error
if a module contains an attribute other than "config", "options" or
"imports".
* The new implementation is faster and uses much less memory.
