Building a python environment with python3Minimal requires hydra
to bootstrap pip and build all packages used in the environment
which would otherwise not be built. This reduces cache re-use and duplicates things.
Also common dependencies normally included in python itself
are not properly checked and can cause hard to debug errors
because everyone just assumes those modules are there.
they're no longer necessary for us and will almost definitely start to
rot now (like commonmark and asciidoc outputs did previously). most
existing users seem to take the docbook output and run it through pandoc
to generate html, those can easily migrate to use commonmark instead.
other users will hopefully pipe up when they notice that things they rely
on are going away.
optionsUsedDocbook has only been around for one release and only exposed
to allow other places to generate warnings, so that does not deserve
such precautions.
with everything being rendered from markdown now we no longer need to
postprocess any options.xml that may be requested from elsewhere. we'll
don't need to keep the module path check either since that's done by
optionsJSON now.
docbook is now gone and we can flip the defaults. we won't keep the
command line args around (unlike the make-options-docs argument) because
nixos-render-docs should not be considered an exposed API.
with docbook no longer supported we can default to markdown option docs.
we'll keep the parameter around for a bit to not break external users
who set it to true. we don't know of any users that do, so the
deprecation period may be rather short for this one.
it's been long in the making, and with 23.05 out we can finally disable
docbook option docs and default to markdown instead. this brings a
massive speed boost in manual and manpage builds, so much so that we may
consider enabling user module documentation by default.
we don't remove the docbook support code entirely yet because it's a lot
all over, and probably better removed in multiple separate changes.
the old method of pasting parts of options.json into a markdown document
and hoping for the best no longer works now that options.json contains
more than just docbook. given the infrastructure we have now we can
actually render options.md properly, so we may as well do that.
options processing is pretty slow right now, mostly because the
markdown-it-py parser is pure python (and with performance
pessimizations at that). options parsing *is* embarassingly parallel
though, so we can just fork out all the work to worker processes and
collect the results.
multiprocessing probably has a greater benefit on linux than on darwin
since the worker spawning method darwin uses is less efficient than
fork() on linux. this hasn't been tested on darwin, only on linux, but
if anything darwin will be faster with its preferred method.
using environment variables isn't great once multiple input or output
formats get involved (which will happen soon). now is a good time to set
a pattern for future converters.
this new package shall eventually contain the rendering code necessary
to produce the entirety of the nixos (not nixpkgs) manual, in all of its
various output formats.
the rest of the nixos manual has them enabled, so we should enable them
here too for consistency.
this changes rendered output pervasively. changes also include quotes in
types (eg in `strings concatenated with "\n"`), but since those are not
code this is probably fine. if not we can probably add a myst role to
inhibit replacements.
the rules are fixed, and we want to support all of them (or throw a
useful error message). this will also become the base for a generic
renderer system, so let's just list all the rules statically.
this restores mergeJSON to its former glory if…merging json, and
extracts the MD rendering into a new script that will run instead of the
py+nix+xslt pipeline we previously ran to convert options.json to docbook.
this change alone gives a noticable performance boost when building
docs (18s instead of 27s to build optionsDocBook).
no changes to rendered output, except for a single example in the
rsnapshot module that uses hard tabs for indentation instead of spaces.
this probably isn't important.
docbook warnings remain with mergeJSON since the other processing steps
output single files instead of directories. since we'll only keep the
check until 23.11 this is probably also not important to fix.
also contains a few improvements to error reporting in the MD renderers.
only whitespace changes (mostly empty descriptions rendered as literal
line breaks and trailing space toPretty generates, but that were dropped
by mistune).
don't generate docbook for related packages, generate markdown instead.
this could be extended further to not even generate markdown but have
mergeJSON handle all of the rendering. markdown will work fine for now
though.
only whitespace changes to rendered outputs, all in the vicinity or body
of admonitions. previously admonitions would not receive paragraph
breaks even when they should have because the description postprocessing
did not match on their contents.
markdown-it-py creates different whitespace leaders/trailers than are
currently emitted, and when we convert examples and defaults to render
via markdown the spacing will change too. this has no effect on rendered
output.
mistune already does escaping. it does escaping for html, but the
difference is small enough that can just ignore that we're actually
targeting docbook here.
this was done only to make the conversion to MD easier to verify. we no
longer need it, and not keeping whitespace does not affect rendered outputs.
stripping will have to stay for now because description postprocessing
would add empty paragraphs otherwise.
following the plan in https://github.com/NixOS/nixpkgs/pull/189318#discussion_r961764451
also adds an activation script to print the warning during activation
instead of during build, otherwise folks using the new CLI that hides
build logs by default might never see the warning.
mkAliasOptionModule should not default to mdDoc descriptions because
that can break out-of-tree users of documentation infrastructure. add an
explicitly-MD variant for now, to be removed some time after the MD
transition is complete.
Render un`_type`d defaults and examples as `literalExpression`s using
`lib.generators.toPretty` so that consumers don't have to reinvent Nix
pretty-printing. `renderOptionValue` is kept internal for now intentionally.
Make `toPretty` print floats as valid Nix values (without a tilde).
Get rid of the now-obsolete `substSpecial` function.
Move towards disallowing evaluation of packages in the manual by
raising a warning on `pkgs.foo.{outPath,drvPath}`; later, this should
throw an error. Instead, module authors should use `literalExpression`
and `mkPackageOption`.
Unlike the XML doc renderer, the AsciiDoc and CommonMark renderers don't
pretty-print certain complex types, like literal expressions, DocBook
literals, and derivations. These types are dumped into the documentation
as JSON.
This commit parses and unwraps these types when loading the
JSON-formatted NixOS options. The AsciiDoc and CommonMark renders have
also been combined into a single script to allow code reuse.
\<foo\> will often be displayed like \<foo>, for example by mkdocs.
I've tested a number of markdown renderers and they render html escape
sequences fine.
