add the ability to set the info string for a newly created fenced code
block, and a flag to always emit a fenced block. the commonmark
converter will need this to faithfully recreate fenced and indented code
blocks.
with mypy type checking and Mapping types this is a lot less useful than
anticipated. let's drop it for simplicity and having fewer dependencies.
frozendict 2.3.5 also broke the mypy checks.
options processing is pretty slow right now, mostly because the
markdown-it-py parser is pure python (and with performance
pessimizations at that). options parsing *is* embarassingly parallel
though, so we can just fork out all the work to worker processes and
collect the results.
multiprocessing probably has a greater benefit on linux than on darwin
since the worker spawning method darwin uses is less efficient than
fork() on linux. this hasn't been tested on darwin, only on linux, but
if anything darwin will be faster with its preferred method.
this adds support for structural includes to nixos-render-docs.
structural includes provide a way to denote the (sub)structure of the
nixos manual in the markdown source files, very similar to how we used
literal docbook blocks before, and are processed by nixos-render-docs
without involvement of xml tooling. this will ultimately allow us to
emit the nixos manual in other formats as well, e.g. html, without going
through docbook at all.
alternatives to this source layout were also considered:
a parallel structure using e.g. toml files that describe the document
tree and links to each part is possible, but much more complicated to
implement than the solution chosen here and makes it harder to follow
which files have what substructure. it also makes it much harder to
include a substructure in the middle of a file.
much the same goes for command-line arguments to the converter, only
that command-lined arguments are even harder to specify correctly and
cannot be reasonably pulled together from many places without involving
another layer of tooling. cli arguments would also mean that the manual
structure would be fixed in default.nix, which is also not ideal.
<part> is different from all other blocks we care about in that it
requires textual content to be wrapped in <partintro>. add support for
this to the generic docbook renderer, which will just assume that a part
is the whole document start to finish. we do make provision for the
manual renderer to close a partintro tag early though.
__context__ is always set to the prior exception, even when not using
the raise from form. __cause__ is only set during raise from. use
__cause__ so we can override a leaf exception (eg KeyError to something
more meaningful).
render all manual chapters to docbook from scratch every time the manual
is built. nixos-render-docs is quick enough at this to not worry about
the cost (needing only about a second), and it means we can remove
md-to-db.sh in the next commit.
no changes to the rendered html manual except for replacements and smartquotes.
we'll soon add another docbook converter that does not emit a section as
a collection of chapters, but sections or chapters on their own. this
should clarify naming a bit before there can be any confusion.
this is currently only supported by the docbook exporter, and even the
docbook exporter doesn't do much with them. we mirror the conversion
pandoc did for consistency with the previous manual chapter conversion,
which is to add just an anchor with the given id. future exporters that
go directly to html might want to do more.
this is a subset of pandoc's fenced divs. currently we only use this for
admonitions (which get a new name to differentiate them from other kinds
of blocks), but more users will appear soon.
this is used by release notes (and we don't want to break links to
those), and is also technically allowed anyway. we will *not* extend the
regex to allow more characters just yet due to a mozilla recommendation
against it (cf https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/id)
this should've been a core rule from the beginning. not being a core
rule made it always run after smartquotes and replacements, which
could've wrecked the id.
this lets us parse the `[F12]{.keycap}` syntax we recently introduced to
the nixos manual markdown sources. the docbook renderer emits the keycap
element for this class, the manpage renderer will reject it because it's
not entirely clear what to do with it: while html has <kbd> mandoc has
nothing of the sort, and with no current occurences in options doc we
don't have to settle on a (potentially bad) way to render these.
this is pretty much what pandoc calls bracketed spans. since we only
want to support ids and classes it doesn't seem fair to copy the name,
so we'll call them "attributed span" for now. renderers are expected to
know about *all* classes they could encounter and act appropriately, and
since there are currently no classes with any defined behavior the most
appropriate thing to do for now is to reject all classes.
now that the renderer produces the output we want to keep for the future
we can add a test that checks all of its features. this test notably
does not include markdown headings since we don't want to have those in
manpages (at least right now), but tests for other converters may add
headings for themselves.
for the longest time we completely dropped link targets in
configuration.nix.5. let's stop doing this now and instead provide a
footnote for each link in a given option, numbered locally per option.
we will currently duplicate the link for <labelless-links> because it
makes it easier to get the collection of all links in a given option.
this may not be useful enough, so over time we might decide to drop the
footnotes for such links.
indent the entire list by 4, just like each definition is already
indented by 4. this matches rendering in html, which indents terms once
and indents definitions twice.
other output types already have markings for inline code, manpages do
not. this can be somewhat confusing, so we'll do the least intrusive
thing: surrounding inline code blocks in ‘’. doing so separates inline
code from the rest of the text and is unlikely to collide with the
quoted contents. it's also what mdoc does with its Ql macro.
no reason to differentiate between links by source of their label. this
feature seems to be mostly used to change labels of links to other
options, but this should ultimately be done by auto-linking from
{option}`...`. at some point we may want to introduce a warning when
this pattern is encountered, but there's a lot to work out still before
we can do that.
most of the lists in option docs are actually compact, but docbook to
manpage processing always rendered them as non-compact. compactifying
these lists improves readability somewhat since most lists and their
contents are pretty short.
mdoc is just too slow to render on groff, and semantic markup doesn't
help us any for generated pages.
this produces a lot of changes to configuration.nix.5, but only few
rendering changes. most of those seem to be place losing a space where
docbook emitted roff code that did not faithfully represent the input
text, though a few places also gained space where docbook dropped them.
notably we also don't need the compatibility code docbook-xsl emitted
because that problem was fixed over a decade ago.
this will handle block quotes, which the docbook stylesheets turned into
a mess of roff requests that ended up showing up in the output instead
of being processed.
Token.attr is a dict[str, str | int | float], meta has no restriction on
the value type. attrs is ostensibly meant for html attributes, meta for
any information whatsoever.
these work together with render and renderInline to produce an output
from either of the two. rendering manpages will need both: to join
blocks with newlines, and to run some postprocessing and the rendered inlines.
pulling mypy into the build closure is unfortunately not reasonable, the
closure for mypy is rather large and takes a long time to build. if we
have the type checks hooked into CI we'll get most of the benefit though.
this is not yet able to produce manual-combined.xml, but the intention
is to add that support before too long. for now we'll concentrate on
getting the basics working: concatenating a list of chapters into a
manual-combined fragment, which will be rendered via docbook.
previously we did not detect whether lists were supposed to be compact
or not. this will make a difference for manual chapters, so let's stop
not doing that.
headings are not supported in options docs (since it's unclear what that
would be in the final manual, and the docbook stylesheets already have
trouble rendering all docbook constructs correctly). ordered lists
should be supported, but obviously nothing uses them yet.
options do not use comments, but a number of manual chapters do. since
we don't want to enable html just so we can then inspect the html and
figure out whether it's a comment we'll instead add a plugin that
detects comments natively.
we will soon add plugins to this tool to support nixos markdown features
that aren't readily supported with markdown-it plugins. since we will
have to test these plugin we'll need access to the parser, and since
we'll also want to add functions that require postprocessing of a parsed
token stream we also add the necessary hooks now.
with some fiddling and custom validation logic we can avoid needing
multiple instances of this plugin. originally this wasn't done because
it does need a type stack to emit correct docbook, but since we can
easily do that now we may as well.
