Technical documentation with Sphinx
This talk explains how one can use Sphinx for technical documentation, in particular this very site scicomp.aalto.fi. The focus is to make an overview for contributing to this site (or similar ones), but it will also provide a strong basis for creating such a site yourself.
About this site for a quick guide for editing this site.
Home of Aalto Scientific Computing’s documentation
Before 2017, was Triton’s documentation using Confluence (wiki software)
Now has information on many different topics about scientific computing.
Rather highly ranked in search engines.
Converted from wiki.aalto.fi (Triton) using
_meta/confluence2html.pyand then pandoc to convert HTML→ReST.
CC-BY license agreed at that time
Properties of good documentation
Organized, easy to use
Anyone can contribute
Shareable, reuseable, licensed
No lock-in, can migrate later
Plain text so 50 years of text processing development (
sed), etc all work.
Not standalone, can integrate with other materials (e.g.
git? (naturally comes out of the above)
The basic documentation stack
Hosted on Github
Documentation written in ReStructured Text or MyST-Markdown
Built with Sphinx
With various extensions
Hosted on ReadTheDocs
GitHub actions validate basic syntax
Demo: making a change
I want to add the Journal of Open Source Software (JOSS) review checklist (https://joss.readthedocs.io/en/latest/review_checklist.html) to the RSE checklists section (https://scicomp.aalto.fi/rse/#checklists).
Through this, we will see:
Git repository layout
ReStructructured Text format
Sphinx table of contents directives (
Creating a pull request with git-pr
Reviewing the pull request
See the rendered version.
Building the site
It has a
requirements.txtlike a normal Python project.
Until recently, was buildable with stock Debian/Ubuntu packages. Now it may require custom extensions.
conf.pycontains all configuration
index.rstis the root of all docs.
make htmlto make it
make clean htmlto rebuild
make clean checkto build and check for any errors
sphinx-autobuild . _build/html/may be useful - start a web server that automatically reloads on changes.
View results in
Editing on the web
The Github web interface is suitable for making simple changes.
You can either directly commit or open a PR.
Can we use this more?
Sphinx toctree (table of contents tree)
toctreedirective is the fundamental building block of the site.
It organizes documents into a tree, and that three is used to make the sidebar. This directive can be put into any page.
.. toctree:: :maxdepth: 2 aalto/* data/index README
Example: Follow it from
aalto/jupyterhub-instructors/index.rst→ various subpages.
It makes sense, but for complicated case I often do trial and error.
Arrangement of the site
scicomp.aalto.fi started from the Triton wiki
It then grew top-level sections for Aalto, Triton, Data, Training, RSE, etc.
It is about time that we rethink how it is organized.
rkdarst is currently the one with the overall picture in mind - for consultations about big changes.
Sphinx is a full-fledged extendable documentation generator
We use many extensions such as
Very useful to know for other projects in general
ReStructured Text syntax
Why ReST? Because it’s not a thin mapping on HTML like Markdown.
Markdown is syntactic substitution, ReST is semantic meaning.
MyST is now a reasonable alternative, but it is closer to a different ReST syntax than Markdown.
See syntax quickstart at https://scicomp.aalto.fi/README/
Most surprising ReST points:
Double quotes for literals:
Run ``nano`` to begin
Links are scoped:
Two underscores under links:
The main `Aalto website <https://aalto.fi/>`__
Github Action checks
make clean checkwill warn about the same errors that Github will fail on.
Github provides error tracking for pushes and pull requests (demo?).
I purposely have checks as rather strict and disabled some options that would allow us to do more flexible ReST: “explicit is better than implicit”.
https://readthedocs.org provides a management interface for the docs
There is a joint aalto-scicomp account to manage it
Demo if time, but pretty much self-explanatory
Occasionally a build fails for no reason and rkdarst needs to go wipe and rebuild, or fix dependency versions.
We could use Markdown or Jupyter
Compatible with many other projects
Standard documentation system for many projects
Used in recent CodeRefinery lessons, for example
Turn any site into a presentation
Can anyone help do this properly?
Redirect to HTTPS
ReadTheDocs doesn’t natively do this for external domains
Can anyone improve?
Other output formats
Sphinx can output to PDF, single-page HTML, epub, manual pages, and more.
Can anyone think of a use for this?
Written for Hands-on Scientific Computing
Bottom of every page lists date that exact page was actually modified.
Pull requests or not?
When should we use pull requests? When should we push directly?
In practice both are fine, up to you to decide what you want
rkdarst believes that, if you aren’t sure, push directly and ask for review.
Others at Aalto can use scicomp.aalto.fi
Should we encourage others to join our project here?
Our dream would be to make examples in a testable form, where one can automatically run them all and find errors.
For example, this python-openmp example includes everything needed to submit and run the file.
Can this be automatically tested? A bit too complex for the typical doctest.
Don’t use ReadTheDocs anymore?
Github Actions + GitHub Pages or other hosting sites would work instead of ReadTheDocs now.
How can we keep things up to date?
Requires continuous work, like any docs.
What should the threshold be for removing old material?
We now have a last updated time at the top.
We clearly need to think about this more.
ReadTheDocs provides limited stats based on web server logs.
rkdarst is against detailed web tracking.
Can we find a way to get both?.
2022 update: we have Plausible analytics which is sufficiently anonymous.
Building a community
How can we get more people to contribute?