From Sphinx to DocFX - Generating Python API Documentation with DocFx

Overview | Terminology | An Example

Overview

In a previous post Sphinx Quickstart, we covered a very basic setup of Sphinx. In this post, we go farther and talk about Sphinx DocFX YAML, an exporter for the Sphinx Autodoc module. Our goal is to produce YAML files that can be consumed by DocFX, a documentation generator for .NET that also converts YAML files to HTML. Many doc sets at https://docs.microsoft.com/ are generated with DocFX, including Python doc sets that use Sphinx to generate YAML, which is then converted to HTML with DocFX.

Terminology

To understand how to go from Sphinx to DocFX using the Sphinx DocFX YAML exporter, we need to break down some of the terms used...or at least we did to make sense of it all.

Sphinx

• Sphinx is a documentation generator, it was originally created for Python documentation, but can be used for a range of languages.
• Sphinx uses reStructuredText (rST) as its markup language. Sphinx's utility comes from the power and straightforwardness of reStructuredText (reST) and its parsing and translating suite, Docutils. reST is used both in .rst files and in docstrings in .py files.
• In the Quickstart, we created an example reSt file (foo.rst) and built HTML documentation from it. 

Autodoc

• Autodoc is an extension for Sphinx. (Sphinx is extensible to support the needs of different projects. An extension is simply a Python module.)
• Autodoc adds directives like "autofunction" and "automodule". These directives determine what API is used to generate docs.
• When using the autodoc extension (added in the conf.py file) with Sphinx, you are including documentation from Python docstrings. A Python docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. Such a docstring becomes the __doc__ special attribute of that object.
• When you run the command sphinx-build (or make html if it was created), Sphinx autodoc generates the API documentation for your Python project code using the index.rst (this is the default name, but it can be any name you want). Sphinx imports the code via standard Python import mechanisms, and then generates the proper reST in the Sphinx Python domain. The reST files are then parsed to create doctree files used internally in Sphinx to generate HTML. If you only want the HTML output from Sphinx (and not DocFX), then you can stop here. This is the point at which the post Sphinx Quickstart stops.

sphinx-build

• Usage: sphinx-build [options] <sourcedir> <outdir> [filenames...]
• (If you ran sphinx-quickstart, you had the option of creating a make file so that you can just type make html instead of sphinx-build.)
• This command creates documentation from files in <sourcedir>and places HTML in <outputdir>.
• This command command looks for <sourcedir>/conf.py for configuration settings.
• This command creates documentation in different formats. A format can be specified on the command line, otherwise it defaults to HTML. (Check the conf.py file if in doubt.)
• By default, everything that is outdated is built. Output only for selected files can be built by specifying individual filenames.
• Since Sphinx has to read and parse all source files before it can write an output file, the parsed source files are cached as “doctree pickles”. Normally, these files are put in a directory called .doctrees under the build directory.
• If you didn't run the Sphinx Quickstart and don't have an index.rst file to start with, then you could use the sphinx-apidoc command to create module .rst files that would be equivalent to index.rst.

Sphinx DocFx YAML

• Sphinx DocFX YAML is an exporter for the Sphinx Autodoc module that produces YAML files adhering to the DocFX YAML metadata specification. For more information, see readthedocs.
• DocFX YAML describes language metadata for programming languages. The main user scenario for language metadata is to generate reference documentation. Specifically, we can use the YAML as input to DocFX and let DocFX generate HTML.
• YAML files represent the API documentation. Example.
• DocFX stands for Document Frameworks. To use it, add the extension to the source\conf.py file like so:
  
  extensions = ['sphinx.ext.autodoc', 'docfx_yaml.extension']
• With exporter added to conf.py, use Sphinx DocFx as usual by running the command make html.

DocFX

• DocFX generates API reference documentation from triple-slash comments in C#\VB  source code. Or, it can consume YAML files and render them as HTML.
• It also allows you to use Markdown files to create additional topics such as tutorials and how-tos, and to customize the generated reference documentation.
• The punchline is this: From a Python project using Autodoc and SphinxDocFX YAML exporter, you can generate YML files to be used with DocFX. This is what the example below does.
• Why? Because HTML generated from DocFX has a number of benefits beyond the HTML generated from Sphinx, including API cross referencing, generating from markdown files (.md) alongside API reference, customizable themes and templates.  

An Example

Prerequisites:

• Sphinx is installed.
• DocFx is installed.

Step 1: Clone the travelmarx-blog repo and start in the sphinx-docfx-example directory.

sphinx-docfx-example folder is the root folder. in subsequent steps. You should have the following:

.
└───mycode
    ├───core_api
    │   ├───package1
    │   └───package2
    └───test_api

Step 2: Create config.py and index.rst files.

See the Sphinx Quickstart for information about running the sphinx-quickstart command. Your folder structure should look like this.

.
├───build
├───mycode
│   ├───core_api
│   │   ├───package1
│   │   └───package2
│   └───test_api
└───source
    ├───_static
    └───_templates

Step 3: Edit source\config.py.

Configure the extensions:

extensions = ['sphinx.ext.autodoc', 'docfx_yaml.extension']

Point to the code folder:

import os
import sys
sys.path.insert(0, os.path.abspath('../mycode'))

Some of these lines in the config.py file may already exist and you'll have to uncomment them.

Step 4: Run sphinx-apidoc to create .rst (reStructuredText) files describing the code.

Starting in the sphinx-docfx-example (root) folder, run:

sphinx-apidoc -o source .\mycode

This creates .rst files in the \source folder.

Step 5: Modify the source\index.rst to include modules to document.

Running sphinx-apidoc will produce a source\modules.rst file by default. The modules.rst file is the entry point for documenting the code in \mycode.

source\index.rst (snippet, add the part in red)

Test documentation
=======================
.. toctree::
    :maxdepth: 4
    :caption: Table of Contents 

    modules

Step 6: Run sphinx-build to create Sphinx's HTML.

sphinx-build source build

Besides building the Sphinx HTML (which you may not care about), this also creates .yml files in the \build\docfx_yaml folder. These will be used in a later step with DocFx.

To view the Sphinx HTML, starting in the root folder, run:

build\index.html

For comparison with docFx HTML (which is generated in Step 7), here is the Sphinx-generated HTML:

Step 7: Confirm that YAML files were generated.

Starting in root folder, run:

dir build\docfx_yaml

You should see a listing of .yml files like "core_api.package1.someclass.SomeClass.yml".

Step 8: Generate an initial docfx.json file.

Starting in the root folder, run:

docfx init -q

This will create a docfx_project folder with the docfx.json configuration file.

Step 9: Copy the Sphinx YAML files to the \docfx_project folder.

Copy .\build\docfx_yaml\* to .\docfx_project\api\*

Step 10: Build the DocFx HTML and serve the docs.

Starting in docfxtest folder, run:

docfx docfx_project\docfx.json --serve

Step 11: View the HTML docs produced by DocFx.

Go to http://localhost:8080.

Some points to note:

• The difference in the look between Sphinx HTML and DocFx HTML. Both can be customized as needed.
• How the link to "AnotherClass" is an active link in the DocFX screenshot. This is one of the benefits of using DocFx, cross reference linking.
• We didn't add any "Articles" (.md files) but that is also a nice feature of DocFx, to integrate API and conceptual (articles) docs. For example of how that could be done, see our Scrapbook101core site.
• On subsequent runs through the steps above (say, if you changed a docstring in the code), you will typically:
• delete content in \build folder
• run steps 6, 9, and 10.

Next Steps:

• Customize docfx.json file.
• Read up on cross-linking with DocFx.
• Add other markdown files (.md) along with API docs.