# Pipelines package docs hack day instructions **URL:** \ **Category:** Data Management\ **Tags:** doc-engineering\ **Created:** [March 9, 2018, 12:59am UTC](https://www.rubin.community/t/pipelines-package-docs-hack-day-instructions/2742 "2018-03-09T00:59:07Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jsick](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jsick/32/672_2.png) [@jsick](https://www.rubin.community/u/jsick)\ **Post date:** [March 9, 2018, 12:59am UTC](https://www.rubin.community/t/pipelines-package-docs-hack-day-instructions/2742/1 "2018-03-09T00:59:07Z")
**Update #2: As of 2018-12-17, these instructions have been superseded by the [Developer Guide](https://developer.lsst.io/stack/layout-of-doc-directory.html), which takes precedence. (Added by @swinbank)** **Update: be sure to [read the later instructions](http://www.rubin.community/t/pipelines-package-docs-hack-day-instructions/2742/7?u=jsick) for how to continue doing numpydoc work after this hack day.** These are some rough notes for those working on Pipelines package documentation at the JTM 2018 hack day. The goal for the hack day is to get packages configured with the new `doc/` directory infrastructure, and begin to migrate docstrings over as well. The coordination page is [https://confluence.lsstcorp.org/pages/viewpage.action?pageId=73573930](https://confluence.lsstcorp.org/pages/viewpage.action?pageId=73573930) For the hack day, we won’t be using JIRA tickets to track earned value. We’ll use user branches, like: ```auto u/username/sphinx-docs ``` To claim a package to work on, add your name to the Notes column on the [Confluence page](https://confluence.lsstcorp.org/pages/viewpage.action?pageId=73573930). The goal is not to get a package fully converted to Numpydoc today. As soon as you have a Sphinx-buildable `doc/` directory (see below), feel free to merge your work. Lets not leave too many branches open after today. ## Set up In your set up stack environment, install documenteer: ``` pip install documenteer[pipelines] ``` (run with `pip install --user ...` if you’re on `lsst-dev01`). This package brings in documentation dependencies, and includes our own Sphinx customizations. ## Install the new doc/ infrastructure The first step is to install some boilerplate files in the package’s `doc/` directory. Refer to these: - **Templates:** [https://github.com/lsst/templates/tree/master/project\_templates/stack\_package/{{cookiecutter.package\_name}}/doc](https://github.com/lsst/templates/tree/master/project_templates/stack_package/%7B%7Bcookiecutter.package_name%7D%7D/doc) Note that these use the Jinja2 template format. Text wrapped in `{{ ... }}` are your cues to add a customization. - **Example:** [https://github.com/lsst/templates/tree/master/project\_templates/stack\_package/example/doc](https://github.com/lsst/templates/tree/master/project_templates/stack_package/example/doc) The files you need to add are: - `.gitignore` - `conf.py` - `manifest.yaml` - `index.rst` - `package_name/index.rst` (`package_name` is the EUPS package name) - `lsst.module.name/index.rst` (`lsst.module.name` is the namespace of the Python package. For example, `lsst.pipe.base`, or `lsst.afw.table`) All of these files, except for the `.gitignore` need to be customized. ### `conf.py` Set your package’s name and import the `version` module. ### `index.rst` Set the package title and set the paths in the `toctree` to the `index.rst` files in the `lsst.module.name` and `package_name` directories. ### `index.rst` in the module and package subdirectories Customize the titles of these pages and the anchor links. Fill in the introductory paragraph you see from the template. Don’t worry about making it completely awesome. For the `package_name/index.rst`, it shows a place to add the JIRA component. It’s hard to look up the JIRA component at the moment, so just put the name of the component there in plain text, like this: ```auto JIRA component ``ap_association`` ``` We’ll clean this up later. ### manifest.yaml In `manifest.yaml`, you’ll set the EUPS package name and the Python package name (or names, for situations like `afw`). Here’s an example `manifest.yaml` from `pipe_base`: [https://github.com/lsst/pipe\_base/blob/master/doc/manifest.yaml](https://github.com/lsst/pipe_base/blob/master/doc/manifest.yaml) ## Build the package documentation locally Once the boilerplate is added, you can try to build the package’s documentation locally. First, make sure you’ve `setup` the package and run `scons` as usual. Since the package is being imported, any `pybind11` wrappers need to be built first. Then from the `doc/` directory: ``` sphinx-build . _build/html ``` The root of the built HTML is `_build/html/index.html`. Note that this is a _single-package_ build. This is useful for building a package’s docs without building the entire `pipelines.lsst.io` site. Of course, at some point, there may be broken cross-package hyperlinks, but this is useful as a preview tool. ## Beginning the Numpydoc conversion, one module at a time You’ll find it easiest if you migrate docstrings one module (Python file) at a time, and rebuild the Sphinx site frequently to test for syntax issues. Guidelines for writing Numpydoc docstrings are at [https://developer.lsst.io/python/numpydoc.html](https://developer.lsst.io/python/numpydoc.html) Once you think you’ve got a module documented, go to the `doc/lsst.module.name/index.rst` file and add a `.. automodapi` directive to the “Python API reference” section. For example, if the namespace of the module is `lsst.module.name.exampleA`, the directive should be: ``` .. automodapi:: lsst.module.name.exampleA ``` For each module you convert, add an additional `automodapi` directive: ``` .. automodapi:: lsst.module.name.exampleA .. automodapi:: lsst.module.name.exampleB ``` Eventually, once all modules are converted, you should consolidate all the individual `automodapi` statements to point to just the _public_ module namespace of the package. For example, if the ` __init__.py` at `lsst.module.name` imports `exampleA` and `exampleB`, then their public-facing namespace is `lsst.module.name`. The consolidated `automodapi` directive is: ``` .. automodapi:: lsst.module.name ``` ## Integrated builds It’s not necessary for the initial package hacks, but if you’re curious, you can preview your package within the integrated `pipelines.lsst.io` site. This involves using the `tickets/DM-11216` branch of the `pipelines_lsst_io` repo. If you’d like you can create a temporary branch from `tickets/DM-11216`, but keep in mind that `tickets/DM-11216` itself may be rebased at some point. `tickets/DM-11216` will be merged to `master` once the new-style documentation becomes the default and is fully supported with a Jenkins workflow. ### Building the integrated pipelines documentation These are the steps to build the integrated docs at the moment: ``` git clone https://github.com/lsst/pipelines_lsst_io cd pipelines_lsst_io git checkout tickets/DM-11216 setup -r . build-stack-docs -d . ``` ### Adding your package To test your package’s documentation in the integrated build, you’ll need to make the following changes to the `pipelines_lsst_io` repo: 1. Add your package to `ups/pipelines_lsst_io.table` 2. Add the `lsst.module.name/index` of your module to the “Python modules” `toctree` in the `pipelines_lsst_io` `index.rst`. 3. Add the `package_name/index.rst` of the EUPS package to the “Packages” toctree in `pipelines_lsst_io` Then try to rebuild the documentation with the `build-stack-docs` command. I (@jsick ) will take care of adding packages to the DM-11216 branch as their upgrades land on `master`. ---
**Author:** ![jsick](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jsick/32/672_2.png) [@jsick](https://www.rubin.community/u/jsick)\ **Post date:** [March 9, 2018, 5:48pm UTC](https://www.rubin.community/t/pipelines-package-docs-hack-day-instructions/2742/2 "2018-03-09T17:48:46Z")
When you’re using `automodapi`, you’ll find that it will attempt to document _all_ the APIs in your module, including those from `import`s of other packages. What you need to do is use ` __all__ ` to curate the namespace of your modules to just the public APIs (functions, classes, module-level variables) that you are creating in that module. ---
**Author:** ![jsick](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jsick/32/672_2.png) [@jsick](https://www.rubin.community/u/jsick)\ **Post date:** [March 9, 2018, 6:31pm UTC](https://www.rubin.community/t/pipelines-package-docs-hack-day-instructions/2742/3 "2018-03-09T18:31:39Z")
Another thing is that you might need to clean up the cached files that Sphinx/automodapi creates. From the `doc/` directory, ```auto rm -R py-api ``` Do this if the Sphinx build shows a lot of missing links while you experimenting with Numpydoc. ---
**Author:** ![jsick](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jsick/32/672_2.png) [@jsick](https://www.rubin.community/u/jsick)\ **Post date:** [March 9, 2018, 6:32pm UTC](https://www.rubin.community/t/pipelines-package-docs-hack-day-instructions/2742/4 "2018-03-09T18:32:27Z")
Don’t worry about the warnings for missing `_static` directories. That can get cleaned up later in tooling. ---
**Author:** ![jsick](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jsick/32/672_2.png) [@jsick](https://www.rubin.community/u/jsick)\ **Post date:** [March 9, 2018, 8:15pm UTC](https://www.rubin.community/t/pipelines-package-docs-hack-day-instructions/2742/5 "2018-03-09T20:15:42Z")
Merge advice: Try to merge once your package is buildable with Sphinx (that `sphinx-build` command). In other words, you can do this as soon as you have the `doc/` boilerplate in place. When you do this, mark off the “doc/ status” column on the Confluence page. If you’re partway through Numpydoc conversion, you can still merge if the `automodapi` directive(s) are only pointing to the modules that have been successfully converted. In principle, you should be able to do this merge without Jenkins, though if you’ve changed the docstrings and broke a PEP8 rule, flake8 may mark it as a failure. If you’re being cautious, a Jenkins run is always a good idea. cc @mrawls ---
**Author:** ![parejkoj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/parejkoj/32/138_2.png) [@parejkoj](https://www.rubin.community/u/parejkoj)\ **Post date:** [March 28, 2018, 8:04pm UTC](https://www.rubin.community/t/pipelines-package-docs-hack-day-instructions/2742/6 "2018-03-28T20:04:57Z")
For posterity, we had two recommended commits as part of the hack day: 1. `Update doc/ directory for sphinx` - once you’ve got the relevant things copied from `templates/example/doc` and modified for the package. 2. `Cleanup numpydoc` - once you’ve got numpydoc cleaned up enough that the sphinx-build command works. Those would be the commits that go into your userbranch and once merged are enough to put the first check box on the confluence page. ---
**Author:** ![jsick](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jsick/32/672_2.png) [@jsick](https://www.rubin.community/u/jsick)\ **Post date:** [March 28, 2018, 8:35pm UTC](https://www.rubin.community/t/pipelines-package-docs-hack-day-instructions/2742/7 "2018-03-28T20:35:37Z")
If there is still numpydoc work left uncommitted after the hack day, here’s what I recommend doing: 1. Move any commits that update docstrings and [` __all__ ` attributes](http://www.rubin.community/t/pipelines-package-docs-hack-day-instructions/2742/2?u=jsick) to a ticket branch (create a documentation ticket) 2. Comment out the `automodapi` directive in the module’s `index.rst`. Put two full stops at the start of a line to comment it out: 3. Merge the **`doc/`** directory work. You shouldn’t need to do a review or Jenkins run for this specific merge. 4. Continue the docstring / ` __all__ ` work in the ticket branch. We recommend that you do the full ticket review process and Jenkins CI run for this work. ---
**Author:** ![jsick](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jsick/32/672_2.png) [@jsick](https://www.rubin.community/u/jsick)\ **Post date:** [December 17, 2018, 10:40pm UTC](https://www.rubin.community/t/pipelines-package-docs-hack-day-instructions/2742/8 "2018-12-17T22:40:43Z")
Closed on December 17, 2018, 10:40pm UTC