# Output directory soon required for CmdLineTasks **URL:** \ **Category:** DM Notifications\ **Tags:** argumentparser, pipe\_base, cmdlinetask\ **Created:** [March 12, 2016, 12:52am UTC](https://www.rubin.community/t/output-directory-soon-required-for-cmdlinetasks/598 "2016-03-12T00:52:56Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [March 12, 2016, 12:52am UTC](https://www.rubin.community/t/output-directory-soon-required-for-cmdlinetasks/598/1 "2016-03-12T00:52:56Z")
There are three ways of specifying an output directory through the `ArgumentParser` used by our `CmdLineTask`s: - `--output` command-line argument: specify an output directory directly - `--rerun` command-line argument: specify an output directory as `/rerun/` - `PIPE_OUTPUT_ROOT` environment variable: specify an output directory directly There is [general agreement](http://www.rubin.community/t/new-argument-parser-behavior-rerun-flag-introduction-discussion/345) that it is not good if none of these is specified (because you get clutter in the top-level data repo, and data in shared repos may be clobbered unwittingly). **On [DM-4236](https://jira.lsstcorp.org/browse/DM-4236), we will introduce a change that makes it mandatory to specify an output directory (using any of the above means).** The only command that is excluded from this requirement currently is `ingestImages.py` (since that should ingest into the top-level data repo). I am concerned that this will break some CI or validation packages that I am not aware of or am not set up to check. I have already run Jenkins against `lsst_sims lsst_distrib ci_hsc`, and [it passes](https://ci.lsst.codes/job/stack-os-matrix/9142/). Please let me know what else needs to be tested (and how), or test the package against `tickets/DM-4236` (with changes in pipe\_base, pipe\_tasks and ci\_hsc). I hope to merge DM-4236 on Monday (March 14) afternoon in advance of the sprint closure. ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [March 12, 2016, 4:52am UTC](https://www.rubin.community/t/output-directory-soon-required-for-cmdlinetasks/598/2 "2016-03-12T04:52:40Z")
Note that it is a bug that you can specify output using only `$PIPE_OUTPUT_ROOT`. I would like to fix this, leaving `--output` and `--rerun` as the two options for specifying output. One question: do we envision any tasks for which an output repository is optional? At present I only know know of tasks for which output is always written or never written, but none which write optionally. DM-4236 presently supports tasks that require output and tasks that have optional output, but the flag used could easily be modified to enable/disable output instead. The advantage of disabling output is that the `--output` and `--rerun` argument are not wanted and could be omitted (never added to the argument parser) giving a more accurate help string. In any case, we do have a few tasks in pipe\_tasks that never write output (they summarize information about data in a repo, but do not produce new data products). These will need a minor fix to be compatible with DM-4236. ---
**Author:** ![mwv](https://avatars.discourse-cdn.com/v4/letter/m/e79b87/32.png) [@mwv](https://www.rubin.community/u/mwv)\ **Post date:** [March 12, 2016, 8:38pm UTC](https://www.rubin.community/t/output-directory-soon-required-for-cmdlinetasks/598/3 "2016-03-12T20:38:52Z")
`lsst_ci` will very soon be the thing to test. It’s (for now) just a metapackage to make sure `obs_cfht`, `obs_subaru`, `obs_decam`, `obs_lsstSim`, `obs_sdss` pass with their full `tesdata_*` -enabled tests. `lsst_distrib` almost does most of this, but because of the ambiguity about setupOptional/setupRequired under different build systems, it’s not required to test these. It does not run `ci_hsc` – that will be left for a daily build of something likely to be called `lsst_qa`. The `lsst_ci` pacake may also see future evolution as we come up with additional short-turn-around tests. `lsst_ci` will be introduced next week – it just passed review yesterday (thanks @ktl). `tickets/DM-4236` passes `lsst_ci` on Jenkins. [https://ci.lsst.codes/job/stack-os-matrix/label=centos-6/9173/console](https://ci.lsst.codes/job/stack-os-matrix/label=centos-6/9173/console) ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [March 15, 2016, 5:20am UTC](https://www.rubin.community/t/output-directory-soon-required-for-cmdlinetasks/598/4 "2016-03-15T05:20:13Z")
Thank you @mwv for your work on `lsst_ci`. This work has just passed (along with DM-5453, a fix to ci\_hsc) [Jenkins](https://ci.lsst.codes/job/stack-os-matrix/9255/), and a merge is imminent… tomorrow. ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [March 15, 2016, 2:53pm UTC](https://www.rubin.community/t/output-directory-soon-required-for-cmdlinetasks/598/5 "2016-03-15T14:53:10Z")
Merged. Let the mayhem ensue! ---
**Author:** ![ctslater](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ctslater/32/28_2.png) [@ctslater](https://www.rubin.community/u/ctslater)\ **Post date:** [March 16, 2016, 4:45pm UTC](https://www.rubin.community/t/output-directory-soon-required-for-cmdlinetasks/598/6 "2016-03-16T16:45:31Z")
This makes `--show data` or `--show config` somewhat harder to use. Is it feasible exempt those from this requirement? I believe `ingestCalibs.py` should also be treated the same way as `ingestImages.py`; is that right @price? I have a few other tiny fix-ups to that coming soon, so I can make the change. ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [March 16, 2016, 5:24pm UTC](https://www.rubin.community/t/output-directory-soon-required-for-cmdlinetasks/598/7 "2016-03-16T17:24:00Z")
I had deliberately put the check after the `--show` handling for this reason, but maybe @rowen moved it. You’re right that `ingestCalibs.py` should not require outputs (I had been thinking it inherited, but apparently not). ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [March 16, 2016, 5:51pm UTC](https://www.rubin.community/t/output-directory-soon-required-for-cmdlinetasks/598/8 "2016-03-16T17:51:10Z")
> [@price](#): > > I had deliberately put the check after the --show handling for this reason, but maybe @rowen moved it. I moved the check that one of --output or --rerun was supplied before creating a butler, because otherwise I saw problems from creating a butler with an invalid output. I guess finer tuning is wanted. The butler is wanted for some --show output, but not all of it.