# Storing knowledge of Stack runs in a production environment

**URL:** <https://www.rubin.community/t/storing-knowledge-of-stack-runs-in-a-production-environment/429>\
**Category:** Data Management\
**Created:** [December 8, 2015, 2:08pm UTC](https://www.rubin.community/t/storing-knowledge-of-stack-runs-in-a-production-environment/429 "2015-12-08T14:08:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ChristianArnault](https://avatars.discourse-cdn.com/v4/letter/c/3be4f8/32.png) [@ChristianArnault](https://www.rubin.community/u/ChristianArnault)\
**Post date:** [December 8, 2015, 2:08pm UTC](https://www.rubin.community/t/storing-knowledge-of-stack-runs-in-a-production-environment/429/1 "2015-12-08T14:08:51Z")

</div>

…

We plan to start sessions/runs of the Stack in a production environment.  
We’ll need to ensure that ANY data produced in these runs are properly tagged in reference to conditions used during these runs:

- date
- software version (Stack, …)
- origin of input data
- description of input date
- values of run parameters

Is there anythng foreseen or available in the LSST software, in terms of logging data, conventions, formats, or even database, to conventionnally store and retrieve run data?

Thanks for any suggestion

Christian Arnault

---

<div class="post-metadata">

**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\
**Post date:** [December 8, 2015, 3:44pm UTC](https://www.rubin.community/t/storing-knowledge-of-stack-runs-in-a-production-environment/429/2 "2015-12-08T15:44:07Z")

</div>

In calendar 2016 we hope to investigate and develop “workflow” solutions that would provide storage and discovery of run information (“processing metadata”) in a database such as you are asking for, but our current facilities are rather more primitive.

Our `CmdLineTask`s persist the configuration (“values of run parameters”) to the output repository. Software versions can be obtained using “`eups list --setup`” and are persisted by the `ctrl_orca` orchestration layer, if that is used (itself or with `ctrl_execute`).

([DM-3372](https://jira.lsstcorp.org/browse/DM-3372), which is not currently scheduled, may also be relevant.)

---

<div class="post-metadata">

**Author:** ![fritzm](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fritzm/32/32_2.png) [@fritzm](https://www.rubin.community/u/fritzm)\
**Post date:** [December 8, 2015, 8:00pm UTC](https://www.rubin.community/t/storing-knowledge-of-stack-runs-in-a-production-environment/429/3 "2015-12-08T20:00:49Z")

</div>

I think some of this would also relate to the provenance system that @jbecla is prototyping?

---

<div class="post-metadata">

**Author:** ![jbecla](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbecla/32/47_2.png) [@jbecla](https://www.rubin.community/u/jbecla)\
**Post date:** [December 9, 2015, 2:47am UTC](https://www.rubin.community/t/storing-knowledge-of-stack-runs-in-a-production-environment/429/4 "2015-12-09T02:47:34Z")

</div>

Yup, I’d be very interested in connecting my provenance prototype with pipelines but I heard pipeline orchestration code is not ready for that yet and it is better to wait.

---

<div class="post-metadata">

**Author:** ![srp](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/srp/32/748_2.png) [@srp](https://www.rubin.community/u/srp)\
**Post date:** [December 9, 2015, 3:23pm UTC](https://www.rubin.community/t/storing-knowledge-of-stack-runs-in-a-production-environment/429/5 "2015-12-09T15:23:55Z")

</div>

ctrl\_orca when executed through ctrl\_execute (via [runOrca.py](http://runOrca.py)) will execute “eups list --setup” and deposit it in the output directory.

The orchestration software itself used ctrl\_provenance when policy files were in use, and saved the software versions to the database (along with other things). When the switchover to pex\_config happened, orchestration switched to that as well. When the new provenance code is ready, we can get that functionality added back into Orca.