# Setting in-repository policy via Butler

**URL:** <https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268>\
**Category:** Data Management\
**Created:** [October 24, 2016, 11:55pm UTC](https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268 "2016-10-24T23:55:46Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![natepease](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/natepease/32/2086_2.png) [@natepease](https://www.rubin.community/u/natepease)\
**Post date:** [October 24, 2016, 11:55pm UTC](https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268/1 "2016-10-24T23:55:46Z")

</div>

I’m working on creating API to allow scripts to add Policy to butler output repositories. (in DM-7777)

The ‘new’ butler API allows a repository-arguments structure to be passed to butler when loading input repositories and when creating output repositories. I’m thinking of adding a parameter to that structure that accepts a dict (or Butler.Policy instance, which extends dict) for that purpose. The new policy would get written into the repository, used while the repository is getting used as an output, and loaded when the repository is loaded as an input.

Setting the in-repository policy should only be allowed on output repositories (input repositories should not be modified, generally speaking).

The only API change would be adding a keyword to lsst.daf.persistence.RepositoryArgs:

```
class RepositoryArgs(object):
    def __init__ (..., policy=None):
        ...

```

so that then a policy can be created in a script e.g.:

```
newPolicy = {
    'datasets': {
        'testDataset': {
            'python': 'lsst.daf.persistence.test.TestObject',
            'template': 'basic/id%(id)s.pickle',
            'storage': 'PickleStorage'
        }
    }
}

```

and passed to butler:

```
repoArgs = lsst.daf.persistence.RepositoryArgs(root='path/to/output/repo', policy=newPolicy)
butler = lsst.daf.persistence.butler(inputs='path/to/inputs', outputs=repoArgs)

```

People that are interested in adding policy to repositories - does this work for you? Do you have other ideas, or requirements that I’ve overlooked?

---

<div class="post-metadata">

**Author:** ![natepease](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/natepease/32/2086_2.png) [@natepease](https://www.rubin.community/u/natepease)\
**Post date:** [October 24, 2016, 11:56pm UTC](https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268/2 "2016-10-24T23:56:26Z")

</div>

I’d specifically like +/- feedback from @ktl on this.

---

<div class="post-metadata">

**Author:** ![RHL](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rhl/32/25_2.png) [@RHL](https://www.rubin.community/u/RHL)\
**Post date:** [October 25, 2016, 12:09am UTC](https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268/3 "2016-10-25T00:09:00Z")

</div>

I’m a bit confused. As a naïve user, does this change the way that I create a butler? I currently only need to know the root to read or write it.

---

<div class="post-metadata">

**Author:** ![natepease](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/natepease/32/2086_2.png) [@natepease](https://www.rubin.community/u/natepease)\
**Post date:** [October 25, 2016, 12:26am UTC](https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268/4 "2016-10-25T00:26:25Z")

</div>

It will not change the way you create a butler (unless you want to add Policy details to the repository, which it sounds like you do not.)

---

<div class="post-metadata">

**Author:** ![RHL](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rhl/32/25_2.png) [@RHL](https://www.rubin.community/u/RHL)\
**Post date:** [October 25, 2016, 12:37am UTC](https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268/5 "2016-10-25T00:37:50Z")

</div>

So if someone else has configured the Policy details I’ll still get them? Sorry to be dense, but I don’t understand all the changes you are making.

---

<div class="post-metadata">

**Author:** ![natepease](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/natepease/32/2086_2.png) [@natepease](https://www.rubin.community/u/natepease)\
**Post date:** [October 25, 2016, 12:49am UTC](https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268/6 "2016-10-25T00:49:03Z")

</div>

You’re not being dense. Thank you for considering it.

> [@RHL](#):
>
> So if someone else has configured the Policy details I’ll still get them?

The short answer is yes.

Slightly longer: when ‘someone else’ creates the repository (as an output repository) and adds new policy details during butler init, that policy will get written to the repo. Later, when you use that repo as an input repository, butler will load that in-repo policy that it finds in the repo.

The new butler api has not been seen much in the wild yet. (I don’t really know how to push it out & get people using it). But you can read more about it at [LDM-463: Data Access Design](https://ldm-463.lsst.io/v/draft/)

---

<div class="post-metadata">

**Author:** ![jbosch](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbosch/32/12_2.png) [@jbosch](https://www.rubin.community/u/jbosch)\
**Post date:** [October 25, 2016, 2:18pm UTC](https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268/7 "2016-10-25T14:18:47Z")

</div>

For other Science Pipelines people wondering how this affects them, I _think_ (confirmation from someone knows welcome) the idea is that `(Super)Task`s will define their output data products in these per-repository policies rather than in the per-camera policy files we use now (or even the base-class policy file @pgee has been moving definitions to recently). Until that’s possible, I don’t think we have a direct use for this feature.

---

<div class="post-metadata">

**Author:** ![RHL](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rhl/32/25_2.png) [@RHL](https://www.rubin.community/u/RHL)\
**Post date:** [October 25, 2016, 6:05pm UTC](https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268/8 "2016-10-25T18:05:49Z")

</div>

Don’t we still need a common set of definitions even if the outputs are per-repository? I’m expecting that HSC and DECam data will look basically identical to the analysis scripts.

If not, it’s even more important that @jalt arranges the data sets at NCSA so that all datasets from a given camera share a common root that the butler points at.

---

<div class="post-metadata">

**Author:** ![RHL](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rhl/32/25_2.png) [@RHL](https://www.rubin.community/u/RHL)\
**Post date:** [October 25, 2016, 6:30pm UTC](https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268/9 "2016-10-25T18:30:02Z")

</div>

I think it’d be really helpful to have a relatively short document describing (or at least mentioning) the features that are in the butler that the `classic’ butler didn’t have.

---

<div class="post-metadata">

**Author:** ![natepease](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/natepease/32/2086_2.png) [@natepease](https://www.rubin.community/u/natepease)\
**Post date:** [October 25, 2016, 9:12pm UTC](https://www.rubin.community/t/setting-in-repository-policy-via-butler/1268/10 "2016-10-25T21:12:28Z")

</div>

> [@RHL](#):
>
> I think it’d be really helpful to have a relatively short document describing (or at least mentioning) the features that are in the butler that the `classic’ butler didn’t have.

That sounds reasonable. I created [https://jira.lsstcorp.org/browse/DM-8080](https://jira.lsstcorp.org/browse/DM-8080), probably I’ll do this by adding a section to LDM-463.