# Best way, if any, to modify/undo a butler certify-calibrations call?

**URL:** <https://www.rubin.community/t/best-way-if-any-to-modify-undo-a-butler-certify-calibrations-call/7273>\
**Category:** Support\
**Created:** [December 5, 2022, 1:11am UTC](https://www.rubin.community/t/best-way-if-any-to-modify-undo-a-butler-certify-calibrations-call/7273 "2022-12-05T01:11:38Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ameisner](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ameisner/32/2392_2.png) [@ameisner](https://www.rubin.community/u/ameisner)\
**Post date:** [December 5, 2022, 1:11am UTC](https://www.rubin.community/t/best-way-if-any-to-modify-undo-a-butler-certify-calibrations-call/7273/1 "2022-12-05T01:11:38Z")

</div>

On a few occasions I’ve accidentally specified the wrong start/end date(s) when certifying a calibration. The certification commands that I use look like:

```auto
butler certify-calibrations \
$REPO DECam/calib/bias DECam/calib bias \
--begin-date 2012-01-01T00:00:00 --end-date 2020-12-31T23:59:59

```

I would like to be able to “edit” the certification start/end dates after having already run `butler certify-calibrations` one or more times. When I try to simply re-run the same above command with different start/end dates, I get an error:

```auto
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/cli/cmd/commands.py", line 421, in certify_calibrations
    script.certifyCalibrations(*args, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/script/certifyCalibrations.py", line 72, in certifyCalibrations
    registry.certify(output_collection, refs, timespan)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/core/utils.py", line 272, in inner
    return func(self, *args, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registries/sql.py", line 586, in certify
    storage.certify(collectionRecord, refsForType, timespan)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registry/datasets/byDimensions/_storage.py", line 243, in certify
    raise ConflictingDefinitionError(
lsst.daf.butler.registry._exceptions.ConflictingDefinitionError: 62 validity range conflicts certifying datasets of type bias into DECam/calib for range [2012-01-01 00:00:00.000000, 2021-12-31 23:59:59.000000).

```

I checked `butler certify-calibrations --help` but don’t see any available options along the lines of `--clobber` to override previously supplied start/end dates. Currently, the best workaround I’ve come up with is to make a new/different output collection, for instance the `DECam/calib_alt` input parameter in:

```auto
butler certify-calibrations \
$REPO DECam/calib/bias DECam/calib_alt bias \
--begin-date 2012-01-01T00:00:00 --end-date 2021-12-31T23:59:59

```

Is there a better/recommended way to redefine the certification start/end dates of a calibration without needing to create a new output collection with a different name? My question seems at least somewhat similar/related to [this one](http://www.rubin.community/t/remove-or-insert-new-certified-calibrations-from-into-a-collection/6463) from 2022 April. I am using LSST pipelines `v23_0_2` on the Rubin Science Platform. Thanks very much.

---

<div class="post-metadata">

**Author:** ![jbosch](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbosch/32/12_2.png) [@jbosch](https://www.rubin.community/u/jbosch)\
**Post date:** [December 5, 2022, 3:48pm UTC](https://www.rubin.community/t/best-way-if-any-to-modify-undo-a-butler-certify-calibrations-call/7273/2 "2022-12-05T15:48:29Z")

</div>

I think your best bet is to use the Python [`Registry.decertify`](https://pipelines.lsst.io/v/weekly/py-api/lsst.daf.butler.registry.Registry.html#lsst.daf.butler.registry.Registry.decertify) interface to undo the original certification that you want to change, and then try again with the certify command. We just don’t have a command-line interface for that yet.

---

<div class="post-metadata">

**Author:** ![ameisner](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ameisner/32/2392_2.png) [@ameisner](https://www.rubin.community/u/ameisner)\
**Post date:** [December 5, 2022, 4:24pm UTC](https://www.rubin.community/t/best-way-if-any-to-modify-undo-a-butler-certify-calibrations-call/7273/3 "2022-12-05T16:24:33Z")

</div>

Thanks! 🙂

I’ll try out `Registry.decertify` and then mark this as the solution.

---

<div class="post-metadata">

**Author:** ![ameisner](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ameisner/32/2392_2.png) [@ameisner](https://www.rubin.community/u/ameisner)\
**Post date:** [December 5, 2022, 4:59pm UTC](https://www.rubin.community/t/best-way-if-any-to-modify-undo-a-butler-certify-calibrations-call/7273/4 "2022-12-05T16:59:21Z")

</div>

I tried the following:

```auto
import lsst.daf.butler as dafButler                                                                                            
butler = dafButler.Butler('repo')

qda = butler.registry.queryDatasetAssociations
coll = "DECam/calib_alt"

biases = qda("bias", collections=coll)

butler.registry.decertify(coll, 'bias', next(biases).timespan)

```

But I end up with an error message:

```auto
  File "try_decertify_alt.py", line 9, in <module>
    butler.registry.decertify(coll, 'bias', next(biases).timespan)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/core/utils.py", line 272, in inner
    return func(self, *args, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registries/sql.py", line 601, in decertify
    storage.decertify(collectionRecord, timespan, dataIds=standardizedDataIds)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registry/datasets/byDimensions/_storage.py", line 297, in decertify
    self._db.delete(self._calibs, ["id"], *rowsToDelete)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registry/interfaces/_database.py", line 1491, in delete
    self.assertTableWriteable(table, f"Cannot delete from read-only table {table}.")
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registry/interfaces/_database.py", line 601, in assertTableWriteable
    raise ReadOnlyDatabaseError(msg)
lsst.daf.butler.registry.interfaces._database.ReadOnlyDatabaseError: Cannot delete from read-only table dataset_calibs_00000004.

```

I was also a little surprised that the `Timespan` argument for `Registry.decertify` seemed to be required, even though my read of the documentation was that it’s optional. Without the `Timespan` argument included the error I get is:

`TypeError: decertify() missing 1 required positional argument: 'timespan'`

---

<div class="post-metadata">

**Author:** ![ameisner](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ameisner/32/2392_2.png) [@ameisner](https://www.rubin.community/u/ameisner)\
**Post date:** [December 5, 2022, 5:11pm UTC](https://www.rubin.community/t/best-way-if-any-to-modify-undo-a-butler-certify-calibrations-call/7273/5 "2022-12-05T17:11:38Z")

</div>

I added `writeable=True` to my script’s Butler instantiation call, and now it appears to be running the `decertify` command/operation:

`butler = dafButler.Butler('repo', writeable=True)`

`decertify` seems to be taking a while (hasn’t finished yet as of this writing).

---

<div class="post-metadata">

**Author:** ![ameisner](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ameisner/32/2392_2.png) [@ameisner](https://www.rubin.community/u/ameisner)\
**Post date:** [December 5, 2022, 5:18pm UTC](https://www.rubin.community/t/best-way-if-any-to-modify-undo-a-butler-certify-calibrations-call/7273/6 "2022-12-05T17:18:26Z")

</div>

It ended up giving an error about the database being locked (traceback appended at end of this post). The code I’m currently using is:

```auto
import lsst.daf.butler as dafButler                                                                                            
butler = dafButler.Butler('repo', writeable=True)

qda = butler.registry.queryDatasetAssociations
coll = "DECam/calib_alt"

biases = qda("bias", collections=coll)

butler.registry.decertify(coll, 'bias', next(biases).timespan)

```

My only guess would be that this might have something to do with the `biases` generator. Will continue trying to make this work…

```auto
Traceback (most recent call last):
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1015, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 685, in do_commit
    dbapi_connection.commit()
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "try_decertify_alt.py", line 9, in <module>
    butler.registry.decertify(coll, 'bias', next(biases).timespan)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/core/utils.py", line 272, in inner
    return func(self, *args, **kwargs)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registries/sql.py", line 237, in transaction
    yield
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registry/interfaces/_database.py", line 523, in transaction
    trans.commit()
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2378, in commit
    self._do_commit()
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2568, in _do_commit
    self._connection_commit_impl()
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2539, in _connection_commit_impl
    self.connection._commit_impl()
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1017, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
    util.raise_(
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1015, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 685, in do_commit
    dbapi_connection.commit()
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
(Background on this error at: https://sqlalche.me/e/14/e3q8)

```

---

<div class="post-metadata">

**Author:** ![ameisner](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ameisner/32/2392_2.png) [@ameisner](https://www.rubin.community/u/ameisner)\
**Post date:** [December 5, 2022, 5:33pm UTC](https://www.rubin.community/t/best-way-if-any-to-modify-undo-a-butler-certify-calibrations-call/7273/7 "2022-12-05T17:33:33Z")

</div>

The following worked, having pre-written a pickle file with the desired `Timespan` object in it so as to avoid the database being locked:

```auto
import lsst.daf.butler as dafButler                                                                                            
butler = dafButler.Butler('repo', writeable=True)
import pickle

with open('timespan.pickle', 'rb') as f:
    timespan = pickle.load(f)

coll = "DECam/calib_alt"

butler.registry.decertify(coll, 'bias', timespan)

```

---

<div class="post-metadata">

**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\
**Post date:** [December 6, 2022, 3:58am UTC](https://www.rubin.community/t/best-way-if-any-to-modify-undo-a-butler-certify-calibrations-call/7273/8 "2022-12-06T03:58:33Z")

</div>

It would also work to turn your query result into a list or set. This would terminate the database query, allowing your subsequent modification to go through.

---

<div class="post-metadata">

**Author:** ![ameisner](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ameisner/32/2392_2.png) [@ameisner](https://www.rubin.community/u/ameisner)\
**Post date:** [December 6, 2022, 5:26am UTC](https://www.rubin.community/t/best-way-if-any-to-modify-undo-a-butler-certify-calibrations-call/7273/9 "2022-12-06T05:26:16Z")

</div>

Thanks, K-T! I just tried the list version of your suggestion and it worked:

```auto
import lsst.daf.butler as dafButler                                                                                            
butler = dafButler.Butler('repo', writeable=True)

qda = butler.registry.queryDatasetAssociations
coll = "DECam/calib_alt"

biases = list(qda("bias", collections=coll))

butler.registry.decertify(coll, 'bias', biases[0].timespan)

```