# Completely/automatically deleting portions of a run's outputs with butler prune-datasets **URL:** \ **Category:** Support\ **Created:** [October 3, 2022, 7:11am UTC](https://www.rubin.community/t/completely-automatically-deleting-portions-of-a-runs-outputs-with-butler-prune-datasets/7093 "2022-10-03T07:11:23Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ameisner](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ameisner/32/2392_2.png) [@ameisner](https://www.rubin.community/u/ameisner)\ **Post date:** [October 3, 2022, 7:11am UTC](https://www.rubin.community/t/completely-automatically-deleting-portions-of-a-runs-outputs-with-butler-prune-datasets/7093/1 "2022-10-03T07:11:23Z")
I would like to completely delete all contents within the `postISRCCD` and `icExp` subdirectories of an output run (without resorting to `rm -rf`, unless that’s recommended…). I am using `v23_0_1` of the LSST pipelines. The following two commands appear to delete all _files_ within the `icExp` and `postISRCCD` directories: ```auto butler prune-datasets $REPO --purge DECam/runs/prune_test/20220926T234050Z --datasets icExp DECam/runs/prune_test/20220926T234050Z butler prune-datasets $REPO --purge DECam/runs/prune_test/20220926T234050Z --datasets postISRCCD DECam/runs/prune_test/20220926T234050Z ``` But many empty directories are left behind: > $find repo/DECam/runs/prune\_test/20220926T234050Z/postISRCCD -type f |wc -l > 0 > $find repo/DECam/runs/prune\_test/20220926T234050Z/postISRCCD -type d |wc -l > 34 > $find repo/DECam/runs/prune\_test/20220926T234050Z/icExp -type f |wc -l > 0 > $find repo/DECam/runs/prune\_test/20220926T234050Z/icExp -type d |wc -l > 50 In this run there are only 25 CCD’s worth of outputs, so the numbers of empty directories left behind aren’t huge. But I want to soon process millions of CCD’s, in which case it seems that I’d be left with O(10 million) inodes consumed by empty directories within `icExp` and `postISRCCD`. What is the recommended way to get rid of these directories, in addition to the files that they once contained? Also, a related but different question which maybe should be its own separate forum topic: is there a way to embed `butler prune-datasets` directly into my YAML-defined pipeline? I think that’d be preferable to running my pipeline and then running separate `butler prune-datasets` commands after the fact. I did a brief/superficial search for any instances of “prune” within all of the YAML files in our v23\_0\_1 installation but came up empty: > $ find lsst\_stack\_v23\_0\_1/stack -name “\*.yaml” |wc -l > 4523 > $ find lsst\_stack\_v23\_0\_1/stack -name “\*.yaml” -exec grep -i prune {} \; |wc -l > 0 Thanks very much. ---
**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\ **Post date:** [October 3, 2022, 3:12pm UTC](https://www.rubin.community/t/completely-automatically-deleting-portions-of-a-runs-outputs-with-butler-prune-datasets/7093/2 "2022-10-03T15:12:51Z")
> [@ameisner](#): > > without resorting to `rm -rf`, unless that’s recommended Deleting datasets using `rm` is a bad idea because both registry and datastore will still think the datasets exist. > [@ameisner](#): > > But many empty directories are left behind: This is a deliberate choice at the moment. Deleting files is a safe operation for a system where many processes can be accessing butler simultaneously. Deleting a directory is a lot more problematic because in theory it’s entirely possible that another process trying to write a dataset will crash with a race condition if it tries to use the directory that just got deleted. For a timestamp directory this is probably fine but things get more and more dangerous the higher up the directory tree you go. We haven’t really resolved how to deal with this. If you are in a single-user butler environment who can guarantee that no other butler accesses are ongoing then after your call to `prune datasets` you can clean up the directories. > [@ameisner](#): > > embed `butler prune-datasets` directly into my YAML-defined pipeline No. Pipelines can never delete datasets. See my answer on the other topic. When we run batch processing (With bps) we don’t use registry at all, we create a throwaway registry and then transfer datasets back on batch completion. In theory you can modify the transfer job to not include the intermediates (having configured a chained datastore where your intermediates go to scratch disk). ---
**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\ **Post date:** [October 3, 2022, 5:10pm UTC](https://www.rubin.community/t/completely-automatically-deleting-portions-of-a-runs-outputs-with-butler-prune-datasets/7093/3 "2022-10-03T17:10:11Z")
It might be reasonable to have a manual `--remove-empty-directories` option on `prune-datasets` for times when you know that nothing else is ever going to write to them. ---
**Author:** ![ameisner](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ameisner/32/2392_2.png) [@ameisner](https://www.rubin.community/u/ameisner)\ **Post date:** [October 3, 2022, 9:02pm UTC](https://www.rubin.community/t/completely-automatically-deleting-portions-of-a-runs-outputs-with-butler-prune-datasets/7093/4 "2022-10-03T21:02:04Z")
Thanks again, Tim! I really appreciate the responses/answers. ---
**Author:** ![ameisner](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ameisner/32/2392_2.png) [@ameisner](https://www.rubin.community/u/ameisner)\ **Post date:** [October 3, 2022, 9:03pm UTC](https://www.rubin.community/t/completely-automatically-deleting-portions-of-a-runs-outputs-with-butler-prune-datasets/7093/5 "2022-10-03T21:03:27Z")
Thanks, K-T! This idea sounds good to me as an end user who doesn’t have to worry about broader design principles or other implications…