# Coadding DP0 calexps from different detectors for the same deepCoadd **URL:** \ **Category:** Support\ **Tags:** butler, dp0\ **Created:** [April 5, 2022, 1:16am UTC](https://www.rubin.community/t/coadding-dp0-calexps-from-different-detectors-for-the-same-deepcoadd/6436 "2022-04-05T01:16:30Z")\ **Posts on this page:** 1\ **Showing post:** 8
**Author:** ![natelust](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/natelust/32/358_2.png) [@natelust](https://www.rubin.community/u/natelust)\ **Post date:** [May 13, 2022, 5:54pm UTC](https://www.rubin.community/t/coadding-dp0-calexps-from-different-detectors-for-the-same-deepcoadd/6436/8 "2022-05-13T17:54:51Z")
Hey, I’m just getting caught up on this now and want to make sure I understand everything, and offer some reading material for background. If you are already familiar with some of what I say, I apologize I just want to make sure everyone is on the same page. In Rubin we use a `Butler` to mediate access to all data so that all the complexities of where data is located, relations between data, and what is available is abstracted away. The `Butler` is a system that has recorded all of this information inside of itself, and knows how to deliver the appropriate results (be it data, or metadata about the data), but its `Butler` repository does not contain the actual data. There can be multiple `Butler` repositories each with their own metadata, processing histories etc. When you are in python you instantiate a `Butler` object that connects with one of these repositories. You can kind of loosely think of it like a mount point on a file system can be the same directory, but different storage devices (eg different usb thumb drives) might be mounted to that location. The python `Butler` class can be pointed at different backed repositories where data may be found, but each repository is independent. Data products within a butler are organized into what are called `collections`. These are kind of logically similar to directories, but the analogy is not 1:1, and there can be multiple kinds (though that is mostly transparent, and is often an edge case). A collection contains all the inputs that go into processing data (running a `pipeline`), as well as the outputs of that processing. So say collection “A” contains `raws` (and all other appropriate inputs) that can be used as an input to run a `pipeline` that produces `calexps` that are output in a collection named “B” . Looking in collection “B” you will then see all the `calexps` as well as all the `raws` that went into processing those specific `calexps`. Collection “B” can be used as an input to running further pipelines and output into collection “c”, etc. What collection data associated with is metadata within the `Butler` repository, if you were to connect to a different `Butler` repository, you would see different data, and different collections. In that sense there really is not _one_ butler, but it is a concept that encompass a system built from all the Butler pieces created at a specific location. In the example you followed, there was a brand new `Butler` repository created at the location `./GEN3_run` that contained only the `raws` and such that were ingested into that specific `Butler` repository and contained only the `collections` created within that repository. The `Butler` repository available to delegates containing all the already processes data can be found at the url `s3://butler-us-central1-dp01`, and the collection that is associated with all the data available is named `2.2i/runs/DP0.1`. So if we want to process some data, that collection can be used as an input collection, and you can specify what output collection you want the results to be associated with. This will then be available withing the `s3://butler-us-central1-dp01` `Butler` repository. Lets break down the command line you wrote above. - The `-b` option specifies which `Butler` repo should be used in the data-processing. - `-d` is like a filter that is applied when the system tries to work out what data to run on. If there is no where filter, then they system will try to run all combinations of input data that it finds in the butler that matches the input types that the processing pipelines wants. This specific where statement is saying only process data that was taken in band ‘g’, is defined on skymap ‘DC2’ (for now lets skip over how a skymap comes in, but it is just a tessellation of the sky) and the input data must spatially overlap the spatial region defined for tract 4638. - `-i` specifies the input collection to use when deciding what data can be used as an input - `-o` specifies the name to use for the output collection to be associated with the processed data - `-p` defines the data processing pipeline to run, in this case it is only a subset of the pipeline to run (see [here](https://pipelines.lsst.io/modules/lsst.pipe.base/creating-a-pipeline.html) for more information on pipelines - `--register-dataset-types` is something that needs to be run when a `Butler` repository is created, and should not be needed or used for the DP0 butler repository. Another point of clarification is the relation to the terms used when you describe what data you want to manipulate. A `visit` is a single pointing of the telescope, it contains all the detectors in the focal plane. Coadds are built from `visits`, but are defined purely in terms of spatial terms. The sky is divided into a series of `tracts`, and each `tract` is further divided into several `patches`. A given `tract, patch` combination will bound a certain area of the sky. When we build coadds, we find all the visits where the spatial extents of the visit overlap the specified `tract,patch`. Note that a complete `visit` may not fall within a `tract,patch` as they are arbitrary pointings on the sky, and can also be arbitrarily rotated. For each of these visits we consider which detectors in that visit are contained or overlap the `tract, patch` position. Each of these visits are then resampled onto a common grid, which accounts for rotations, distortion models, etc. This is the `makeWarp` process. The coaddition process then combines each of these appropriately to form the deeper coadded image. If you restrict your coadd to only specific detectors within the camera (which is doable if this is really what you want), you greatly decrease the depth of your coadd, and in fact may not get much more than one visit for any given area withing the coadd. This is due to the fact that these `visits` again can be arbitrarily aligned with the `tract, patch` and there is no reason persay that a given detector ever line up with the same detector in a subsequent visit. And in general the better strategy is not NOT have the same detector always fall on the same part of sky, this prevents imprinting detector features into your coadd, and ensure a varied sampling of the sky such that defects and such can be rejected. So the question is, when you want an image composed of only a few detectors, what do you mean by that? Do you want to have an image of a single `visit` where the detectors are plotted in focal plane coordinates? Or do you want only a small area of the coadd? Now on to your initial question, running a specific task. There are three interfaces for this, using the command line interface as above, directly using the python task and worrying about all the i/o yourself, or using the basic python `pipetask` interface. The latter is the way I am recommending at this point until you get more familiar with the system. This interface is designed to be a simplified interface suitable for most tests, one offs, and notebook use. It does not have all the options that the command line interface has, and is never intended to. At some point in the future we will be delivering a more feature rich python API. That said this interface is nice, and takes care of a lot of the work for you, and is exactly what you want here. The documentation for`SimplePipelineExecutor` can be found [here](https://pipelines.lsst.io/v/weekly/py-api/lsst.ctrl.mpexec.SimplePipelineExecutor.html#lsst.ctrl.mpexec.SimplePipelineExecutor). A good example of using it can be found in [this](http://www.rubin.community/t/figuring-out-how-to-call-the-python-api/6432/11) community post. For this case you can ignore the `quanta` variable, after the `run` command has finished, you can load the resulting coadd from the butler (though it would be easier to not re-use the butler from the linked snippit, just re-instantiate it, pointing at the collection you used for the output) --- _[View the full topic](https://www.rubin.community/t/coadding-dp0-calexps-from-different-detectors-for-the-same-deepcoadd/6436)._