Hey, I’m just getting caught up on this now and want to make sure I understand everything, and offer some reading material for background. If you are already familiar with some of what I say, I apologize I just want to make sure everyone is on the same page.
In Rubin we use a Butler to mediate access to all data so that all the complexities of where data is located, relations between data, and what is available is abstracted away. The Butler is a system that has recorded all of this information inside of itself, and knows how to deliver the appropriate results (be it data, or metadata about the data), but its Butler repository does not contain the actual data. There can be multiple Butler repositories each with their own metadata, processing histories etc. When you are in python you instantiate a Butler object that connects with one of these repositories. You can kind of loosely think of it like a mount point on a file system can be the same directory, but different storage devices (eg different usb thumb drives) might be mounted to that location. The python Butler class can be pointed at different backed repositories where data may be found, but each repository is independent.
Data products within a butler are organized into what are called collections. These are kind of logically similar to directories, but the analogy is not 1:1, and there can be multiple kinds (though that is mostly transparent, and is often an edge case). A collection contains all the inputs that go into processing data (running a pipeline), as well as the outputs of that processing. So say collection “A” contains raws (and all other appropriate inputs) that can be used as an input to run a pipeline that produces calexps that are output in a collection named “B” . Looking in collection “B” you will then see all the calexps as well as all the raws that went into processing those specific calexps. Collection “B” can be used as an input to running further pipelines and output into collection “c”, etc. What collection data associated with is metadata within the Butler repository, if you were to connect to a different Butler repository, you would see different data, and different collections.
In that sense there really is not one butler, but it is a concept that encompass a system built from all the Butler pieces created at a specific location. In the example you followed, there was a brand new Butler repository created at the location ./GEN3_run that contained only the raws and such that were ingested into that specific Butler repository and contained only the collections created within that repository. The Butler repository available to delegates containing all the already processes data can be found at the url s3://butler-us-central1-dp01, and the collection that is associated with all the data available is named 2.2i/runs/DP0.1. So if we want to process some data, that collection can be used as an input collection, and you can specify what output collection you want the results to be associated with. This will then be available withing the s3://butler-us-central1-dp01 Butler repository.
Lets break down the command line you wrote above.
- The
-boption specifies whichButlerrepo should be used in the data-processing. -
-dis like a filter that is applied when the system tries to work out what data to run on. If there is no where filter, then they system will try to run all combinations of input data that it finds in the butler that matches the input types that the processing pipelines wants. This specific where statement is saying only process data that was taken in band ‘g’, is defined on skymap ‘DC2’ (for now lets skip over how a skymap comes in, but it is just a tessellation of the sky) and the input data must spatially overlap the spatial region defined for tract 4638. -
-ispecifies the input collection to use when deciding what data can be used as an input -
-ospecifies the name to use for the output collection to be associated with the processed data -
-pdefines the data processing pipeline to run, in this case it is only a subset of the pipeline to run (see here for more information on pipelines -
--register-dataset-typesis something that needs to be run when aButlerrepository is created, and should not be needed or used for the DP0 butler repository.
Another point of clarification is the relation to the terms used when you describe what data you want to manipulate. A visit is a single pointing of the telescope, it contains all the detectors in the focal plane. Coadds are built from visits, but are defined purely in terms of spatial terms. The sky is divided into a series of tracts, and each tract is further divided into several patches. A given tract, patch combination will bound a certain area of the sky.
When we build coadds, we find all the visits where the spatial extents of the visit overlap the specified tract,patch. Note that a complete visit may not fall within a tract,patch as they are arbitrary pointings on the sky, and can also be arbitrarily rotated. For each of these visits we consider which detectors in that visit are contained or overlap the tract, patch position. Each of these visits are then resampled onto a common grid, which accounts for rotations, distortion models, etc. This is the makeWarp process. The coaddition process then combines each of these appropriately to form the deeper coadded image.
If you restrict your coadd to only specific detectors within the camera (which is doable if this is really what you want), you greatly decrease the depth of your coadd, and in fact may not get much more than one visit for any given area withing the coadd. This is due to the fact that these visits again can be arbitrarily aligned with the tract, patch and there is no reason persay that a given detector ever line up with the same detector in a subsequent visit. And in general the better strategy is not NOT have the same detector always fall on the same part of sky, this prevents imprinting detector features into your coadd, and ensure a varied sampling of the sky such that defects and such can be rejected.
So the question is, when you want an image composed of only a few detectors, what do you mean by that? Do you want to have an image of a single visit where the detectors are plotted in focal plane coordinates? Or do you want only a small area of the coadd?
Now on to your initial question, running a specific task. There are three interfaces for this, using the command line interface as above, directly using the python task and worrying about all the i/o yourself, or using the basic python pipetask interface. The latter is the way I am recommending at this point until you get more familiar with the system.
This interface is designed to be a simplified interface suitable for most tests, one offs, and notebook use. It does not have all the options that the command line interface has, and is never intended to. At some point in the future we will be delivering a more feature rich python API. That said this interface is nice, and takes care of a lot of the work for you, and is exactly what you want here.
The documentation forSimplePipelineExecutor can be found here. A good example of using it can be found in this community post. For this case you can ignore the quanta variable, after the run command has finished, you can load the resulting coadd from the butler (though it would be easier to not re-use the butler from the linked snippit, just re-instantiate it, pointing at the collection you used for the output)