# The obs\_decam nullIsr task **URL:** \ **Category:** Data Management\ **Created:** [October 9, 2020, 3:17pm UTC](https://www.rubin.community/t/the-obs-decam-nullisr-task/4462 "2020-10-09T15:17:17Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![raphaelshirley](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/raphaelshirley/32/1230_2.png) [@raphaelshirley](https://www.rubin.community/u/raphaelshirley)\ **Post date:** [October 9, 2020, 3:17pm UTC](https://www.rubin.community/t/the-obs-decam-nullisr-task/4462/1 "2020-10-09T15:17:17Z")
Hi, We have been developing an [obs package](https://github.com/lsst-uk/obs_vista) to process VISTA imaging with the LSST stack. A big question for us has been whether to ingest the community 6 exposure stacks (ISR already performed, deep enough to get good photocal matches and measure ‘PSF’) or the direct exposures (harder to measure PSF because of depth but measurement done as expected by the stack on single exposures, requires a lot of effort to rewrite ISR etc). We have been told that if we use the stacks then we need to ingest the variance images made by the VISTA (Cambridge Astronomical Survey Unit - CASU) pipeline. I have therefore been looking at the ‘instcal’ ingestion and [nullISR](https://github.com/lsst/obs_decam/blob/d4cf91f659ebaa28f5a7f66c615b24e4ab1396f1/python/lsst/obs/decam/decamNullIsr.py) task in [obs\_decam](https://github.com/lsst/obs_decam). I can see how the variance and masks are being ingested but as far as I can see the nullIsr task simply puts the ISR ‘instcal’ images after the ISR stage. I don’t see how it is propagating the ‘wtmap’ which has been ingested. What our obs package is currently doing is simply pretending the stacks are exposures and letting it estimate the variance image. Is there a different processCcd task developed somewhere other than the main GitHub repo? What has been the general experience with the obs\_decam and instcal images? Would those with experience recommend using the community pipeline images or is it always better to use the LSST exactly as designed on raw images. Could I see a full example of running the pipeline with instcal images? Many thanks for everyone’s help over various questions, Raphael. ---
**Author:** ![jrmullaney](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jrmullaney/32/502_2.png) [@jrmullaney](https://www.rubin.community/u/jrmullaney)\ **Post date:** [October 9, 2020, 4:13pm UTC](https://www.rubin.community/t/the-obs-decam-nullisr-task/4462/2 "2020-10-09T16:13:01Z")
It seems that what obs\_decam does is “retarget” the nullIsrTask if one wants to use the instcal images (see step 6 in their README file). Retargeting is where one writes a separate bit of code in your python directory, which then overrides a default step. The instruction to retarget is provided in the config file for that task (so, you’d put the snippet of code in step 6 of the readme in config/isr.py). There are quite a number of retargets in obs\_goto (plus I expect most other obs packages), if you want to see some more examples or retargeting. ---
**Author:** ![raphaelshirley](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/raphaelshirley/32/1230_2.png) [@raphaelshirley](https://www.rubin.community/u/raphaelshirley)\ **Post date:** [October 9, 2020, 4:38pm UTC](https://www.rubin.community/t/the-obs-decam-nullisr-task/4462/3 "2020-10-09T16:38:07Z")
I see that in retargeting isr to the null isr task all that happens is that it propagates the instcal image to “postISRCCD” stage which presumably inputs to the subsequent stage of processCcd but I don’t see how that incorporates the wtmap. Presumably if we skip all of isr we would have to also produce other products from isr (in particular the variance). I’m also not entirely sure what the “postISRCCD” product is since it isn’t specified in the mapper. I’m also not sure how to run this with processCcd since it needs an id to select the images for processing can I simply add the retarget to the isr config and then run the processCcd command line tasks with the instcal images as inputs. If so why ingest the wtmap and dqmask products into the Butler repo? Or are they exactly the products generated by ISR normally? And if so do I need to make a wtmap and dqmask plane in the format expected by the stack? ---
**Author:** ![mrawls](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/mrawls/32/2810_2.png) [@mrawls](https://www.rubin.community/u/mrawls)\ **Post date:** [October 9, 2020, 5:03pm UTC](https://www.rubin.community/t/the-obs-decam-nullisr-task/4462/4 "2020-10-09T17:03:59Z")
In general I would _not_ recommend modeling anything after how obs\_decam handles instcals. The Data Management team is moving away from that with a strong preference for ingesting raw images and calibrations, building our own master-calibs, running ISR, etc. I do not think anything pertaining to how instcals presently work is guaranteed to continue working with the Gen 3 middleware. For what it’s worth, the current Gen 2 `processCcd` runs ISR, image characterization (background measurement), and photometric and astrometric calibration. `postISRCCD` is what comes out of ISR, `icExp` is what comes out of background characterization, and `calexp` is what comes out after photometric and astrometric calibration. The new Gen 3 way of doing this will be defining a `pipetask` which runs each of these tasks in succession (plus whatever other tasks you want!) rather than the monolithic `processCcd`. ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [October 9, 2020, 5:38pm UTC](https://www.rubin.community/t/the-obs-decam-nullisr-task/4462/5 "2020-10-09T17:38:34Z")
The output from the `IsrTask` is an `lsst.afw.image.Exposure` object (with image, mask and variance planes) where counts on the image (roughly) corresponds to astronomical flux. Usually it reads a `raw` image using the butler, applies various corrections, and sets the mask and variance plane appropriately for CCDs. Your `VistaIsrTask` (to which you’ll `retarget` the default `isr`) needs to have the same result. How you achieve that result is up to you, but you need to respect the interface. I don’t know what the Vista pipeline provides so I can’t make a hard recommendation, but here are some possibilities: - Mask plane: - Looking for magic values in the image plane (e.g., `NaN` or `-999`); - Using a bad pixel mask from the Vista pipeline. - Variance plane: - Estimate using metadata (sky brightness, gain, read noise, number of exposures, integration time); - Measure the noise in the background (e.g., with `lsst.afw.math.makeStatistics`); - Using a variance (or noise) image from the Vista pipeline. If you want to read something from the Vista pipeline in order to set the mask or variance in your `VistaIsrTask`, you have to decide whether or not to use the LSST data butler. We think that the data butler is generally helpful, but perhaps it’s an annoying abstraction for you right now. If you don’t use the LSST data butler, you may have more trouble moving up to the Gen3 middleware when that becomes ready, but it might let you get things done quicker right now if you know the file paths to use and read things directly, perhaps using `afw` primitives or `astropy.io.fits` or some other I/O code, so long as you’re putting everything into an `lsst.afw.image.Exposure` object in the end. I don’t think it’s too hard to use the data butler (though perhaps I’m too familiar with to it to be unbiased). All you need to do to get started is to add an entry to your obs package’s `policy/WhateverMapper.yaml` file, e.g., [for `instcal` in `DecamMapper`](https://github.com/lsst/obs_decam/blob/a79ddab4cbea29ff40bb1c72418a714651cccb38/policy/DecamMapper.yaml#L40-L46). The trick is that the filename `template` is within the butler’s data repo, so the files need to live in the butler’s data repo, with some programmatic directory structure; so you need to get the files in the right place. If you have a suitable layout for your files already, then you can just `cp -r` the files into the data repo and you’re done. Otherwise, you’re going to need to create the layout. The `ingestImages.py` script will not work for this (it’s designed to work on `raw` files, not supplementary files). You could hack that up to install additional files alongside the `raw` images (that’s [what we do for PFS](https://github.com/Subaru-PFS/obs_pfs/blob/6ad9f1c3b5372b045724f90d025b898075d0b03a/python/lsst/obs/pfs/ingest.py#L447-L517)), but possibly the easiest solution would be to write your own script which will install the additional files in the right place in the butler data repo. ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [October 9, 2020, 5:57pm UTC](https://www.rubin.community/t/the-obs-decam-nullisr-task/4462/6 "2020-10-09T17:57:24Z")
I just took a look at the [obs\_decam ingest code](https://github.com/lsst/obs_decam/blob/master/python/lsst/obs/decam/ingest.py), and it looks a lot more sophisticated than I had thought. Something like that might work for you. The `dqmask` and `wtmap` are being integrated into the `instcal` dataset by [some butler magic](https://github.com/lsst/obs_decam/blob/master/python/lsst/obs/decam/decamMapper.py#L205-L234) rather than being done in their `NullIsrTask`. I think this is a matter of personal preference (I would have done the construction in their `NullIsrTask`, but doing it the way it’s done just has different pros and cons).