# Testing DM-4692: the new ProcessCcdTask **URL:** \ **Category:** Data Management\ **Tags:** dm-dev\ **Created:** [January 29, 2016, 5:02pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507 "2016-01-29T17:02:07Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [January 29, 2016, 5:02pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/1 "2016-01-29T17:02:08Z")
The new ProcessCcdTask is ready to be tested on real data. This topic is intended as a place for help and to record feedback and questions. I would like to emphasis the algorithms and proper values for configuration parameters here (minor code issues are probably best discussed on the ticket or as github comments). Please see [DM-4692](https://jira.lsstcorp.org/browse/DM-4692) itself for a description of how the new ProcessCcdTask works. Some settings that may be of interest when testing on real data: - `charImage.psfIterations`: number of iterations to converge on a PSF solution - `charImage.detectAndMeasure.detection.includeThresholdMultiplier` and `thresholdValue` Te product gives the # sigma for detecting sources used to fit the PSF and which appear in the `icSrc` catalog - `calibrate.detectAndMeasure.detection.includeThresholdMultiplier` and `thresholdValue` (the product gives the # sigma for detecting sources used to fit WCS and zero-point and which appear in the `Src` catalog The long-term plan is to use only one bin/processCcd.py, but for now SDSS and the version of DECam that uses output from the community pipeline will both need trivial variants that set the dataset type of the `--id` argument (e.g. for SDSS the value is `fpC`). Simply retargeting the ISR task is not sufficient. obs\_sdss has the needed code. ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [January 30, 2016, 12:39am UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/2 "2016-01-30T00:39:23Z")
Changes to be aware of: - The new code does not yet copy the “used for PSF” flags from `icSrc` to `Src` - `icSrc` has fewer measurements than it used to have; not it is just those needed to measure PSF ---
**Author:** ![hsinfang](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/hsinfang/32/1986_2.png) [@hsinfang](https://www.rubin.community/u/hsinfang)\ **Post date:** [February 2, 2016, 5:06pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/3 "2016-02-02T17:06:31Z")
Two visits of raw DECam data are processed through ISR and processCcd.py using stack b1884 and the ticket branch (built on late afternoon Jan 29). They are Strire 82 data in r band, and the sdss-dr8 astrometry\_net\_data is used. After ISR, the default configuration parameters are used. These are the check\_astrometry.png plots generated by validateDecam.py in validate\_drp. (stack b1884) ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/5b714744f6e43a7ec21ac48d2ee4954d64a716ff.png) (ticket branch DM-4692) ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/faea1a1fb191e603c4df8eb0906771c9efccd4cb.png) ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [February 4, 2016, 12:10am UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/4 "2016-02-04T00:10:21Z")
FYI: it turns out I had the deblender disabled. I have pushed a change that enables it for CalibrateTask, but not CharacterizeImageTask. Unfortunately, it seems to be overly aggressive and I have not yet figured out why. As far as I can see the settings are identical between master and DM 4692 yet the deblender on DM 4692 finds many more children and some of them cause problems such as “Input shape is singular” and runtime errors from calcmom. On known difference is that master runs astrometry and photoCal before the final deep detection and deblending pass. But both old and new fit a PSF model before deblending. ---
**Author:** ![jbosch](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbosch/32/12_2.png) [@jbosch](https://www.rubin.community/u/jbosch)\ **Post date:** [February 4, 2016, 2:17pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/5 "2016-02-04T14:17:28Z")
The difference may be appearing before we get to the deblender: an aggressive deblender could actually just be more peaks being found in detection, which could come from convolving the image by the wrong PSF (probably one too small) during detection. ---
**Author:** ![RHL](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rhl/32/25_2.png) [@RHL](https://www.rubin.community/u/RHL)\ **Post date:** [February 4, 2016, 4:55pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/6 "2016-02-04T16:55:23Z")
What do the images look like, overlain with the parents and children? ---
**Author:** ![KSK](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ksk/32/393_2.png) [@KSK](https://www.rubin.community/u/KSK)\ **Post date:** [February 4, 2016, 4:56pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/7 "2016-02-04T16:56:13Z")
@hsinfang It looks like the ticket branch is way better than the last master build. Am I reading your plots correctly? ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [February 4, 2016, 5:14pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/8 "2016-02-04T17:14:36Z")
> [@KSK](#): > > @hsinfang It looks like the ticket branch is way better than the last master build. Am I reading your plots correctly? At risk of speaking for @hsinfang I agree that DM 4692 results are better. Unfortunately that was with the deblender disabled, and as noted above, the deblender is not yet behaving well for DM 4692. Using the deblender results in a faint plot that is a mess and a bright plot that is fairly similar to master. ---
**Author:** ![hsinfang](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/hsinfang/32/1986_2.png) [@hsinfang](https://www.rubin.community/u/hsinfang)\ **Post date:** [February 4, 2016, 5:55pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/9 "2016-02-04T17:55:25Z")
Right, they are not the best comparison as the deblender was on in the first plot and off in the second. I will rebuild the ticket branch with the newer default and rerun. ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [February 9, 2016, 11:01pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/10 "2016-02-09T23:01:17Z")
I fixed two significant errors in the last few days (with thanks to @price for identifying them): - Detection was not using smoothing. This is what caused the overly aggressive deblending. - Aperture correction was being mis-measured due to using the wrong test for suitable candidate sources. This was partly caused by having not yet implemented copying of a few important fields from `icSrc` to `src` catalog. At this point I think the results from `ProcessCcdTask` should be fine. Please give it a try. ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [February 10, 2016, 4:45pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/11 "2016-02-10T16:45:21Z")
Aperture correction still needed more work. As of this morning I believe it is working. There are significant changes to `pipe_tasks` and a minor change to `obs_decam` (including allowing more cosmic rays, as one image in `validate_drp` `runDecamTest.sh` was exceeding the limit). If you get a complaint: `AttributeError: 'DetectAndMeasureConfig' object has no attribute 'starSelector'` this probably means that you have an outdated config file (indeed DetectAndMeasure no longer has its own star selector; this is part of the fix to measuring aperture correction). Running `runDecamTest.sh` the median astrometric error for bright sources has dropped from 61.7 mas (running master) to 49.0 mas. ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [February 10, 2016, 5:53pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/12 "2016-02-10T17:53:43Z")
One worrisome thing I found: the output of lsst\_dm\_stack\_demo now includes some sources whose RA/Dec are nans. (It also includes more sources than before.) ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [February 10, 2016, 6:04pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/13 "2016-02-10T18:04:30Z")
Here is another, worse: validate\_drp’s examples/runCfthTest.sh is giving worse astrometric errors, and the graphs show high errors even for bright sources, which is new and unsettling. Something is wrong. ---
**Author:** ![hsinfang](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/hsinfang/32/1986_2.png) [@hsinfang](https://www.rubin.community/u/hsinfang)\ **Post date:** [February 10, 2016, 7:16pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/14 "2016-02-10T19:16:55Z")
The following are from half of the DECam (northern CCDs only) with the same two visits using the branch and master (version of this morning) branch: ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/fd4c1a6720df4e1ececa9eb9b4cf2e6307b14027.png) master ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/061a15accf497c1c146641fafc3fbae9ebbf2422.png) ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [February 10, 2016, 7:31pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/15 "2016-02-10T19:31:44Z")
@hsinfang so the branch gives slightly worse astrometric scatter for all sources, and slightly better for bright sources. If you still have the run log available for the branch, one thing to look at is whether the branch finds a reasonable number of matches for all CCDs (this shows up near the end). I find that most of the CCDs look fine, but ccd 12 has only 50 matches, instead of over 1200. ---
**Author:** ![hsinfang](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/hsinfang/32/1986_2.png) [@hsinfang](https://www.rubin.community/u/hsinfang)\ **Post date:** [February 10, 2016, 8:08pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/16 "2016-02-10T20:08:16Z")
@rowen For the above plots I only processed the northern CCDs, so ccd 12 is not inlcude d. The matches between two visits (output from [validateDecam.py](http://validateDecam.py)) are between ~1200-1800 for all northern CCDs. ---
**Author:** ![hsinfang](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/hsinfang/32/1986_2.png) [@hsinfang](https://www.rubin.community/u/hsinfang)\ **Post date:** [February 11, 2016, 10:31pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/17 "2016-02-11T22:31:06Z")
Reran the same DECam data with the rebased branch and newer master. This time both northern and southern CCDs are included. branch ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/66b49968c314f50024926cfdc705c20434e0d29a.png) master ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/cb05fc6a0451e7de0b611b388c0fd72d82ad6848.png) ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [February 13, 2016, 7:47pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/18 "2016-02-13T19:47:41Z")
Yesterday @ctslater tracked down the problem that the output of the stack demo included sources with NaNs in positions. This was a bug in how I was copying icSrc files to src (thanks to @price and @jbosch for the solution). That is now fixed. In addition I pushed minor changes to obs\_cfht master’s config overrides for processCcd: this eliminates all remaining need to use overrides when running validate\_drp on master or DM-4692. However, until DM-5005 is implemented, avoiding those overrides requires manually emptying the override files in validate\_drp’s config directory or processing the data manually using commands such as the following: ``` processCcdDecam.py DECam/input --output=decam_saved/dm4692 --id -j 8 --logdest=dm4692_log.txt processCcd.py CFHT/input --output=cfht_saved/master --id -j 8 --logdest=master_log.txt ``` this is safer than using the examples in any case, due to the fact that the examples delete all data in DECam and Cfht before running (so those are not a safe location to save data from previous runs). These changes had no effect on the output of validate\_drp, as far as I can see. So the earlier graphs still apply and the mystery remains: why is the astrometric scatter reported by validate\_drp somewhat worse using DM-4692 than using master? This result is obtained by comparing the catalogs of two different fields. ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [February 15, 2016, 9:32pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/19 "2016-02-15T21:32:54Z")
The current results from validate\_drp are as follows: CFHT astrometric scatter (comparing two visits): - master: 9.8 mas bright sources, 59.7 mas all sources - DM-4692: 12.4 mas bright sources (20% worse), 64.3 mas all sources (7% worse) CFHT photometric scatter: - master: 13.2 mmag bright sources, 116.4 mmag all sources - DM-4692: 14.3 mmag bright sources (8% worse), 119.2 mmag all sources (3% worse) DECam astrometric scatter: - master: 35.7 mas bright sources, 72.4 mas all sources - DM-4692: 28.5 mas bright sources (20% better), 66.8 mas all sources (8% better) DECam photometric scatter: - master: 63.3 mmag bright sources, 81.2 mmag all sources - DM-4692: 69.3 mmag bright sources (9% worse), 86.0 mmag all sources (6% worse) ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/ab76d895d515e4e890da772f78824bf4a4d266a8.png) ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/b221689cc289f3857321283dec6ea98bde8e57aa.png) ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/d55f87d522b4dc1357d2f70f7b3081bbb6cfc81b.png) ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/00b39aec4ebb8dea8426f3e2fee59d74672e8ae2.png) ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [February 16, 2016, 11:14pm UTC](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507/20 "2016-02-16T23:14:31Z")
Making the astrometry task ignore deblended sources improved the astrometric fit significantly for CFHT on DM 4692; the scatter plot looks much better, though the results are still not as good as master, and there is a small noise floor for bright sources on DM 4692. ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/3d43bedb83dfdfdaa73a4d7f6e7536b3b7b04bdd.png) [Next page](https://www.rubin.community/t/testing-dm-4692-the-new-processccdtask/507.md?page=2)