# No PSF candidates survive **URL:** \ **Category:** Support\ **Tags:** dm-dev\ **Created:** [September 17, 2020, 2:07pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400 "2020-09-17T14:07:19Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![raphaelshirley](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/raphaelshirley/32/1230_2.png) [@raphaelshirley](https://www.rubin.community/u/raphaelshirley)\ **Post date:** [September 17, 2020, 2:07pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/1 "2020-09-17T14:07:19Z")
During the processCcd stage I am struggling to get any CCDs to pass without throwing the ‘No PSF candidates survive error’. Interestingly if I run on six stacked exposures I can get the stacks to find enough psf candidates. I have tried changing config for both the star selection and psf determination: ``` charImage.measurePsf.starSelector['objectSize'].example charImage.measurePsf.psfDeterminer['pca'].example ``` Star selection usually finds a handful of candidates (~10) but none of these survive. I have experimented with setting all requirements to as lenient a value as appears possible to me. Does anybody have any advice for how to change configuration options to allow more PSF candidates to survive the PSF determination stage in shallower exposures? Here is a screenshot comparison of the exposures which fail (left) and the six exposure stacks that pass (right) which have also had some further artefact removal: ![Screen Shot 2020-09-16 at 12.25.42](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/2X/a/a0bf8702f7e5cf70601c55cc8fff19446974bc69.png) The fits files are available below. You can see my current configurations here: > Many thanks, Raphael. Actual files available here (‘\_st.fit’ file is the stack): > **[Google Drive: Sign-in](https://accounts.google.com/v3/signin/identifier?continue=https:%2F%2Fdrive.google.com%2Fdrive%2Ffolders%2F1y63qfCTlsHicicCLDM7G7BBH2nBjIViC?usp%3Dsharing&followup=https:%2F%2Fdrive.google.com%2Fdrive%2Ffolders%2F1y63qfCTlsHicicCLDM7G7BBH2nBjIViC?usp%3Dsharing&osid=1&passive=1209600&service=wise&flowName=GlifWebSignIn&flowEntry=ServiceLogin&dsh=S253836124:1790276347047163)** > > Access Google Drive with a Google account (for personal use) or Google Workspace account (for business use). ---
**Author:** ![jrmullaney](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jrmullaney/32/502_2.png) [@jrmullaney](https://www.rubin.community/u/jrmullaney)\ **Post date:** [September 17, 2020, 2:52pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/2 "2020-09-17T14:52:50Z")
It looks like the psfDeterminer is throwing that error, but I expect because the selector is selecting too few to begin with. I’m no expert on PSF characterisation, but I’d worry that ~ten PSF candidates per frame wouldn’t be enough to characterise the PSF well, even if they did all survive. So, I think you want to work on the selector first to boost the numbers getting through to the PCA. Do you know how many sources are actually being detected in each frame, i.e., prior to the objectSize selection? ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [September 17, 2020, 3:04pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/3 "2020-09-17T15:04:33Z")
I don’t believe anyone has used the `pca` psfDeterminer in earnest for quite a long time. We use `psfex` (via the `meas_extensions_psfex` package) for HSC, and I believe that every other camera has followed our lead on this. There are config parameters in the starSelector that sets the minimum flux (`fluxMin`, `signalToNoiseMin`) that you should play with. But I think the root cause of the problem is that you simply don’t have a lot of suitable PSF stars to work with in the left-hand image. How would you generate a PSF model from this image if you weren’t using the LSST stack? ---
**Author:** ![raphaelshirley](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/raphaelshirley/32/1230_2.png) [@raphaelshirley](https://www.rubin.community/u/raphaelshirley)\ **Post date:** [September 18, 2020, 11:13am UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/4 "2020-09-18T11:13:44Z")
I think in the orginal pipeline they estimate the PSF from the stack. We are considering doing this also but I have been informed that due to dithering between exposures and the fact we undersample the psf this could be suboptimal. I wanted to try running on the exposures to try to quantify differences. We may well end up importing the six exposure stacks as if they were exposures but I’m nervous of what effects this will have. ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [September 18, 2020, 2:22pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/5 "2020-09-18T14:22:33Z")
If you measure the PSF from a stack, it won’t have the same fidelity as if you measured the PSF on the individual images. But if you can’t measure the PSF from an individual image because of low signal-to-noise, you’d be forced to guess at a PSF, which means you’re not going to have a good stack PSF either. I think you’re caught between a rock and a hard place, and no matter what you choose, you’re just not going to have an A1 quality PSF. So you may as well make it easy for yourself and work with the stacked images. ---
**Author:** ![jrmullaney](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jrmullaney/32/502_2.png) [@jrmullaney](https://www.rubin.community/u/jrmullaney)\ **Post date:** [September 18, 2020, 3:59pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/6 "2020-09-18T15:59:47Z")
If you did want to work with coadded images – and get the stack to coadd them, as opposed to ingest your own coadds – then you could take a look at [obs\_goto](https://github.com/GOTO-OBS/obs_goto)’s [singleVisitDriver.py](https://github.com/GOTO-OBS/obs_goto/blob/master/python/lsst/obs/goto/singleVisitDriver.py), which we wrote to do exactly that. It astrometrically registers, warps, and combines back-to-back exposures before running charImage on the result. The way it’s written, you’d need a reliable way to identify the exposures that you want to coadd. In our case, we were able to use our visit number, since a set of three back-to-back exposures were tagged with the same visit number. ---
**Author:** ![raphaelshirley](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/raphaelshirley/32/1230_2.png) [@raphaelshirley](https://www.rubin.community/u/raphaelshirley)\ **Post date:** [September 21, 2020, 2:39pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/7 "2020-09-21T14:39:40Z")
The VISTA pipeline already has the 6 exposure stacks everywhere so I can use them. Conceivably we might want to reproduce these so the singleVisitDriver might be useful thanks. I am going to try to push the Star Selector to find more sources. I have pasted the full numbers below for a typical exposure. I can increase the number selected but not get it to pass the determination stage. Thanks for your ongoing help. Raphael. ``` processCcd.charImage WARN: Source catalog detected and measured with placeholder or default PSF processCcd.charImage.detection INFO: Detected 686 positive peaks in 382 footprints and 220 negative peaks in 178 footprints to 50 sigma processCcd.charImage.detection INFO: Resubtracting the background after object detection processCcd.charImage.measurement INFO: Measuring 382 sources (382 parents, 0 children) processCcd.charImage.measurePsf INFO: Measuring PSF processCcd.charImage.measurePsf INFO: PSF star selector found 299 candidates processCcd.charImage.measurePsf.reserve INFO: Reserved 0/299 sources processCcd.charImage.measurePsf INFO: Sending 299 candidates to PSF determiner processCcd.charImage.measurePsf.psfDeterminer WARN: Please provide at least one Image for me to update: reducing number of eigen components processCcd.charImage.measurePsf.psfDeterminer WARN: Please provide at least one Image for me to update: reducing number of eigen components processCcd.charImage.measurePsf.psfDeterminer WARN: Please provide at least one Image for me to update: reducing number of eigen components processCcd FATAL: Failed on dataId={'dateObs': '2017-10-27', 'visit': 614753, 'obsNum': '00133', 'filter': 'VISTA-Y', 'hdu': 2, 'ccd': 1, 'dataType': 'BINTABLE', 'expTime': 30.0, 'taiObs': '2017-10-28T02:03:30.899', 'ccdnum': 1}: IndexError: No viable PSF candidates survive ``` ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [September 21, 2020, 2:59pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/8 "2020-09-21T14:59:02Z")
220 negative peaks sounds rather large, and comparable to the number of positive peaks, so probably you’re getting a lot of noise. I suspect your variance map is inaccurate. ---
**Author:** ![raphaelshirley](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/raphaelshirley/32/1230_2.png) [@raphaelshirley](https://www.rubin.community/u/raphaelshirley)\ **Post date:** [September 23, 2020, 10:07am UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/9 "2020-09-23T10:07:55Z")
I was under the impression that the exposures did not have a variance map. Does the processCcd stage expect one? ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [September 23, 2020, 1:44pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/10 "2020-09-23T13:44:50Z")
In regular `processCcd` use, a variance map is constructed during ISR. If you’re overriding ISR, then you need to build a variance map yourself. If you’re using the regular ISR, then you’ve probably got the gain and readnoise set incorrectly (and/or there’s a lot of covariance, which we don’t track). ---
**Author:** ![raphaelshirley](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/raphaelshirley/32/1230_2.png) [@raphaelshirley](https://www.rubin.community/u/raphaelshirley)\ **Post date:** [September 23, 2020, 2:16pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/11 "2020-09-23T14:16:55Z")
Ok thank you. I think I have switched off ISR but will look into what is happening there. I also have variance maps from the six exposure stacks but am not sure how to ingest them. Is there an example of this in any of the other obs packages? ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [September 23, 2020, 2:34pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/12 "2020-09-23T14:34:12Z")
I don’t think you can ingest the variances separately without doing some deep magic (though maybe the Gen3 butler makes it easier?). But ultimately, I think you want something similar to the [`DecamNullIsrTask`](https://github.com/lsst/obs_decam/blob/master/python/lsst/obs/decam/decamNullIsr.py), except that instead of just reading and returning the `instcal` dataset (or whatever you want to call your pre-processed data), you also set the `exposure.variance` to your known variance map. ---
**Author:** ![MelissaGraham](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/melissagraham/32/2549_2.png) [@MelissaGraham](https://www.rubin.community/u/MelissaGraham)\ **Post date:** [October 21, 2020, 8:46pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/13 "2020-10-21T20:46:44Z")
Hi @raphaelshirley, I am just following up on unsolved topics in Support. It looks like, in this case, some of the suggestions unearthed some underlying issues that have been pursued in other topics: " [The obs\_decam nullIsr task](http://www.rubin.community/t/the-obs-decam-nullisr-task/4462)" and " [Negative peaks in calexp files](http://www.rubin.community/t/negative-peaks-in-calexp-files/4479)". I was wondering if (for posterity) there are any final comments or takeaways for anyone else who might have similar issues in finding and using PSF stars in low SNR images? ---
**Author:** ![raphaelshirley](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/raphaelshirley/32/1230_2.png) [@raphaelshirley](https://www.rubin.community/u/raphaelshirley)\ **Post date:** [October 21, 2020, 9:31pm UTC](https://www.rubin.community/t/no-psf-candidates-survive/4400/14 "2020-10-21T21:31:39Z")
Hi, Yes this has been solved by fully implementing the DecamNulISR task. There is further discussion regarding how to implement it with the gen 2 Butler [here](http://www.rubin.community/t/negative-peaks-in-calexp-files/4479/2). The three aspects of the pstISRCCD (map, mask, and variance map) must be tied together and propagated past the ISR stage.