# Verification Datasets Meeting 2016-01-13 Minutes **URL:** \ **Category:** Data Management\ **Tags:** verificationdataset\ **Created:** [January 14, 2016, 11:57pm UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486 "2016-01-14T23:57:54Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![afausti](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/afausti/32/934_2.png) [@afausti](https://www.rubin.community/u/afausti)\ **Post date:** [January 14, 2016, 11:57pm UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486/1 "2016-01-14T23:57:54Z")
Attendees: Colin, Mario, Simon, Dominique F., Hsin-Fang, Gruendl, Juan, MWV, Angelo, David Regrets: Notetaker: Angelo **Colin:** - Measurement on difference images. Fixed bugs preventing to get measurements, now able to quantify detections as function of magnitude, etc - _Action Item:_ Include link to results in verification wiki - Tired to loop over src FITS catalogs, used daf\_ingest to set up a mysql database and running queries again lsst10 improved analysis a lot. [ingestProcessed.py](http://ingestProcessed.py) should be used for metadata, where is it? - _Action item:_ Talk with Jacek or Serge about database access. See discussion on community about the need of having the visit and ccd metadata in addition to the src table. - _Results:_ many detections of probable “noise”. Noise model is wrong, off by ~15%, CP noise model issue. Distribution of im1-im2/variance is too wide. - Plan to proceed? Try our own ISR from raw images and see how good our noise model is. MWV suggested a generic test of noise model. **David:** - Gave talk at AAS, good summary of what we have done so far ([http://zenodo.org/record/44673](http://zenodo.org/record/44673)) - COSMOS data: taking the src catalog from each ccd, matched sources between visits. Which matching method from stack should use? R: ```auto import lsst.afw.table as afwTable match = afwTable.matchRaDec(srcRef, srcVis, afwGeom.Angle(1, afwGeom.arcseconds)) ``` - src cat from individual ccds and visits, how to preserve the match result and original visit result? - Need of a multi-band matcher (in progress) - Changed strategy and focus on ProcessCdd to understand better each step and progress on QA **Michael:** - CFHT astrometry checks see `bin/check-astrometry.py` in [https://github.com/wmwv/validate\_drp](https://github.com/wmwv/validate_drp) - Median value of astrometric scatter (from the positions recorded by the pipeline) is 23 mas (mag \< 21). Checks internal astrometry between visits. - demo running check-astrometry on different output data repositories - COSMOS DECam large astrometric scatter (\>200 mas), probably CP WCS issue - generic `validate_drp` ? - Colin: good design choice, small datasets and small pieces of code - Mario: suggests to look into `pipeQA`, it has two components test (?) `pipeQA` that runs on a given data repository (it should handle any obs package) and display `pipeQA` which produce the plots. Which parts of the framework can bre reused? - _Action item:_ Mario send presentations and references on `pipeQA` - Will it work with SQuaSH? - _David:_ Simultaneous astrometry. JimB and Dominque B. talked about what’s needed to bring simulateous astrometry - UW is planing to do this in this cycle, @parejkoj is leading the effort to bring HSC code (simultaneous photometric solution) over - _Action Item:_ David look into the that first and contact @parejkoj to see how to include Dominiques code. Set up a phone conf week from Tuesday. See also [https://jira.lsstcorp.org/browse/RFC-123](https://jira.lsstcorp.org/browse/RFC-123) **Simon:** - successfuly run 3 bands coadd through DRP, results look reasonable, next ingest variable sources - updated online cookbook here: [https://github.com/DarkEnergyScienceCollaboration/Twinkles/blob/master/code/twinkles\_cookbook.md](https://github.com/DarkEnergyScienceCollaboration/Twinkles/blob/master/code/twinkles_cookbook.md) - lower floor 1% in g, 0.6% r and i - what is causing the residual systematic floor? - check aperture photometry **Angelo:** - Looking at the astrometry matcher failures, see examples at [https://confluence.lsstcorp.org/display/SQRE/Bulge+Survey+Processing](https://confluence.lsstcorp.org/display/SQRE/Bulge+Survey+Processing) - Using default configuration, maximum offset 3 arcsec, might turn that up (`calibrate.astrometry.matcher.maxMatchDistArcSec = 3.0`) - two solvers, default matcher requires you to be close while [astrometry.net](http://astrometry.net) can solve blind option - minimum matching number of pairs looks ok (`calibrate.astrometry.matcher.minMatchedPairs = 30`) **Yusra/Francisco:** - not on **Dominique:** - not on **Mario:** - MOPS - Joachim Moeyens has run MOPS on test datasets, got it run all the way through to tracks, only works on his Mac, produces garbage on linux, maybe memory issues, working on it **Domonique F.:** - introduced Juan Pablo Gomez, working on CFHT images ---
**Author:** ![KSK](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ksk/32/393_2.png) [@KSK](https://www.rubin.community/u/KSK)\ **Post date:** [January 15, 2016, 12:14am UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486/2 "2016-01-15T00:14:57Z")
> [@afausti](#): > > successfuly run 3 bands coadd through DRP, For more information, I’ve run 10 visits each of g, r, and i band simulated data through single frame processing, coadd, coadd detection and measurement, and forced photometry. These are single chips per visit with rotational dithering only. ---
**Author:** ![RHL](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rhl/32/25_2.png) [@RHL](https://www.rubin.community/u/RHL)\ **Post date:** [January 15, 2016, 1:20am UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486/3 "2016-01-15T01:20:37Z")
> two solvers, default matcher requires you to be close while [astrometry.net](http://astrometry.net) can solve blind option I Inherited the use of [astrometry.net](http://astrometry.net) in LSST. It proved to be unreliable, slow, and hard to debug. The unreliability was I think mostly due to it not liking us saturating bright stars. Dustin Lang made a number of changes and we got it mostly working on HSC. I don’t believe that we need a blind solver, hence the switch to a new solver. We’ve run it on a couple of thousand degrees of HSC data without problem (a total of 99.4 polychromatic degrees of coverage if you’re interested). If we **do** need a blind solver, I think we should do this as a whole-camera step using a smallish number of bright stars before solving for each CCD. ---
**Author:** ![swinbank](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/swinbank/32/24_2.png) [@swinbank](https://www.rubin.community/u/swinbank)\ **Post date:** [January 15, 2016, 1:31pm UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486/4 "2016-01-15T13:31:25Z")
> [@afausti](#): > > - UW is planing to do this in this cycle, J. Parecko is leading the effort to bring HSC code (simultaneous photometric solution) over The HSC tool for simultaneous astrometry and photometry is `meas_mosaic`. There is an effort I’m aware of to bring this over to LSST (that’s [DM-2674](https://jira.lsstcorp.org/browse/DM-2674)), but it’s not being led by JohnP. As far as I know, he is focusing on `meas_simastrom`, which is not HSC code, and doesn’t currently implement a simultaneous photometry (but see [DM-3871](https://jira.lsstcorp.org/browse/DM-3871)). Am I confused? ---
**Author:** ![nidever](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/nidever/32/29_2.png) [@nidever](https://www.rubin.community/u/nidever)\ **Post date:** [January 15, 2016, 7:25pm UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486/5 "2016-01-15T19:25:41Z")
I think what was said on the call is that JohnP is adding in simultaneous photometry fitting as well because HSC needs it. ---
**Author:** ![yoachim](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/yoachim/32/74_2.png) [@yoachim](https://www.rubin.community/u/yoachim)\ **Post date:** [January 15, 2016, 10:06pm UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486/6 "2016-01-15T22:06:52Z")
Under Mario’s report, I think that’s a different “Peter”. ---
**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\ **Post date:** [January 15, 2016, 10:20pm UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486/7 "2016-01-15T22:20:01Z")
> [@yoachim](#): > > I think that’s a different “Peter” Actually, I think it’s a different “Yoachim”. I will fix. ---
**Author:** ![parejkoj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/parejkoj/32/138_2.png) [@parejkoj](https://www.rubin.community/u/parejkoj)\ **Post date:** [January 16, 2016, 12:12am UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486/8 "2016-01-16T00:12:55Z")
I’m guessing the J. Parecko is supposed to be me (Parejko)? I’m looking into simastrom, which is only astrometry, not photometry (as far as I’m aware). ---
**Author:** ![afausti](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/afausti/32/934_2.png) [@afausti](https://www.rubin.community/u/afausti)\ **Post date:** [January 16, 2016, 12:19am UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486/9 "2016-01-16T00:19:23Z")
Thanks @timj for fixing and sorry for the mistakes. ---
**Author:** ![jbosch](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbosch/32/12_2.png) [@jbosch](https://www.rubin.community/u/jbosch)\ **Post date:** [January 18, 2016, 3:43pm UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486/10 "2016-01-18T15:43:08Z")
There are plans to include a photometry solution in meas\_simastrom (I’ve discussed it with @boutigny and @PierreAstier), but I don’t know what state it’s in. ---
**Author:** ![PierreAstier](https://avatars.discourse-cdn.com/v4/letter/p/e5b9ba/32.png) [@PierreAstier](https://www.rubin.community/u/PierreAstier)\ **Post date:** [January 18, 2016, 5:20pm UTC](https://www.rubin.community/t/verification-datasets-meeting-2016-01-13-minutes/486/11 "2016-01-18T17:20:13Z")
There is a photometric fit in meas\_simastrom. It consists of a minimizer (of an abstract model) and the implementation of one model: one photometric scale per input “calexp” (but one). We only miss a scheme to store the results in a way that the image stacking will use. The minimizer code could also fit a model of the “illumination correction” using a set of dithered exposures of the same scene. Again, one question here is how we store this product. This is needed for serious photometry on Megacam/CFHT images because the corrections to flatfield are important.