# Fixing Astrometry Matching **URL:** \ **Category:** Data Management\ **Tags:** algorithms, astrometry\ **Created:** [October 19, 2016, 5:38pm UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249 "2016-10-19T17:38:46Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![jbosch](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbosch/32/12_2.png) [@jbosch](https://www.rubin.community/u/jbosch)\ **Post date:** [October 19, 2016, 5:38pm UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/1 "2016-10-19T17:38:46Z")
I’m wrapping up [some work](https://jira.lsstcorp.org/browse/DM-3549) that adds a new SIP distortion fitter to our codebase, and some preliminary tests by @boutigny are not as promising as we’d like. The new code might be better than our existing SIP fitter (it’s certainly cleaner and its algorithm is better motivated), but it doesn’t solve all of the problems in matching that he’s seeing in processing CFHT data. I think the fundamental problem is in the matchers; the hope of the above exercise was that better outlier-rejection in the fitter could help us to work around problems in the matcher, but I think that may have always been an unreasonable hope encouraged by the ugliness of the outlier-rejection iteration in the old one. And the _right_ solution for fixing the matcher is both simple and structurally difficult (maybe impossible) right now: we need to match over full visits, not CCDs. And that means that if we could get a matcher that works reliably - either by writing a new driver that does process full visits at once, somehow, or by writing a new CCD-level matcher - we’d be able to really clean up `meas_astrom`: - We could drop the `ANetAstrometry` and `ANetBasicAstrometry` tasks, which use `astrometry.net` to do the non-SIP part of the WCS fitting as well as the matching. Because `astrometry.net` combines those two operations, these tasks don’t fit into the same interface that other matchers and fitters use, and because they rely on the old SIP-fitting code to do the distortion part, keeping them around means we have to keep the old SIP-fitting code around (or update them to the new SIP-fitting code, which would then involve code duplication since they don’t use the same interfaces as the new fitters). On top of that, the algorithmic logic in these is horribly contorted - we run `ANetBasicAstrometry` in a context in which it thinks it’s fitting the full WCS but it’s actually fitting a temporary one using pixel positions that have already had the optical distortion removed, resulting in confusing log messages and diagnostic plots. But we can’t remove them yet, because they apparently still succeed in some cases where our default matcher fails. - We now have two SIP fitters. I believe the new one is better than the old one, but it’s hard to test that as long as we’re limited by the matchers. If we could verify that, _and_ we could remove or rewrite the SIP fitting part of the `astrometry.net` tasks, we could remove the old SIP fitter. - We’re still using `astrometry.net`-format catalogs even when we’re not using the `astrometry.net` solvers. My understanding is that the new reference catalog loaders are now ready to be used, and we’re still using `astrometry.net` simply because that’s the format most of our reference catalog datasets are currently in. Converting all of those is obviously a different issue than getting a reliable matcher, but it’d be very nice to get that done and get away from `astrometry.net` (and its installation difficulties and custom LSST Swig interface) entirely. @KSK, is it just a matter of updating those reference catalog packages, or am I unaware of additional work that needs to be done here? On top of that, someday I hope we’ll just be able to use the `jointcal` fitter for single-visit astrometry fitting as well. I don’t think its matcher will ever work in that role, however, because it doesn’t try to look for the unknown offset that’s necessary when working with approximate telescoped-provided WCSs. For now, I don’t know what to do here. This really is one of our tallest poles in terms of having a reliable pipeline (perhap mostly on CFHT, but we’ve had problems with matching on HSC in the past too). But if the solution is going to be full-visit processing, I think we may be blocked by `SuperTask` (etc) work that’s not really even been started, let alone nearing completion. I don’t think it’s viable to continue to try to solve this problem by processing individual CCDs - it just isn’t worth our time. I honestly think we’d be better off writing an alternate driver that avoids `CmdLineTask` and uses multiprocessing to process all the CCDs in a visit together, even if we’d ultimately end up throwing it away in the glorious multithreaded GIL-releasing `SuperTask` future. I have heard, however, that @ctslater may have thrown together a new matcher based on a Pan-STARRS algorithm that seemed to work better than our existing one. While I don’t want us to put a lot of work into any single-CCD matcher, even one that’s moderately far along, if it is good enough allow us to get off of `astrometry.net` and adequately test the SIP fitters, that could be a huge win. @ctslater, is that something we could easily make generally available? ---
**Author:** ![KSK](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ksk/32/393_2.png) [@KSK](https://www.rubin.community/u/KSK)\ **Post date:** [October 19, 2016, 6:47pm UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/2 "2016-10-19T18:47:17Z")
@jbosch I haven’t fully parsed your message, but I wanted to say that we have a new postdoc working on this very problem as we speak. @cmorrison is both looking at the “simple shift” matcher and also looking at the sensitivities of the OPM-B matching algorithm. We should know more in a couple of weeks. ---
**Author:** ![parejkoj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/parejkoj/32/138_2.png) [@parejkoj](https://www.rubin.community/u/parejkoj)\ **Post date:** [October 19, 2016, 8:50pm UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/3 "2016-10-19T20:50:48Z")
Yay, full-visit processing. It definitely seems like it’ll be the most robust solution for things like this. > [@jbosch](#): > > On top of that, someday I hope we’ll just be able to use the jointcal fitter for single-visit astrometry fitting as well. I don’t think its matcher will ever work in that role, however, because it doesn’t try to look for the unknown offset that’s necessary when working with approximate telescoped-provided WCSs. However, the jointcal fitter is going to want to be full-visit exposure as well, in the future. But yes, jointcal’s matcher is not really designed for searching a large probability space. > [@jbosch](#): > > I honestly think we’d be better off writing an alternate driver that avoids CmdLineTask and uses multiprocessing to process all the CCDs in a visit together, even if we’d ultimately end up throwing it away in the glorious multithreaded GIL-releasing SuperTask future. As much as I don’t like throwing away work, this may be a viable option. ---
**Author:** ![ctslater](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ctslater/32/28_2.png) [@ctslater](https://www.rubin.community/u/ctslater)\ **Post date:** [October 19, 2016, 8:57pm UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/4 "2016-10-19T20:57:54Z")
> [@jbosch](#): > > I don’t think it’s viable to continue to try to solve this problem by processing individual CCDs - it just isn’t worth our time. I’m confused by this. I understand why full focal plane fitting is conceptually superior, but I don’t know what suggests that single CCD fitting is infeasible even with improvements to the matcher. Are we talking about catastrophic matcher failures, or just that the matcher isn’t finding all the potential matches that are present in the data? Also just to be sure, I suspect @boutigny is matching against Gaia? If not, I would be interested in knowing if that changes the result. ---
**Author:** ![jbosch](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbosch/32/12_2.png) [@jbosch](https://www.rubin.community/u/jbosch)\ **Post date:** [October 19, 2016, 9:01pm UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/5 "2016-10-19T21:01:13Z")
I’m talking about catastrophic failures, in which we either don’t get enough matches to do any kind of fit, or we have so many bad matches we can’t recover from it with any kind of outlier rejection. I’m not certain that single CCD matching is infeasible in general - it just seems like a much harder problem to solve than full-visit matching, and given the _possibility_ that it’s infeasible on some datasets we care about, I’m worried about playing whack-a-mole. ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [October 19, 2016, 9:24pm UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/6 "2016-10-19T21:24:20Z")
Don’t over-estimate how easy it’s going to be to do full-visit matching. All matching algorithms require that you know the distortion. If your guess is slightly off (e.g., because working at low altitude) then you’re going to have trouble over the full focal plane simply because the domain is larger which is going to amplify any little imprecisions. In that case, the single CCD matching is going to be easier because of the limited domain, essentially approximating the distortion piecewise. Maybe a multiple-step process (e.g., CCD matching, fit entire visit, match for CCDs that originally failed to match, fit entire visit) would be better than full-visit matching. ---
**Author:** ![KSK](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ksk/32/393_2.png) [@KSK](https://www.rubin.community/u/KSK)\ **Post date:** [October 20, 2016, 12:11am UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/7 "2016-10-20T00:11:00Z")
I’ve had a bit fuller read and here are some comments. I don’t think we can give up on single CCD matching for AP quite yet. It is very clearly baked into our resourcing and pipelines for AP that we will deal exclusively in CCD level operations. Even talk of doing raft level crosstalk corrections makes people start to get a little bit antsy. So, though it may be the right thing for DRP, I think we have to spend some time on single sensor astrometry for AP. As I said before, we have a new postdoc looking at the matcher as we speak. He will both look for bugs in our implementation of OPM\_B and look at the Pan-STARRS algorithm. I hope we will see significant improvement because of his work. > [@jbosch](#): > > And that means that if we could get a matcher that works reliably - either by writing a new driver that does process full visits at once, somehow, or by writing a new CCD-level matcher - we’d be able to really clean up meas\_astrom: > > - We could drop the ANetAstrometry and ANetBasicAstrometry tasks, which use [astrometry.net](http://astrometry.net) to do the non-SIP part of the WCS fitting as well as the matching. … But we can’t remove them yet, because they apparently still succeed in some cases where our default matcher fails. The fact that [astrometry.net](http://astrometry.net) succeeds where our matcher fails means we have a bug in our matcher. I think we’ve known this for some time. I firmly believe that if we constrain ourselves to reasonable situations: i.e. \< 5 arcmin pointing error, we should do no worse than the [astrometry.net](http://astrometry.net) matcher. > [@](#): > > We now have two SIP fitters. I believe the new one is better than the old one, but it’s hard to test that as long as we’re limited by the matchers. If we could verify that, and we could remove or rewrite the SIP fitting part of the [astrometry.net](http://astrometry.net) tasks, we could remove the old SIP fitter. I think we will know more when we have fixed the single CCD matcher, but I think making the new SIP fitter the default would be a good step. > [@](#): > > We’re still using astrometry.net-format catalogs even when we’re not using the [astrometry.net](http://astrometry.net) solvers. My understanding is that the new reference catalog loaders are now ready to be used, and we’re still using [astrometry.net](http://astrometry.net) simply because that’s the format most of our reference catalog datasets are currently in. Converting all of those is obviously a different issue than getting a reliable matcher, but it’d be very nice to get that done and get away from [astrometry.net](http://astrometry.net) (and its installation difficulties and custom LSST Swig interface) entirely. @KSK, is it just a matter of updating those reference catalog packages, or am I unaware of additional work that needs to be done here? @ctslater has already produced the new version of the reference catalogs for the GAIA catalog, so has some knowledge about how to do this. I also think @rowen may have looked at doing this for the SDSS catalogs, but I’m not sure how far he got. There is no reason not to use new style reference catalogs that I know of. I hope I’m not being too optimistic about the reliability of single CCD matching, but it really does have big implications for the L1 system. ---
**Author:** ![boutigny](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/boutigny/32/26_2.png) [@boutigny](https://www.rubin.community/u/boutigny)\ **Post date:** [October 20, 2016, 10:00am UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/8 "2016-10-20T10:00:29Z")
> [@ctslater](#): > > Also just to be sure, I suspect @boutigny is matching against Gaia? If not, I would be interested in knowing if that changes the result. I haven’t been able to find the time to test with the new Gaia catalog. I will do it as soon as possible. ---
**Author:** ![boutigny](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/boutigny/32/26_2.png) [@boutigny](https://www.rubin.community/u/boutigny)\ **Post date:** [October 20, 2016, 10:14am UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/9 "2016-10-20T10:14:05Z")
While I understand the mid/long term strategy to switch to full focal plane astrometry fit, I am really concerned if we give up with the individual CCD astrometry now. Having a correct astrometry is the minimum if we want to test the science pipelines. @nchotard and I are going to spend some time in testing the matcher in order to try to understand what is going wrong. To give a sense of the current problem, here is a portion of an image (top part) where the WCS has been computed with the default matcher + the new fitter from Jim and with the SDSS DR9 reference sources superimposed. ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/05e2ab73155df38f80949b4ac11b9eecf5b17994.png) And here is the same image with the [astrometry.net](http://astrometry.net) matcher + old fitter : ![](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/1X/74e13531712d12f9f72f1f6ef691269905f687f2.png) Nevertheless, I want to stress that we should not overestimate the size of problem and in 99% of the cases the astrometry fit is giving a correct answer. This is also the reason why I think that now that we have a healthy fitter (thanks Jim), fixing the matcher should not be too complicated. Unfortunately I have very little time to work on this at the moment, things will be better after the Pittsburgh hack week. Finally I would also like to express some concerns with the ` SuperTask` if it means that we have to give up with the sequential `CmdLine` as not all computing infrastructures are sized for large multi-node tasks. If this is becoming a pre-requisite to run LSST pipelines, it needs to be planned a little bit in advance. ---
**Author:** ![KSK](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ksk/32/393_2.png) [@KSK](https://www.rubin.community/u/KSK)\ **Post date:** [October 20, 2016, 5:45pm UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/10 "2016-10-20T17:45:06Z")
@boutigny As I’ve mentioned before, @cmorrison is starting to look at this at UW now. Can you share with him the the data to reproduce this example? It’s very useful to have data that fail. ---
**Author:** ![jbosch](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbosch/32/12_2.png) [@jbosch](https://www.rubin.community/u/jbosch)\ **Post date:** [October 20, 2016, 6:24pm UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/11 "2016-10-20T18:24:39Z")
So, overall it seems I was a bit premature in proposing that we give up on single-CCD matching, and I didn’t realize there was already work underway to understand and fix our matchers. I’ve got a few more replies to specific comments I’ll try to cover together: > [@price](#): > > Don’t over-estimate how easy it’s going to be to do full-visit matching. > … > Maybe a multiple-step process (e.g., CCD matching, fit entire visit, match for CCDs that originally failed to match, fit entire visit) would be better than full-visit matching. Operationally, I think that multi-step process still amounts to full-visit matching, though I agree it may be a better algorithm that a naive match to all of the sources in a visit (and I can imagine even smarter algorithms in between those two But your point that we shouldn’t expect matching to automatically work better if we give it a larger area is a very good one. > [@KSK](#): > > I don’t think we can give up on single CCD matching for AP quite yet. It is very clearly baked into our resourcing and pipelines for AP that we will deal exclusively in CCD level operations. I have to admit I wasn’t thinking at all about AP’s CCD-parallel processing requirements at all here, mostly because I actually wasn’t thinking about LSST data at all (which I expect to be easier to solve than the heterogeneous precursor data we’re dealing with now, because we’ll have better characterizations of the optical distortion, a better reference catalog, and hopefully a better initial estimate from the observatory). I certainly understand that AP would like to avoid the complexity of full visit processing, and that’s a solid argument for putting quite a bit more work into a single-CCD matcher. If it does prove difficult, though, I think it might be worth thinking about where the complexity tradeoff would be in adding full-visit processing to AP. I suspect the added complexity mostly lands in the Task framework or workflow system in terms of supporting inter-node IPC, but DRP might need that anyway. And if our hardware has enough cores/node to process a full visit on a single node, there may actually be no tradeoff at all. > [@KSK](#): > > The fact that [astrometry.net](http://astrometry.net) succeeds where our matcher fails means we have a bug in our matcher. I think we’ve known this for some time. I firmly believe that if we constrain ourselves to reasonable situations: i.e. \< 5 arcmin pointing error, we should do no worse than the [astrometry.net](http://astrometry.net) matcher. Interesting. I honestly don’t know enough about the matcher algorithms to judge, but I did wonder whether the [astrometry.net](http://astrometry.net) algorithm might simply be a better algorithm even for small pointing error. > [@boutigny](#): > > Finally I would also like to express some concerns with the SuperTask if it means that we have to give up with the sequential CmdLine as not all computing infrastructures are sized for large multi-node tasks. If this is becoming a pre-requisite to run LSST pipelines, it needs to be planned a little bit in advance. I do think requiring bigger hardware (more memory/node) is likely inevitable for future versions of the pipeline, not just for full-visit processing but multi-band coadd processing as well. We’ll try very hard to maintain the ability to process subsets of the data (e.g. processing a subset of the focal plane as if it was the entire camera) to allow for some development and testing to happen on small machines. This is really being driven by the needs of the algorithms, not `SuperTask` - which is right now just a vague term for the future middleware that might allow the kind of processing the more sophisticated algorithms need. There’s a chance we’ll be able to preserve the ability to do full-scale processing on small machines with a lot of swapping, but I’m not convinced it’s worth our time trying to support that other than at the OS level. ---
**Author:** ![boutigny](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/boutigny/32/26_2.png) [@boutigny](https://www.rubin.community/u/boutigny)\ **Post date:** [October 21, 2016, 8:41am UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/12 "2016-10-21T08:41:08Z")
@ksk @cmorrison > [@KSK](#): > > Can you share with him the the data to reproduce this example? It’s very useful to have data that fail. The raw data file is on lsst-dev in /lsst8/boutigny/793169p.fits.fz and the problematic ccd is 33 ---
**Author:** ![RHL](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rhl/32/25_2.png) [@RHL](https://www.rubin.community/u/RHL)\ **Post date:** [October 31, 2016, 3:18pm UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/13 "2016-10-31T15:18:20Z")
I think it’d be premature to design the AP to assume that per-CCD astrometric matching always works. Although I can of course see why this is simpler in the short-run, it’s much _harder_ to retrofit something like this later. I can well believe that down in the plane after a long slew when the seeing is especially good/the transparency is especially bad/… that at least some CCDs will fail – and that we should start planning for this failure mode now. It’s not trivial, but a lot of getting robust astrometric solvers isn’t trivial and we need to face this. For example, when the seeing is very good you need a catalogue that has a fainter bright limit (or you need to work on algorithms for centroiding saturated stars). On another (but related) topic, I’m excited that UW is starting to look at matchers, and understanding exactly how and why they fail. The tools that you build to do this will be an important part of the quality analysis systems that we are building. @mjuric has clarified that such tooling is the pipeline groups’ responsibility – which means that we have to be careful not to write them as throw-away components. ---
**Author:** ![KSK](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ksk/32/393_2.png) [@KSK](https://www.rubin.community/u/KSK)\ **Post date:** [November 1, 2016, 12:05am UTC](https://www.rubin.community/t/fixing-astrometry-matching/1249/14 "2016-11-01T00:05:00Z")
> [@RHL](#): > > It’s not trivial, but a lot of getting robust astrometric solvers isn’t trivial and we need to face this. For example, when the seeing is very good you need a catalogue that has a fainter bright limit (or you need to work on algorithms for centroiding saturated stars). I very much agree that we need to work on this, I was only pushing back that we can’t throw away single chip matching/solving completely. It will be perfectly fine for much of the data we take and almost all of the precursor data we are using. It’s also a good place to start with getting a handle on the tricky parts. > [@](#): > > On another (but related) topic, I’m excited that UW is starting to look at matchers, and understanding exactly how and why they fail. The tools that you build to do this will be an important part of the quality analysis systems that we are building. @mjuric has clarified that such tooling is the pipeline groups’ responsibility – which means that we have to be careful not to write them as throw-away components. Thanks for bringing this up. I’ll make sure @cmorrison keeps this in mind as he’s doing his tests.