Bus error when running pipelines

Support
, ,
5

FYI, I’ve found a solution for the large memory usage in the calibrate task when processing very dense fields. The high memory usage comes from using very many (50k+) reference catalog sources in the reference matching portion of WCS fitting in AstrometryTask. There is a configurable present in the pipelines that solves this by limiting the number of reference sources used in the reference match:

tasks:
  calibrate:
    class: lsst.pipe.tasks.calibrate.CalibrateTask 
    config: 
      astrometry.matcher.maxRefObjects: 3000 # lower memory usage of matcher: limit the WCS matcher to 3000 reference objects, starting with the brightest

I am now seeing memory usage of ~2GB instead of ~30GB previously.