# Correct use of imageDifference.py **URL:** \ **Category:** Support\ **Created:** [January 28, 2020, 11:47pm UTC](https://www.rubin.community/t/correct-use-of-imagedifference-py/4013 "2020-01-28T23:47:13Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![fraserw](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fraserw/32/877_2.png) [@fraserw](https://www.rubin.community/u/fraserw)\ **Post date:** [January 28, 2020, 11:47pm UTC](https://www.rubin.community/t/correct-use-of-imagedifference-py/4013/1 "2020-01-28T23:47:13Z")
Hi Community. Trying to get imageDifference.py to subtract a merged template from the warps used to make said template. The command I am trying to execute is (full output copied below): > imageDifference.py DATA\_CFHT --rerun coadd:deepCoadd --id filter=r2 tract=0 patch=0,0 extension=1 visit=2426913 --templateId filter=r2 tract=0 patch=0,0 --clobber-config --config doSelectSources=False kernelSourcesFromRef=True which crashes out eventually with the error: > imageDifference FATAL: Failed on dataId={‘filter’: ‘r2’, ‘tract’: ‘0’, ‘patch’: ‘0,0’, ‘extension’: 1, ‘visit’: 2426913, ‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘ccd’: 0, ‘state’: ‘p’, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}: RuntimeError: Template is not defined for the deepDiff\_diaSrc dataset type, it must be set before it can be used. Images are from Megacam, and processed successfully with processCcd, makeDiscreteSkyMap, makeCoaddTEmpExp, and assembleCoadds. All processed images and warps are in their original butler locations. Probably just a mistake with how I am calling imageDifference. Anyone know the solution? Thanks Full imageDifference output: > CameraMapper INFO: Loading exposure registry from /media/LSSTproc/DATA\_CFHT/registry.sqlite3 > > root WARN: Unexpected ID tract; guessing type is “str” > > root WARN: Unexpected ID patch; guessing type is “str” > > root INFO: Running: /home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe\_tasks/19.0.0+2/bin/imageDifference.py DATA\_CFHT --rerun coadd:deepCoadd --id filter=r2 tract=0 patch=0,0 extension=1 visit=2426913 --templateId filter=r2 tract=0 patch=0,0 --clobber-config --config doSelectSources=False kernelSourcesFromRef=True > > WARNING: You are using OpenBLAS with multiple threads (4), but have not > > specified the number of threads using one of the OpenBLAS environment variables: > > OPENBLAS\_NUM\_THREADS, GOTO\_NUM\_THREADS, OMP\_NUM\_THREADS. > > This may indicate that you are unintentionally using multiple threads, which may > > cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know > > what you are doing and want threads enabled implicitly, set the environment > > variable LSST\_ALLOW\_IMPLICIT\_THREADS. > > imageDifference INFO: Processing {‘filter’: ‘r2’, ‘tract’: ‘0’, ‘patch’: ‘0,0’, ‘extension’: 1, ‘visit’: 2426913, ‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘ccd’: 0, ‘state’: ‘p’, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112} > > imageDifference.getTemplate INFO: Using skyMap tract 0 > > imageDifference.getTemplate INFO: Assembling 2 coadd patches > > imageDifference.getTemplate INFO: exposure dimensions=(2048, 4612); coadd dimensions=(1188, 2568) > > imageDifference.getTemplate INFO: Reading patch {‘datasetType’: ‘deepCoadd\_sub’, ‘bbox’: Box2I(minimum=Point2I(2330, 10914), dimensions=Extent2I(1188, 1186)), ‘tract’: 0, ‘patch’: ‘0,2’, ‘numSubfilters’: 3} > > imageDifference.getTemplate INFO: Reading patch {‘datasetType’: ‘deepCoadd\_sub’, ‘bbox’: Box2I(minimum=Point2I(2330, 11900), dimensions=Extent2I(1188, 1582)), ‘tract’: 0, ‘patch’: ‘0,3’, ‘numSubfilters’: 3} > > imageDifference INFO: Subtracting images > > imageDifference.subtract INFO: Template Wcs : 5.066344,-0.378546 -\> 5.064267,-0.374407 > > imageDifference.subtract INFO: Science Wcs : 5.064279,-0.374418 -\> 5.066335,-0.378531 > > imageDifference.subtract INFO: Astrometrically registering template to science image > > imageDifference.subtract INFO: templateFwhmPix: 3.8609133057836065 > > imageDifference.subtract INFO: scienceFwhmPix: 4.421904801366045 > > ip.diffim.generateAlardLuptonBasisList INFO: Reference psf fwhm is the greater, normal convolution mode > > imageDifference.subtract.selectDetection INFO: Detected 7661 positive peaks in 2662 footprints to 10 sigma > > imageDifference.subtract.selectMeasurement INFO: Measuring 2662 sources (2662 parents, 0 children) > > imageDifference.subtract INFO: Growing 2662 kernel candidate stars by 21 pixels > > imageDifference.subtract INFO: Selected 2219 / 2662 sources for KernelCandidacy > > imageDifference.subtract INFO: Matching Psf FWHM 3.86 -\> 4.42 pix > > ip.diffim.generateAlardLuptonBasisList INFO: Reference psf fwhm is the greater, normal convolution mode > > imageDifference.subtract INFO: Final spatial kernel sum 228.711 > > imageDifference.subtract INFO: Spatial model condition number 4.283e+07 > > imageDifference.subtract INFO: Doing stats of kernel candidates used in the spatial fit. > > imageDifference.subtract INFO: 2219 candidates total, 826 rejected, 1100 used > > imageDifference.subtract INFO: Spatial kernel model well constrained; 1100 candidates, 3 terms, 27 bases > > imageDifference.subtract INFO: Spatial background model appears well constrained; 1100 candidates, 6 terms > > imageDifference INFO: Computing diffim PSF > > imageDifference INFO: Running diaSource detection > > imageDifference.detection INFO: Detected 5461 positive peaks in 1899 footprints and 5727 negative peaks in 2187 footprints to 5.5 sigma > > imageDifference INFO: Merging detections into 1658 sources > > imageDifference INFO: Running diaSource measurement: newDipoleFitting=True > > imageDifference.measurement INFO: Measuring 1658 sources (1658 parents, 0 children) > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545015: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545016: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545017: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545021: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545048: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545094: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545167: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545187: edge failure > > /home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/ip\_diffim/19.0.0+2/python/lsst/ip/diffim/dipoleFitTask.py:871: RuntimeWarning: divide by zero encountered in double\_scalars > > signalToNoise = np.sqrt((fluxVal/fluxVar)\*\*2 + (fluxValNeg/fluxVarNeg)\*\*2) > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545305: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545374: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545431: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545464: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545511: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545541: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545551: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545596: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545649: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545654: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545772: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545881: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545908: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546012: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546099: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546145: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546153: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546236: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546270: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546324: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546331: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546333: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546393: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546397: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546419: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546444: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546450: edge failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546538: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546577: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546588: dipole fit failure > > /home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/ip\_diffim/19.0.0+2/python/lsst/ip/diffim/dipoleFitTask.py:871: RuntimeWarning: divide by zero encountered in double\_scalars > > signalToNoise = np.sqrt((fluxVal/fluxVar)\*\*2 + (fluxValNeg/fluxVarNeg)\*\*2) > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546629: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546637: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546641: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546648: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546652: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546659: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546662: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546665: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546667: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546668: dipole fit failure > > ip\_diffim\_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546671: dipole fit failure > > imageDifference.forcedMeasurement INFO: Performing forced measurement on 1658 sources > > imageDifference FATAL: Failed on dataId={‘filter’: ‘r2’, ‘tract’: ‘0’, ‘patch’: ‘0,0’, ‘extension’: 1, ‘visit’: 2426913, ‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘ccd’: 0, ‘state’: ‘p’, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}: RuntimeError: Template is not defined for the deepDiff\_diaSrc dataset type, it must be set before it can be used. > > Traceback (most recent call last): > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe\_base/19.0.0/python/lsst/pipe/base/cmdLineTask.py”, line 388, in **call** > > result = self.runTask(task, dataRef, kwargs) > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe\_base/19.0.0/python/lsst/pipe/base/cmdLineTask.py”, line 447, in runTask > > return task.runDataRef(dataRef, \*\*kwargs) > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe\_base/19.0.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper > > res = func(self, \*args, \*\*keyArgs) > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe\_tasks/19.0.0+2/python/lsst/pipe/tasks/imageDifference.py”, line 719, in runDataRef > > sensorRef.put(diaSources, self.config.coaddName + “Diff\_diaSrc”) > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf\_persistence/19.0.0/python/lsst/daf/persistence/butlerSubset.py”, line 224, in put > > self.butlerSubset.butler.put(obj, datasetType, self.dataId, doBackup=doBackup, \*\*rest) > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf\_persistence/19.0.0/python/lsst/daf/persistence/butler.py”, line 1421, in put > > locations = self.\_locate(datasetType, dataId, write=True) > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf\_persistence/19.0.0/python/lsst/daf/persistence/butler.py”, line 1297, in \_locate > > location = repoData.repo.map(datasetType, dataId, write=write) > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf\_persistence/19.0.0/python/lsst/daf/persistence/repository.py”, line 242, in map > > loc = self.\_mapper.map(\*args, \*\*kwargs) > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf\_persistence/19.0.0/python/lsst/daf/persistence/mapper.py”, line 166, in map > > return func(self.validate(dataId), write) > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs\_base/19.0.0/python/lsst/obs/base/cameraMapper.py”, line 387, in mapClosure > > return mapping.map(mapper, dataId, write) > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs\_base/19.0.0/python/lsst/obs/base/mapping.py”, line 155, in map > > path = mapper.\_mapActualToPath(self.template, actualId) > > File “/home/ubuntu/lsst\_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs\_base/19.0.0/python/lsst/obs/base/mapping.py”, line 132, in template > > “it must be set before it can be used.”) > > RuntimeError: Template is not defined for the deepDiff\_diaSrc dataset type, it must be set before it can be used. > > imageDifference WARN: Could not persist metadata for dataId={‘filter’: ‘r2’, ‘tract’: ‘0’, ‘patch’: ‘0,0’, ‘extension’: 1, ‘visit’: 2426913, ‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘ccd’: 0, ‘state’: ‘p’, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}: Template is not defined for the deepDiff\_metadata dataset type, it must be set before it can be used. ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [January 29, 2020, 12:24am UTC](https://www.rubin.community/t/correct-use-of-imagedifference-py/4013/2 "2020-01-29T00:24:28Z")
I expect this is a shortcoming in obs\_cfht. We need to add `deepDiff_diaSrc` to [`obs_cfht/policy/MegacamMapper.yaml`](https://github.com/lsst/obs_cfht/blob/master/policy/MegacamMapper.yaml). ---
**Author:** ![fraserw](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fraserw/32/877_2.png) [@fraserw](https://www.rubin.community/u/fraserw)\ **Post date:** [January 29, 2020, 5:48pm UTC](https://www.rubin.community/t/correct-use-of-imagedifference-py/4013/3 "2020-01-29T17:48:37Z")
Thanks for the response @price . For the sake of a project under time pressure, I will stick with using montage, and look again at this in a month or two. Thanks! ---
**Author:** ![gkovacs](https://avatars.discourse-cdn.com/v4/letter/g/9fc348/32.png) [@gkovacs](https://www.rubin.community/u/gkovacs)\ **Post date:** [February 5, 2020, 2:35am UTC](https://www.rubin.community/t/correct-use-of-imagedifference-py/4013/4 "2020-02-05T02:35:11Z")
@fraserw I’m afraid subtraction of two skymap warped coadd/template is not supported in image differencing at all. With getTemplate==GetCoaddAsTemplateTask (default), the following data handling happens in _imageDifference.py_: - _ImageDifferenceTask_ gets the _calexp_ with the --id specified from the Butler as the _science exposure_. I guess in your case, tract= and patch= numbers are ignored and only visit=, filter, date, ccd, extension, etc. may be used. The _templateId_ option is ignored completely. - The butler picks the (_first_ available) template (skymap) in the repository. _The closest_ tract of the skymap is selected and the numbers of all overlapping patches within that tract. - Based on the science exposure (calexp) sky coordinates, and the tract WCS, image coordinates of a _template exposure_ is determined, an empty exposure is created. (`exposure dimensions=(2048, 4612); coadd dimensions=(1188, 2568)`) - Each existing coadd with the selected patch numbers are loaded and clipped into the template exposure. In other words, you have no command line control on which patches are loaded and used as template. (`Reading patch...`) - The template exposure is warped into the calexp then PSF matched and subtracted from the science calexp exposure. - The difference image is written out with the same dataId as the science calexp. I’m actually surprised that this is the first step to fail, I think because of the tract= patch= keys. The _templateId_ option is only used when _2 calexps_ are subtracted (getTemplate == GetCalexpAsTemplateTask), where it can contain keys (typically visit=) to _update_ the dataId of the science exposure (–id) to get the dataId of the _template calexp_. ---
**Author:** ![fraserw](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fraserw/32/877_2.png) [@fraserw](https://www.rubin.community/u/fraserw)\ **Post date:** [April 29, 2020, 7:29pm UTC](https://www.rubin.community/t/correct-use-of-imagedifference-py/4013/5 "2020-04-29T19:29:05Z")
Hi @gkovacs Thanks for the description. Just getting back to this again. I haven’t been confidently able to parse what you described. From your fifth bullet, it sounds like imageDifference.py can remove a template image from a calexp image. Great. I should be able to project the subtracted images onto a rectalinear grid after the subtraction. From what you describe, to control the image subtractoin, I need to make sure that only 1 template image exists in the butler. Right?