# 'Dimension mismatch' issue of constructFlat.py in hscPipe v7.0

**URL:** <https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676>\
**Category:** Support\
**Created:** [April 24, 2019, 11:30pm UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676 "2019-04-24T23:30:05Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Charles](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@Charles](https://www.rubin.community/u/Charles)\
**Post date:** [April 24, 2019, 11:30pm UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/1 "2019-04-24T23:30:05Z")

</div>

Hi,

I am testing the new release of `hsePipe v7.0`, for the HSC data taken in this month. But it came out a error of “dimension mismatch” and “invalid value encountered in true\_divide  
flatness = (skyLevels[good] - skyMedian) / skyMedian” in near the end of process. Does anyone know what might help here?

This is the command and parameters I executed:

> `constructFlat.py $WORKDIR --rerun $CALVRERUN --calib=$WORKDIR/CALIB --id visit=172414..172434:2 --config isr.doBias=False isr.doDark=False isr.doFringe=False isr.doAttachTransmissionCurve=False --job $CALVRERUN-g --time 300 --clobber-config --cores 40`

I also tested the same command with the flat taken at 2018-01-10 (135866..135884:2). I still got a similar errors as previous test . Note that this flat data (2018-01-10) could be processed by v5.4, but not later version.

The following text is the error message:

> 118310 INFO 2019-04-25T07:28:51.720+0800 flat: Writing {‘ccd’: 110, ‘filter’: ‘HSC-G’, ‘calibDate’: ‘2019-04-04’} on ichiro.asiaa.sinica.edu.tw:118310  
> 118298 INFO 2019-04-25T07:28:56.444+0800 flat: Writing {‘ccd’: 107, ‘filter’: ‘HSC-G’, ‘calibDate’: ‘2019-04-04’} on ichiro.asiaa.sinica.edu.tw:118298  
> afw.cameraGeom.utils.makeImageFromCamera ERROR: Unable to fit image for detector “0\_31” into image of camera:  
> File “src/image/Image.cc”, line 183, in void lsst::afw::image::ImageBase::assign(const lsst::afw::image::ImageBase&, const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin) [with PixelT = float]  
> Dimension mismatch: 65x32 v. 32x65 {0}  
> lsst::pex::exceptions::LengthError: ‘Dimension mismatch: 65x32 v. 32x65’  
> afw.cameraGeom.utils.makeImageFromCamera ERROR: Unable to fit image for detector “1\_35” into image of camera:  
> File “src/image/Image.cc”, line 183, in void lsst::afw::image::ImageBase::assign(const lsst::afw::image::ImageBase&, const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin) [with PixelT = float]  
> Dimension mismatch: 65x32 v. 32x65 {0}  
> lsst::pex::exceptions::LengthError: ‘Dimension mismatch: 65x32 v. 32x65’  
> afw.cameraGeom.utils.makeImageFromCamera ERROR: Unable to fit image for detector “0\_35” into image of camera:  
> File “src/image/Image.cc”, line 183, in void lsst::afw::image::ImageBase::assign(const lsst::afw::image::ImageBase&, const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin) [with PixelT = float]  
> Dimension mismatch: 65x32 v. 32x65 {0}  
> lsst::pex::exceptions::LengthError: ‘Dimension mismatch: 65x32 v. 32x65’  
> afw.cameraGeom.utils.makeImageFromCamera ERROR: Unable to fit image for detector “1\_31” into image of camera:  
> File “src/image/Image.cc”, line 183, in void lsst::afw::image::ImageBase::assign(const lsst::afw::image::ImageBase&, const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin) [with PixelT = float]  
> Dimension mismatch: 65x32 v. 32x65 {0}  
> lsst::pex::exceptions::LengthError: ‘Dimension mismatch: 65x32 v. 32x65’  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe\_drivers/7.0-hsc/python/lsst/pipe/drivers/constructCalibs.py:1076: RuntimeWarning: divide by zero encountered in log  
> bgMatrix = np.log(bgMatrix)  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe\_drivers/7.0-hsc/python/lsst/pipe/drivers/constructCalibs.py:1083: UserWarning: Warning: converting a masked element to nan.  
> compScales = np.array([(bgMatrix[i1, :] - expScales).mean() for i1 in range(numCcds)])  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe\_drivers/7.0-hsc/python/lsst/pipe/drivers/constructCalibs.py:145: RuntimeWarning: divide by zero encountered in double\_scalars  
> combined \*= finalScale / background  
> /opt/hscPipe7/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3118: RuntimeWarning: Mean of empty slice.  
> out=out, \*\*kwargs)  
> /opt/hscPipe7/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/numpy/core/\_methods.py:85: RuntimeWarning: invalid value encountered in true\_divide  
> ret = ret.dtype.type(ret / rcount)  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.0-hsc/python/lsst/ip/isr/isrTask.py:2053: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian

---

<div class="post-metadata">

**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\
**Post date:** [April 25, 2019, 2:05am UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/2 "2019-04-25T02:05:16Z")

</div>

That looks like a non-fatal error due to a [recently-introduced bug](https://jira.lsstcorp.org/browse/DM-19371) in the camera overview construction code. Could you please confirm that the individual CCD images have been created successfully?

---

<div class="post-metadata">

**Author:** ![laurenam](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/laurenam/32/128_2.png) [@laurenam](https://www.rubin.community/u/laurenam)\
**Post date:** [April 25, 2019, 3:07am UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/3 "2019-04-25T03:07:52Z")

</div>

I’m still confused as to where/what the bug actually is, but is it possible that there needs to be a

callback = cgUtils.rawCallback if dtype == “raw” else None

type of clause to the imageSource in visualizeVisit’s makeCameraImage() function (I’m thinking the flat construction is starting from “raw” data, so needs the obeyNQuarter clause that was moved to the rawCallback in DM-19371)? Again, kind of shooting at straws…

---

<div class="post-metadata">

**Author:** ![Charles](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@Charles](https://www.rubin.community/u/Charles)\
**Post date:** [April 25, 2019, 11:12am UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/4 "2019-04-25T11:12:50Z")

</div>

What do yo mean “individual CCD images”?  
I don’t see the file created in $WORKDIR/CALIB/FLAT  
, but I do see the 112 files created in $WORKDIR/rerun/ssp-201904-pi-hp7-flat/FLAT/2018-01-10/HSC-G/FLAT-2018-01-10-HSC\*.fits  
I remembered that --calib=$WORKDIR/CALIB should be the default path for all FLAT?

I also tried v7.1 today, but it has same error message.

Charles

---

<div class="post-metadata">

**Author:** ![Charles](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@Charles](https://www.rubin.community/u/Charles)\
**Post date:** [April 29, 2019, 3:31am UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/5 "2019-04-29T03:31:20Z")

</div>

Is there a solution for this issue now?

---

<div class="post-metadata">

**Author:** ![jbosch](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbosch/32/12_2.png) [@jbosch](https://www.rubin.community/u/jbosch)\
**Post date:** [May 9, 2019, 5:03pm UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/6 "2019-05-09T17:03:48Z")

</div>

I believe this has now been fixed on master of `pipe_drivers` (at least it seems like a case of [DM-19412](https://jira.lsstcorp.org/browse/DM-19412)), and should land in the next LSST weekly (w\_2018\_19).

I believe it should be included in hscPipe 7.3 as well.

---

<div class="post-metadata">

**Author:** ![Charles](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@Charles](https://www.rubin.community/u/Charles)\
**Post date:** [May 13, 2019, 2:24am UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/7 "2019-05-13T02:24:29Z")

</div>

Our system administrator noticed that the latest version of `pipe_drivers` is `7.0-hsc+1` but the dependence of hscPipe v7.3 is linked to `pipe_drivers 7.0-hsc`. And I also got the same error message when I running the same process with v7.3. Is there still a issue in the codes?

---

<div class="post-metadata">

**Author:** ![jbosch](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbosch/32/12_2.png) [@jbosch](https://www.rubin.community/u/jbosch)\
**Post date:** [May 13, 2019, 2:42am UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/8 "2019-05-13T02:42:35Z")

</div>

I suspect you may be running into the problem noted in the email announcing the 7.3 release:

> I messed up building the previous releases, in that they were not built in the same stack, which means the version numbering of products with changed dependencies is wrong (the “+1” versions weren’t created). For this reason, it’s necessary to install this release in a completely fresh stack from versions 7.0 through 7.2. Sorry about that!

---

<div class="post-metadata">

**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\
**Post date:** [May 20, 2019, 6:52pm UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/9 "2019-05-20T18:52:13Z")

</div>

This got fixed in hscPipe 7.4.

---

<div class="post-metadata">

**Author:** ![Charles](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@Charles](https://www.rubin.community/u/Charles)\
**Post date:** [May 21, 2019, 2:29am UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/10 "2019-05-21T02:29:51Z")

</div>

I still got following error message from the process by hscPipe 7.4. It (divide by zero encountered in log) looks like the second part of error message of my original post.

> 50850 INFO 2019-05-21T10:09:34.503+0800 flat: Writing {‘ccd’: 99, ‘filter’: ‘HSC-G’, ‘calibDate’: ‘2019-04-04’} on ichiro.asiaa.sinica.edu.tw:50850  
> 50877 INFO 2019-05-21T10:09:47.485+0800 flat: Writing {‘ccd’: 109, ‘filter’: ‘HSC-G’, ‘calibDate’: ‘2019-04-04’} on ichiro.asiaa.sinica.edu.tw:50877  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe\_drivers/7.4-hsc/python/lsst/pipe/drivers/constructCalibs.py:1076: RuntimeWarning: divide by zero encountered in log  
> bgMatrix = np.log(bgMatrix)  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe\_drivers/7.4-hsc/python/lsst/pipe/drivers/constructCalibs.py:1083: UserWarning: Warning: converting a masked element to nan.  
> compScales = np.array([(bgMatrix[i1, :] - expScales).mean() for i1 in range(numCcds)])  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe\_drivers/7.4-hsc/python/lsst/pipe/drivers/constructCalibs.py:145: RuntimeWarning: divide by zero encountered in double\_scalars  
> combined \*= finalScale / background  
> /opt/hscPipe7/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3118: RuntimeWarning: Mean of empty slice.  
> out=out, \*\*kwargs)  
> /opt/hscPipe7/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/numpy/core/\_methods.py:85: RuntimeWarning: invalid value encountered in true\_divide  
> ret = ret.dtype.type(ret / rcount)  
> /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip\_isr/7.1-hsc/python/lsst/ip/isr/isrTask.py:2069: RuntimeWarning: invalid value encountered in true\_divide  
> flatness = (skyLevels[good] - skyMedian) / skyMedian

---

<div class="post-metadata">

**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\
**Post date:** [May 21, 2019, 4:06am UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/11 "2019-05-21T04:06:55Z")

</div>

Those are not errors, but warnings. They might be indicative of something bad going on, but much more likely it’s just `numpy` being a bit too chatty. It looks worse than it is, because it appears to hold them over until the end and dump a bunch all at once. My guess is that they’re coming from `ccd=9`, which is dead.

---

<div class="post-metadata">

**Author:** ![Charles](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@Charles](https://www.rubin.community/u/Charles)\
**Post date:** [May 21, 2019, 5:12am UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/12 "2019-05-21T05:12:28Z")

</div>

Your guess is about right, I think.  
The FLAT files on $WORKDIR/rerun/ssp-201904-pi-hp7-flat/FLAT/201\*/HSC-\*/FLAT-\*.fits with only ~1.1MB (others are ~21MB) are FLAT\*009.fits. The pixel values of FLAT\*009.fits are all NAN. The values of raw HSC-\*009.fits are all “blank” on ds9.

---

<div class="post-metadata">

**Author:** ![Charles](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@Charles](https://www.rubin.community/u/Charles)\
**Post date:** [May 28, 2019, 8:20am UTC](https://www.rubin.community/t/dimension-mismatch-issue-of-constructflat-py-in-hscpipe-v7-0/3676/13 "2019-05-28T08:20:13Z")

</div>

Is there a fix for that in next release?