# Build failure for meas\_algorithms 17.0.1

**URL:** <https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692>\
**Category:** Support\
**Created:** [May 9, 2019, 12:22pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692 "2019-05-09T12:22:14Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Brnikat](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/brnikat/32/866_2.png) [@Brnikat](https://www.rubin.community/u/Brnikat)\
**Post date:** [May 9, 2019, 12:22pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/1 "2019-05-09T12:22:14Z")

</div>

I’ve had a quick rummage but can’t find a mention of the problem I’ve just encountered. Apologies if I’ve missed something relevant buried in the vast amount of information available.

My system runs a stock Ubuntu 19.04. “bash newinstall.sh -ct” doesn’t want to install binary tarballs. Disappointing, but so be it. The script ran to completion.

“eups distrib install -t v17\_0\_1 lsst\_distrib” threw up 2 errors which were easily fixed. The first run complained about the lack of cmake. “apt install cmake” did the job. Likewise, “apt install flex”. The third one has me stumped.

The a portion error report is pasted below. I can provide **much** more …

Thanks for any assistance which may get me going again.

Paul  
…/…/…/…/Linux64/esutil/0.6.2.5.lsst2+1/lib/python/esutil/htm/htmc.py:152: SystemError  
---------------------------- Captured stderr setup -----------------------------  
RuntimeError: FATAL: module compiled as little endian, but detected different endianness at runtime  
\_\_\_\_\_\_\_\_\_\_\_\_\_ ERROR at setup of HtmIndexTestCase.testLoadPixelBox \_\_\_\_\_\_\_\_\_\_\_\_\_\_  
[gw6] linux – Python 3.7.2 /usr/local/src/lsst\_stack/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/bin/python3.7

---

<div class="post-metadata">

**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\
**Post date:** [May 9, 2019, 1:24pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/2 "2019-05-09T13:24:22Z")

</div>

> [@Brnikat](#):
>
> RuntimeError: FATAL: module compiled as little endian, but detected different endianness at runtime

This is a bit odd given that you would presumably have built `esutil` from source (we don’t have ubunutu binaries). Are you building on x86\_64 or something else?

---

<div class="post-metadata">

**Author:** ![Brnikat](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/brnikat/32/866_2.png) [@Brnikat](https://www.rubin.community/u/Brnikat)\
**Post date:** [May 9, 2019, 4:15pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/3 "2019-05-09T16:15:09Z")

</div>

Thanks Tim,

I thought it was a little odd. Yes, this is a perfectly ordinary X86\_64 box running a perfectly ordinary installation of Ubuntu.

I’m guessing that esutil was built from source, though I’ve not checked. All I did wat kick off a couple of programs and then install cmake and flex which appeared to be essential. (TBH, I was mildly surprised they weren’t part of the stock Ubuntu install.)

---

<div class="post-metadata">

**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\
**Post date:** [May 9, 2019, 4:44pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/4 "2019-05-09T16:44:36Z")

</div>

You may need more than just `cmake` and `flex`. See [https://pipelines.lsst.io/install/prereqs/debian.html](https://pipelines.lsst.io/install/prereqs/debian.html)

---

<div class="post-metadata">

**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\
**Post date:** [May 9, 2019, 4:49pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/5 "2019-05-09T16:49:46Z")

</div>

can you `import esutil.htm.htmc` ?

---

<div class="post-metadata">

**Author:** ![Brnikat](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/brnikat/32/866_2.png) [@Brnikat](https://www.rubin.community/u/Brnikat)\
**Post date:** [May 9, 2019, 4:53pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/6 "2019-05-09T16:53:46Z")

</div>

Checked that and tried the apt-get command. The only package from that list which is required is libcurl4-openssl-dev and I’m a little perturbed that iraf-dev iraf-noao-dev & libcurl4-gnutls-dev will be removed. So I didn’t proceed.

---

<div class="post-metadata">

**Author:** ![Brnikat](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/brnikat/32/866_2.png) [@Brnikat](https://www.rubin.community/u/Brnikat)\
**Post date:** [May 9, 2019, 4:57pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/7 "2019-05-09T16:57:55Z")

</div>

Not without further assistance. I’m completely new to this game and I’m sure the environment hasn’t yet been set up properly. For instance:

pcl@thoth:~/Astro$ python  
Python 2.7.16 (default, Apr 9 2019, 04:50:39)  
[GCC 8.3.0] on linux2  
Type “help”, “copyright”, “credits” or “license” for more information.

> > > import esutil.htm.htmc  
> > > Traceback (most recent call last):  
> > > File “”, line 1, in   
> > > ImportError: No module named esutil.htm.htmc

or  
pcl@thoth:~/Astro$ python3  
Python 3.6.8 (default, Apr 9 2019, 04:59:38)  
[GCC 8.3.0] on linux  
Type “help”, “copyright”, “credits” or “license” for more information.

> > > import esutil.htm.htmc  
> > > Traceback (most recent call last):  
> > > File “”, line 1, in   
> > > ModuleNotFoundError: No module named ‘esutil’

which, to be honest, is exactly what I expected.

---

<div class="post-metadata">

**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\
**Post date:** [May 9, 2019, 5:11pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/8 "2019-05-09T17:11:55Z")

</div>

You need to “setup” esutil for this to work. Assuming you have got to the point where `eups distrib` is working then something like:

```auto
$ eups list esutil
   0.6.2.5.lsst2+3 b4164 b4165 b4166
$ setup -t b4166 esutil
$ python
$ import esutil.htm.htmc

```

should work. Where the build number you use in the setup is the one reported by `eups list` (it will be different to my build numbers above).

---

<div class="post-metadata">

**Author:** ![Brnikat](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/brnikat/32/866_2.png) [@Brnikat](https://www.rubin.community/u/Brnikat)\
**Post date:** [May 9, 2019, 5:20pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/9 "2019-05-09T17:20:36Z")

</div>

The import worked. The eups list command returned many strings so w\_2019\_12 was chosen at it appears to be the most recent.

The full list is

(lsst-scipipe-1172c30) # eups list esutil  
0.6.2.5.lsst2+1 sims\_w\_2018\_51 test\_tag w\_2019\_03 w\_2018\_49 sims\_w\_2019\_11 sims\_2\_13\_1 w\_2019\_10 sims\_w\_2018\_48 sims\_w\_2018\_49 sims\_w\_2019\_02 sims\_w\_2018\_50 sims\_w\_2019\_10 w\_2019\_11 w\_2018\_52 w\_2019\_09 v17\_0\_1 v17\_0 w\_2018\_43 w\_2019\_06 w\_2018\_45 sims\_w\_2019\_10\_1 w\_2019\_01 sims\_w\_2018\_47 sims\_w\_2019\_12 sims v17\_0\_rc2 w\_2018\_47 sims\_w\_2018\_45 current v17\_0\_1\_rc1 w\_2019\_12 sims\_w\_2019\_04 sims\_w\_2018\_44 sims\_w\_2019\_06 sims\_w\_2019\_05 sims\_2\_13\_0 w\_2018\_50 sims\_w\_2019\_08 o\_latest w\_2018\_46 w\_2018\_51 w\_2019\_04 sims\_w\_2019\_09 w\_2019\_07 sims\_w\_2019\_01 sims\_w\_2019\_03 sims\_w\_2018\_52 w\_2018\_44 w\_2019\_02 v17\_0\_rc1 w\_2019\_05 w\_2018\_48 w\_2019\_08 simon\_test sims\_w\_2018\_46

---

<div class="post-metadata">

**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\
**Post date:** [May 9, 2019, 5:35pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/10 "2019-05-09T17:35:15Z")

</div>

Hmm. Ok. That means we need to see more of the pytest output. I need to see the contents of the .failed file in the meas\_algorithms tests/.tests/ directory. I’m not sure it’s going to help much though since it’s not clear why Htm importing is failing.

---

<div class="post-metadata">

**Author:** ![Brnikat](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/brnikat/32/866_2.png) [@Brnikat](https://www.rubin.community/u/Brnikat)\
**Post date:** [May 9, 2019, 5:50pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/11 "2019-05-09T17:50:05Z")

</div>

Okey-doke, here you are.[pytest-meas\_algorithms.xml.failed](https://www.rubin.community/uploads/short-url/7PHGYWBKLQR31FwDXdg9moNfeYF.failed) (45.9 KB)

Hope uploading a file is preferable to pasting 757 lines of text. I don’t know the culture here but fora elsewhere prefer that approach.

---

<div class="post-metadata">

**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\
**Post date:** [May 9, 2019, 5:53pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/12 "2019-05-09T17:53:35Z")

</div>

That’s great. Turns out that every test involving htm is failing so here is one snippet:

```auto
============================= test session starts ==============================
platform linux -- Python 3.7.2, pytest-3.6.2, py-1.7.0, pluggy-0.6.0
rootdir: /usr/local/src/lsst_stack/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/meas_algorithms-17.0.1/meas_algorithms-17.0.1, inifile: setup.cfg
plugins: flake8-1.0.4, xdist-1.20.1, forked-0.2, session2file-0.1.9, cov-2.5.1, remotedata-0.3.1, openfiles-0.3.2, doctestplus-0.3.0, arraydiff-0.3
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I
gw0 [282] / gw1 [282] / gw2 [282] / gw3 [282] / gw4 [282] / gw5 [282] / gw6 [282] / gw7 [282] / gw8 [282] / gw9 [282] / gw10 [282] / gw11 [282]

scheduling tests via LoadScheduling
........................................................................................................E....E..EE....E...E..s....E.........E........ss..s..s...E......
.EE..E....E.s......................................................................................................
==================================== ERRORS ====================================
________________ERROR at setup of HtmIndexTestCase.testIngest_________________
[gw6] linux -- Python 3.7.2 /usr/local/src/lsst_stack/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/bin/python3.7

cls = <class 'test_htmIndex.HtmIndexTestCase'>

    @classmethod
    def setUpClass(cls):
        cls.outPath = tempfile.mkdtemp()
        cls.testCatPath = os.path.join(os.path.dirname(os.path.realpath( __file__ )), "data",
                                       "testHtmIndex.fits")
        # arbitrary, but reasonable, amount of proper motion (angle/year)
        # and direction of proper motion
        cls.properMotionAmt = 3.0*lsst.geom.arcseconds
        cls.properMotionDir = 45*lsst.geom.degrees
        cls.properMotionErr = 1e-3*lsst.geom.arcseconds
        cls.epoch = astropy.time.Time(58206.861330339219, scale="tai", format="mjd")
        ret = cls.make_skyCatalog(cls.outPath)
        cls.skyCatalogFile, cls.skyCatalogFileDelim, cls.skyCatalog = ret
        cls.testRas = [210., 14.5, 93., 180., 286., 0.]
        cls.testDecs = [-90., -51., -30.1, 0., 27.3, 62., 90.]
        cls.searchRadius = 3. * lsst.geom.degrees
        cls.compCats = {} # dict of center coord: list of IDs of stars within cls.searchRadius of center
        cls.depth = 4 # gives a mean area of 20 deg^2 per pixel, roughly matching a 3 deg search radius
    
        config = IndexerRegistry['HTM'].ConfigClass()
        # Match on disk comparison file
        config.depth = cls.depth
> cls.indexer = IndexerRegistry['HTM'](config)

tests/test_htmIndex.py:135: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/lsst/meas/algorithms/indexerRegistry.py:46: in makeHtmIndexer
    return HtmIndexer(depth=config.depth)
python/lsst/meas/algorithms/htmIndexer.py:36: in __init__
    self.htm = esutil.htm.HTM(depth)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <esutil.htm.htm.HTM; >, depth = 4

    def __init__ (self, depth=10):
> this = _htmc.new_HTMC(depth)
E SystemError: <built-in function new_HTMC> returned a result with an error set

../../../../Linux64/esutil/0.6.2.5.lsst2+1/lib/python/esutil/htm/htmc.py:152: SystemError
---------------------------- Captured stderr setup -----------------------------
RuntimeError: FATAL: module compiled as little endian, but detected different endianness at runtime
______ERROR at setup of HtmIndexTestCase.testLoadIndexedReferenceConfig_______

```

It seems that this line `esutil.htm.HTM(depth)` is causing the problem. Can you see if you can run that line? (maybe the import wasn’t enough). I can guess that `depth` is an integer but I’m not sure of the API.

---

<div class="post-metadata">

**Author:** ![Brnikat](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/brnikat/32/866_2.png) [@Brnikat](https://www.rubin.community/u/Brnikat)\
**Post date:** [May 9, 2019, 6:23pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/13 "2019-05-09T18:23:09Z")

</div>

A simple copy and paste told me, correctly, that depth wasn’t defined. Using an explicit 4 in the call gave this output:

> > > esutil.htm.HTM(4)  
> > > RuntimeError: FATAL: module compiled as little endian, but detected different endianness at runtime  
> > > ImportError: numpy.core.multiarray failed to import

The above exception was the direct cause of the following exception:

Traceback (most recent call last):  
File “”, line 1, in   
File “/usr/local/src/lsst\_stack/stack/miniconda3-4.5.12-1172c30/Linux64/esutil/0.6.2.5.lsst2+1/lib/python/esutil/htm/htmc.py”, line 152, in **init**  
this = \_htmc.new\_HTMC(depth)  
SystemError: returned a result with an error set

> > >

---

<div class="post-metadata">

**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\
**Post date:** [May 9, 2019, 6:26pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/14 "2019-05-09T18:26:37Z")

</div>

> [@Brnikat](#):
>
> ImportError: numpy.core.multiarray failed to import

Aha. Numpy in your conda env becomes a smoking gun?

---

<div class="post-metadata">

**Author:** ![Brnikat](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/brnikat/32/866_2.png) [@Brnikat](https://www.rubin.community/u/Brnikat)\
**Post date:** [May 10, 2019, 6:26am UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/15 "2019-05-10T06:26:27Z")

</div>

That may be significant. I seem to remember the build process discovering the system installed numpy and using it. Unfortunately that message has long since scrolled off the screen. It may yet be n a log file and I’ll go looking. Would you suggest (temporarily) uninstalling numpy and letting the miniconda build process install its own?

(Edit: removing numpy from the system is not going to happen. It’s an essential component of the local installation of [astrometry.net](http://astrometry.net) and I can’t afford to lose that.)

I confess I dislike anaconda (miniconda in this instance) and would prefer to use native tools where possible. I thought going with the recommendations would make life easier. Ah, the irony.

---

<div class="post-metadata">

**Author:** ![Brnikat](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/brnikat/32/866_2.png) [@Brnikat](https://www.rubin.community/u/Brnikat)\
**Post date:** [May 10, 2019, 12:02pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/16 "2019-05-10T12:02:05Z")

</div>

Despite the above, I just flushed the system numpy and have restarted the install. I can always re-get the Ubuntu packages when it next becomes necessary to do some plate solving.

Fingers crossed …

---

<div class="post-metadata">

**Author:** ![Brnikat](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/brnikat/32/866_2.png) [@Brnikat](https://www.rubin.community/u/Brnikat)\
**Post date:** [May 10, 2019, 4:32pm UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/17 "2019-05-10T16:32:06Z")

</div>

Hmm. That made not the slightest difference. 😢

---

<div class="post-metadata">

**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\
**Post date:** [May 11, 2019, 4:38am UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/18 "2019-05-11T04:38:51Z")

</div>

After doing some research, I find that we’ve actually seen this exact problem before in [DM-9457](https://ls.st/DM-9457) and [DM-9796](https://ls.st/DM-9796).

As the latter ticket says, this has been fixed upstream, but there hasn’t been a release yet, so we never included the fix in our stack. Your alternatives would seem to be to downgrade to gcc 7 or to manually patch esutil, since presumably you don’t want to wait for us to issue a new release.

---

<div class="post-metadata">

**Author:** ![Brnikat](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/brnikat/32/866_2.png) [@Brnikat](https://www.rubin.community/u/Brnikat)\
**Post date:** [May 11, 2019, 6:15am UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/19 "2019-05-11T06:15:38Z")

</div>

Patching holds no terrors. After a bit of digging into DM-9796 I found the patch. I’ll give it a go later today and report back.

Thanks,

---

<div class="post-metadata">

**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\
**Post date:** [May 11, 2019, 6:17am UTC](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692/20 "2019-05-11T06:17:50Z")

</div>

You will probably want to look at [https://developer.lsst.io/stack/packaging-third-party-eups-dependencies.html#testing-the-package](https://developer.lsst.io/stack/packaging-third-party-eups-dependencies.html#testing-the-package) to see how to rebuild after patching.

[Next page](https://www.rubin.community/t/build-failure-for-meas-algorithms-17-0-1/3692.md?page=2)