# Failed to ingest git-cloned HSC raw image files

**URL:** <https://www.rubin.community/t/failed-to-ingest-git-cloned-hsc-raw-image-files/3993>\
**Category:** Support\
**Created:** [January 17, 2020, 7:38am UTC](https://www.rubin.community/t/failed-to-ingest-git-cloned-hsc-raw-image-files/3993 "2020-01-17T07:38:32Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![duho](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/duho/32/1028_2.png) [@duho](https://www.rubin.community/u/duho)\
**Post date:** [January 17, 2020, 7:38am UTC](https://www.rubin.community/t/failed-to-ingest-git-cloned-hsc-raw-image-files/3993/1 "2020-01-17T07:38:32Z")

</div>

Hi all,

I was following the LSST pipeline tutorial

> **[Getting started tutorial part 1: setting up the Butler data repository — LSST...](https://pipelines.lsst.io/getting-started/data-setup.html)**

and had a problem when I tried to ingest the git-cloned HSC raw image files to Butler DATA directory with messages like

> ingest WARN: Failed to ingest file /home/lsst/mnt/testdata\_ci\_hsc/raw/HSCA90401142.fits:  
> File “src/fits.cc”, line 1427, in lsst::afw::fits::Fits::Fits(const string&, const string&, int)  
> cfitsio error: error reading from FITS file (108) : Opening file ‘/home/lsst/mnt/testdata\_ci\_hsc/raw/HSCA90401142.fits’ with mode ‘r’  
> cfitsio error stack:  
> Error reading data buffer from file:  
> /home/lsst/mnt/testdata\_ci\_hsc/raw/HSCA90401142.fits  
> ffopen could not interpret primary array header of file:  
> /home/lsst/mnt/testdata\_ci\_hsc/raw/HSCA90401142.fits  
> {0}  
> lsst::afw::fits::FitsError: 'cfitsio error: error reading from FITS file (108) : Opening file ‘/home/lsst/mnt/testdata\_ci\_hsc/raw/HSCA90401142.fits’ with mode ‘r’  
> cfitsio error stack:  
> Error reading data buffer from file:  
> /home/lsst/mnt/testdata\_ci\_hsc/raw/HSCA90401142.fits  
> ffopen could not interpret primary array header of file:  
> /home/lsst/mnt/testdata\_ci\_hsc/raw/HSCA90401142.fits  
> ’

Any comments would be appreciated, thanks!

Duho from South Korea

---

<div class="post-metadata">

**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\
**Post date:** [January 17, 2020, 1:18pm UTC](https://www.rubin.community/t/failed-to-ingest-git-cloned-hsc-raw-image-files/3993/2 "2020-01-17T13:18:30Z")

</div>

My guess is that you’ve not [set up `git-lfs`](https://pipelines.lsst.io/install/git-lfs.html). What are the contents of `/home/lsst/mnt/testdata_ci_hsc/raw/HSCA90401142.fits`?

---

<div class="post-metadata">

**Author:** ![duho](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/duho/32/1028_2.png) [@duho](https://www.rubin.community/u/duho)\
**Post date:** [January 20, 2020, 2:39am UTC](https://www.rubin.community/t/failed-to-ingest-git-cloned-hsc-raw-image-files/3993/3 "2020-01-20T02:39:25Z")

</div>

Hi Paul, Thanks for your response.  
The contents seem to be just an ascii-text 132 bytes file that reads:

> version [Git Large File Storage | Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise.](https://git-lfs.github.com/spec/v1)  
> oid sha256:00f64a7ae336ce419a9eaac7d16c58f70236736c0781616f7120d3cf7f04591d  
> size 6897600"

I followed the instruction that you linked on the course of the tutorial. Did I miss something? Thanks. - Duho

---

<div class="post-metadata">

**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\
**Post date:** [January 20, 2020, 2:54am UTC](https://www.rubin.community/t/failed-to-ingest-git-cloned-hsc-raw-image-files/3993/4 "2020-01-20T02:54:14Z")

</div>

That’s what you would expect if you haven’t properly set up `git-lfs`. Try running:

```
git lfs checkout
```

---

<div class="post-metadata">

**Author:** ![duho](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/duho/32/1028_2.png) [@duho](https://www.rubin.community/u/duho)\
**Post date:** [January 21, 2020, 2:19am UTC](https://www.rubin.community/t/failed-to-ingest-git-cloned-hsc-raw-image-files/3993/5 "2020-01-21T02:19:56Z")

</div>

Continuing the discussion from [Failed to ingest git-cloned HSC raw image files](http://www.rubin.community/t/failed-to-ingest-git-cloned-hsc-raw-image-files/3993/4):

I’m using Docker container `lsstsqre/centos:7-stack-lsst_distrib-v19_0_0` and git-lfs seems to be not properly set up in it. When I run the commend, it responds:

> git: ‘lfs’ is not a git command. See ‘git --help’.  
> Did you mean this?  
> log

I’m currently figuring out how I can set up ‘git-lfs’ in the Docker environment. Do you have any idea about that? Thank you! - Duho

---

<div class="post-metadata">

**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\
**Post date:** [January 21, 2020, 2:54am UTC](https://www.rubin.community/t/failed-to-ingest-git-cloned-hsc-raw-image-files/3993/6 "2020-01-21T02:54:22Z")

</div>

One option is to install `git-lfs` in a running container. As the container `root`:

```
yum install -y git-lfs

```

You would need to do this every time you start a container wherein you want to use `git lfs`.

If you want `git-lfs` available every time you start a container without needing an extra step, another option is to build a new Docker image over the `lsstsqre/centos:7-stack-lsst_distrib-v19_0_0` image, with a `Dockerfile` something like this (untested!):

```auto
FROM lsstsqre/centos:7-stack-lsst_distrib-v19_0_0
USER root
RUN yum -y update && yum install -y git-lfs && yum clean all -y
USER lsst

```

and build with `docker build -t duho/centos:7-stack-lsst_distrib-v19_0_0`. Then run that image instead of `lsstsqre/centos:7-stack-lsst_distrib-v19_0_0`.

Another option is to install `git-lfs` on your machine’s native system, use it to get the files you need, and then map the relevant directory into your container when you run it:

```
docker run -ti -v /path/to/your/files:/home/lsst/files lsstsqre/centos:7-stack-lsst_distrib-v19_0_0.

```

For example, I have a `bash` alias that maps `~/LSST` from my native system into the container’s `~/LSST`:

```auto
lsstDocker () {
    docker run -ti -v ~/docker:/home/lsst -v ~/LSST:/home/lsst/LSST $DOCKER_OPTIONS --cap-add=SYS_PTRACE --security-opt seccomp=unconfined lsstsqre/centos:7-stack-lsst_distrib-${1-$(lsst_latest_weekly)}
}

```

---

<div class="post-metadata">

**Author:** ![duho](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/duho/32/1028_2.png) [@duho](https://www.rubin.community/u/duho)\
**Post date:** [January 22, 2020, 5:39am UTC](https://www.rubin.community/t/failed-to-ingest-git-cloned-hsc-raw-image-files/3993/7 "2020-01-22T05:39:26Z")

</div>

I resolved this issue by following the first option! Thank you very much! I appreciate it, Paul. - Duho