# Issues using Git LFS for cloning test data repositories **URL:** \ **Category:** Support\ **Created:** [June 23, 2016, 9:43am UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880 "2016-06-23T09:43:28Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![FabioHernandez](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fabiohernandez/32/82_2.png) [@FabioHernandez](https://www.rubin.community/u/FabioHernandez)\ **Post date:** [June 23, 2016, 9:43am UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/1 "2016-06-23T09:43:28Z")
Hello, while following the [DMTN-023 Pipeline Command-Line Drivers](https://dmtn-023.lsst.io) tutorial, I stumbled with the problem of cloning test data. I followed [these instructions](https://developer.lsst.io/tools/git_lfs.html) for installing the `git lfs` extension and for configuring my `~/.gitconfig` and `~/.git-credentials` files. The issue appears when I try to clone a test data repository: `git-lfs` keeps asking my Github password. Here is what I get: ```auto $ git lfs version git-lfs/1.2.0 (GitHub; darwin amd64; go 1.6.1; git 9bd3b8e) $ git clone https://github.com/lsst/testdata_decam.git Cloning into 'testdata_decam'... remote: Counting objects: 86, done. remote: Compressing objects: 100% (9/9), done. remote: Total 86 (delta 2), reused 0 (delta 0), pack-reused 76 Unpacking objects: 100% (86/86), done. Checking connectivity... done. Downloading rawData/calib/bpm/2013-07-16/defect01.fits (32.00 MB) Username for 'https://git-lfs.lsst.codes': airnandez Password for 'https://airnandez@git-lfs.lsst.codes': Username for 'https://git-lfs.lsst.codes': airnandez Password for 'https://airnandez@git-lfs.lsst.codes': Username for 'https://git-lfs.lsst.codes': airnandez Password for 'https://airnandez@git-lfs.lsst.codes': Username for 'https://git-lfs.lsst.codes': airnandez Password for 'https://airnandez@git-lfs.lsst.codes': Username for 'https://git-lfs.lsst.codes': ^Cwarning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry the checkout with 'git checkout -f HEAD' Exiting because of "interrupt" signal. ``` Even if I provide my correct password, `git-lfs` seems not happy with it. I checked that this is the password I use to successfully login to [github.com](http://github.com). I observe the same behaviour when trying to clone other repositories with commands such as `git clone https://github.com/lsst/ci_hsc.git`. I was wondering if some registration of my [github.com](http://github.com) account is needed in order to be able to clone LSST test data. Where should I look at to understand what is happening? The `-v` option of `git clone` does not provide further information. Execution environment: OS X 10.11.5, git version 2.6.4 (Apple Git-63) ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [June 23, 2016, 1:33pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/2 "2016-06-23T13:33:54Z")
I remember having a similar problem, but now it works as I expect. The thing is, I can’t remember what I did to fix it! Here are my relevant `.gitconfig` entries in case it helps: ``` [credential] helper = cache --timeout=3600 [credential "https://lsst-sqre-prod-git-lfs.s3-us-west-2.amazonaws.com"] helper = store [credential "https://s3.lsst.codes"] helper = store [lfs] batch = false [filter "lfs"] smudge = git-lfs smudge %f required = true clean = git-lfs clean %f ``` ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [June 23, 2016, 1:36pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/3 "2016-06-23T13:36:01Z")
Oooh! Try leaving the `Username for 'https://git-lfs.lsst.codes':` blank — no username, no password. ---
**Author:** ![FabioHernandez](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fabiohernandez/32/82_2.png) [@FabioHernandez](https://www.rubin.community/u/FabioHernandez)\ **Post date:** [June 23, 2016, 2:18pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/4 "2016-06-23T14:18:20Z")
Thanks Paul. I tried your configuration but I observe the same behaviour. It seems that your configuration should allow you authenticated read-write access to LFS users, according to [this document](https://developer.lsst.io/tools/git_lfs.html#option-1-anonymous-access-for-read-only-lfs-users). It makes sense that in that case it does not work for me because I want just read-only access to the repository and very likely my github account has not write authorisation in the repository I need to clone. I also tried entering a blank user and password, when prompted for authentication for `https://git-lfs.lsst.codes` but `git` is not happy: it keeps requesting my username and password. This is weird, since in principle for cloning the repository I was expecting [anonymous access](https://developer.lsst.io/tools/git_lfs.html#option-1-anonymous-access-for-read-only-lfs-users). Here is my configuration. Excerpts from file `~/.gitconfig`: ```auto [filter "lfs"] required = true clean = git-lfs clean -- %f smudge = git-lfs smudge -- %f # Cache anonymous access to DM Git LFS S3 servers [credential "https://lsst-sqre-prod-git-lfs.s3-us-west-2.amazonaws.com"] helper = store [credential "https://s3.lsst.codes"] helper = store # Cache anonymous access to DM Git LFS server [credential "https://git-lfs.lsst.codes"] helper = store ``` and full `~/.git-credentials`: ```auto $ cat ~/.git-credentials https://:@lsst-sqre-prod-git-lfs.s3-us-west-2.amazonaws.com https://:@s3.lsst.codes https://:@git-lfs.lsst.codes ``` ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [June 23, 2016, 3:01pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/5 "2016-06-23T15:01:16Z")
What version of git are you using? ---
**Author:** ![FabioHernandez](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fabiohernandez/32/82_2.png) [@FabioHernandez](https://www.rubin.community/u/FabioHernandez)\ **Post date:** [June 23, 2016, 3:13pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/6 "2016-06-23T15:13:00Z")
```auto $ git --version git version 2.6.4 (Apple Git-63) ``` ---
**Author:** ![j.aleksic](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/j.aleksic/32/285_2.png) [@j.aleksic](https://www.rubin.community/u/j.aleksic)\ **Post date:** [June 23, 2016, 3:38pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/7 "2016-06-23T15:38:02Z")
Hello everyone. I encountered the similar problem. I tried to install the latest release of lsst\_sims (eups distrib install lsst\_sims -t v12\_0) and the instalation of sims\_maps failed due to failed tests. Tests failed because git\_lfs was not installed. Then, I installed it from [https://developer.lsst.io/tools/git\_lfs.html](https://developer.lsst.io/tools/git_lfs.html), as suggested in the error message. After restarted sims installation, it asked me the password for git-lfs.lsst.codes. Here I stopped. ```auto > eups distrib install lsst_sims -t v12_0 ... [46/80] pex_exceptions 12.0 (already installed) done. [47/80] utils 12.0 (already installed) done. [48/80] daf_base 12.0 (already installed) done. [49/80] sims_maps 12.0 ... Password for 'https://git-lfs.lsst.codes': ``` ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [June 23, 2016, 3:39pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/8 "2016-06-23T15:39:40Z")
And what version of `git-lfs`? ``` price@price-laptop:~ $ git-lfs env git-lfs/1.0.2 (GitHub; darwin amd64; go 1.5.1) git version 2.6.3 LocalWorkingDir= LocalGitDir= LocalGitStorageDir= LocalMediaDir= TempDir=/var/folders/f_/zxjlcs790p3f2z41k6qp5_k80000gp/T/git-lfs ConcurrentTransfers=3 BatchTransfer=false GIT_PS1_SHOWDIRTYSTATE=1 GIT_PS1_SHOWUNTRACKEDFILES=1 GIT_MERGE_AUTOEDIT=no GIT_PS1_SHOWSTASHSTATE=1 GIT_PS1_SHOWUPSTREAM=auto git config filter.lfs.smudge = "git-lfs smudge %f" git config filter.lfs.clean = "git-lfs clean %f" ``` ---
**Author:** ![FabioHernandez](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fabiohernandez/32/82_2.png) [@FabioHernandez](https://www.rubin.community/u/FabioHernandez)\ **Post date:** [June 23, 2016, 3:53pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/9 "2016-06-23T15:53:16Z")
```auto $ git-lfs env git-lfs/1.2.0 (GitHub; darwin amd64; go 1.6.1; git 9bd3b8e) git version 2.6.4 (Apple Git-63) LocalWorkingDir= LocalGitDir= LocalGitStorageDir= LocalMediaDir= LocalReferenceDir= TempDir=/var/folders/0w/_0f0902d4wl0bbpr5k13hqnh0000gn/T/git-lfs ConcurrentTransfers=3 BatchTransfer=true git config filter.lfs.smudge = "git-lfs smudge -- %f" git config filter.lfs.clean = "git-lfs clean -- %f" ``` ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [June 23, 2016, 3:55pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/10 "2016-06-23T15:55:48Z")
I think the key is the following in your `~/.gitconfig` file: ``` [lfs] batch = false ``` Without it: ``` pprice@tigressdata:~/temp $ git clone https://github.com/lsst/ci_hsc.git Cloning into 'ci_hsc'... remote: Counting objects: 457, done. remote: Total 457 (delta 0), reused 0 (delta 0), pack-reused 457 Receiving objects: 100% (457/457), 74.75 KiB | 0 bytes/s, done. Resolving deltas: 100% (94/94), done. Downloading CALIB/BIAS/2013-11-03/NONE/BIAS-2013-11-03-000.fits (32.63 MB) Username for 'https://git-lfs.lsst.codes': Password for 'https://git-lfs.lsst.codes': Username for 'https://git-lfs.lsst.codes': Password for 'https://git-lfs.lsst.codes': Username for 'https://git-lfs.lsst.codes': Password for 'https://git-lfs.lsst.codes': Username for 'https://git-lfs.lsst.codes': Password for 'https://git-lfs.lsst.codes': Username for 'https://git-lfs.lsst.codes': ^Cwarning: Clone succeeded, but checkout failed. ``` With it: ``` pprice@tigressdata:~/temp $ git clone https://github.com/lsst/ci_hsc.git Cloning into 'ci_hsc'... remote: Counting objects: 457, done. remote: Total 457 (delta 0), reused 0 (delta 0), pack-reused 457 Receiving objects: 100% (457/457), 74.75 KiB | 0 bytes/s, done. Resolving deltas: 100% (94/94), done. Downloading CALIB/BIAS/2013-11-03/NONE/BIAS-2013-11-03-000.fits (32.63 MB) Username for 'https://git-lfs.lsst.codes': Password for 'https://git-lfs.lsst.codes': Downloading CALIB/BIAS/2013-11-03/NONE/BIAS-2013-11-03-001.fits (32.63 MB) Downloading CALIB/BIAS/2013-11-03/NONE/BIAS-2013-11-03-004.fits (32.63 MB) ^Cwarning: Clone succeeded, but checkout failed. ``` I did this with git-lfs 1.0.2 and haven’t confirmed with the latest version, but I think it will help. The `batch` setting is documented on the [LSST git-lfs page](https://developer.lsst.io/tools/git_lfs.html#installing-git-lfs). ---
**Author:** ![FabioHernandez](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fabiohernandez/32/82_2.png) [@FabioHernandez](https://www.rubin.community/u/FabioHernandez)\ **Post date:** [June 23, 2016, 4:07pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/11 "2016-06-23T16:07:12Z")
You are right. With ```auto [lfs] batch = false ``` the anonymous read-only access works for me also. I somehow oversaw that step which is indeed documented. Thanks for having taken time to help with this! ---
**Author:** ![jmatt](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jmatt/32/96_2.png) [@jmatt](https://www.rubin.community/u/jmatt)\ **Post date:** [June 23, 2016, 9:45pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/12 "2016-06-23T21:45:16Z")
@FabioHernandez sorry you ran into these problems. I appreciate everyones comments. They will help improve our git-lfs implementation. There are two independent issues happening here. 1) We don’t support the batch API yet. That will be implemented this summer. 2) The git-lfs client doesn’t allow for empty username and passwords. Through some configuration it is possible to work around this. But the configuration is difficult and error prone. I will investigate fixing this later this summer. I don’t know exactly how it will be solved. I expect the solution will involve some creativity or just implementing the fix in the git-lfs client and trying to convince the git-lfs developers to accept it as an optional configuration. ---
**Author:** ![mjuric](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/mjuric/32/31_2.png) [@mjuric](https://www.rubin.community/u/mjuric)\ **Post date:** [June 24, 2016, 8:07am UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/13 "2016-06-24T08:07:08Z")
Getting the `git-lfs` configurations right can be a pain, but the improvements planned for this summer should make it better. In the meantime (for those using the Anaconda distribution) you could try a package I hacked together a few weeks ago: ```auto conda install -c http://conda.lsst.mjuric.org/dev lsst-git-lfs-config ``` After running this, `git-lfs` should be fully configured for use with LSST. This will also install `git-lfs` (and `git` itself), so nothing else should be needed. ---
**Author:** ![FabioHernandez](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fabiohernandez/32/82_2.png) [@FabioHernandez](https://www.rubin.community/u/FabioHernandez)\ **Post date:** [June 24, 2016, 1:02pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/14 "2016-06-24T13:02:45Z")
> [@jmatt](#): > > @FabioHernandez sorry you ran into these problems. I appreciate everyones comments. They will help improve our git-lfs implementation. In fact, I didn’t notice the command I needed to type for setting the `lfs.batch` option to `false`, which is correctly documented in the section “Installing Git LFS”. That was the root cause of my issue. My suggestion is to slightly modify the documentation to replace the instruction: ```auto git config --global lfs.batch false ``` and to include the option: ```auto [lfs] batch = false ``` as one of the required modifications to the `~/.gitconfig` file. This way, all the configuration required for `git-lfs` to work would be concentrated in a single place. ---
**Author:** ![jsick](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jsick/32/672_2.png) [@jsick](https://www.rubin.community/u/jsick)\ **Post date:** [June 24, 2016, 10:32pm UTC](https://www.rubin.community/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/16 "2016-06-24T22:32:33Z")
Over lunch, @jmatt and I decided that in the LFS doc we’ll add ```auto [lfs] batch = false ``` to the other config blocks as a reminder in case folks miss the earlier command line block. Should solve both the usability and debugging issues.