# How should we store 100 GB + data sets? Is git-lfs still the answer? **URL:** \ **Category:** Data Management\ **Created:** [April 27, 2016, 2:12pm UTC](https://www.rubin.community/t/how-should-we-store-100-gb-data-sets-is-git-lfs-still-the-answer/724 "2016-04-27T14:12:18Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![mwv](https://avatars.discourse-cdn.com/v4/letter/m/e79b87/32.png) [@mwv](https://www.rubin.community/u/mwv)\ **Post date:** [April 27, 2016, 2:12pm UTC](https://www.rubin.community/t/how-should-we-store-100-gb-data-sets-is-git-lfs-still-the-answer/724/1 "2016-04-27T14:12:18Z")
@jmatt How should we store external data sets that are on order 100 GB? Is `git-lfs` still the way forward? @josh From the deployment side, do you have concerns or suggestions for having 100 GB data sets available to build, test, and validation slaves? The particular context motivating this question is expanding the validation data sets. In particular, @price is kindly helping to make available curated (public) data from HSC for use in testing. To fully test co-addition and get large enough areas to have enough objects to perform the variety of tests we plan, we would like to store a reference dataset of 138 GB. ---
**Author:** ![benepo](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/benepo/32/248_2.png) [@benepo](https://www.rubin.community/u/benepo)\ **Post date:** [April 28, 2016, 1:03am UTC](https://www.rubin.community/t/how-should-we-store-100-gb-data-sets-is-git-lfs-still-the-answer/724/2 "2016-04-28T01:03:05Z")
This may not be applicable in this context, but [https://google.github.io/flatbuffers/](https://google.github.io/flatbuffers/) looks interesting from an efficient data storage and access standpoint. ---
**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\ **Post date:** [April 28, 2016, 10:04am UTC](https://www.rubin.community/t/how-should-we-store-100-gb-data-sets-is-git-lfs-still-the-answer/724/3 "2016-04-28T10:04:03Z")
I don’t want to preempt responses from SQuaRE, but my understanding is that `git-lfs` is indeed the way to go, assuming that the data sets need to be versioned (which they would be if used for build/test/validation). I do worry a bit about bandwidth and latency if such data sets are retrieved frequently by automated processes. It may make sense to arrange for them to be pre-staged for automated use, particularly if they have low rates of change. ---
**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/josh/32/72_2.png) [@josh](https://www.rubin.community/u/josh)\ **Post date:** [April 28, 2016, 3:33pm UTC](https://www.rubin.community/t/how-should-we-store-100-gb-data-sets-is-git-lfs-still-the-answer/724/4 "2016-04-28T15:33:13Z")
@mwv I think a 100GB data set would be fine on the build slaves as long as we cache it. I wouldn’t be surprised if the initial download takes over an hour since our git-lfs server implementation doesn’t currently support “batching”. Theoretically, it should take less than 15mins at a sustained 1gbit/s. ---
**Author:** ![frossie](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/frossie/32/9_2.png) [@frossie](https://www.rubin.community/u/frossie)\ **Post date:** [April 28, 2016, 5:33pm UTC](https://www.rubin.community/t/how-should-we-store-100-gb-data-sets-is-git-lfs-still-the-answer/724/5 "2016-04-28T17:33:10Z")
In the general and imminent case, K-T is right and we are likely to git-lfs it right now. In the eventual and scalable case (multiple 100+ GB datasets flowing around) I would prefer to deal with them using the remote butler repository functionality against an object store, or something like that. But we don’t need to design this now. ---
**Author:** ![jmatt](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jmatt/32/96_2.png) [@jmatt](https://www.rubin.community/u/jmatt)\ **Post date:** [April 28, 2016, 7:12pm UTC](https://www.rubin.community/t/how-should-we-store-100-gb-data-sets-is-git-lfs-still-the-answer/724/6 "2016-04-28T19:12:16Z")
Git LFS is certainly capable of storing 100 GB objects. We’re using AWS S3 to store our `git-lfs` objects. The S3 limit for a single object is 5 TB. So we’re unlikely to run into that limit. ---
**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\ **Post date:** [April 28, 2016, 7:15pm UTC](https://www.rubin.community/t/how-should-we-store-100-gb-data-sets-is-git-lfs-still-the-answer/724/7 "2016-04-28T19:15:16Z")
> [@frossie](#): > > the remote butler repository functionality against an object store, Yes, that can make sense, particularly if we manage to build local caching into the Butler. I’m still a little worried that versioning via the Butler will not be as fine-grained, trackable, or synchronizable with code releases as versioning in git-lfs. ---
**Author:** ![benepo](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/benepo/32/248_2.png) [@benepo](https://www.rubin.community/u/benepo)\ **Post date:** [May 4, 2016, 5:43pm UTC](https://www.rubin.community/t/how-should-we-store-100-gb-data-sets-is-git-lfs-still-the-answer/724/8 "2016-05-04T17:43:20Z")
Another interesting option is [RethinkDB](http://www.rethinkdb.com/) (although my focus is more on data querying than data storage so it might be orthogonal to this discussion) - [Scalability supports terabytes of data](http://stackoverflow.com/questions/30367000/rethinkdb-scalability) - Open source - Supports Ruby, Python, and JavaScript - JSON/NoSQL based (nicely pairs with LSE-131 2.7: Catalog Format From DM _“The Data Management System shall deliver catalogs to EPO in compressed JSON format”_) - [Time series data](http://www.rethinkdb.com/docs/dates-and-times/python/) - [Binary data](http://www.rethinkdb.com/docs/storing-binary/python/) - [Geospatial data](http://www.rethinkdb.com/docs/geo-support/python/) - Subscribe to receive notifications (nicely pairs with alert stream model) - [API seems user-friendly](http://www.rethinkdb.com/docs/cookbook/python/) - [Nice admin interface and interactive data explorer](https://glebbahmutov.com/blog/nice-touch-rethinkdb/) [More info…](https://courses.platzi.com/classes/rethinkdb-databases/concepto/rethinkdb-course-distributed-databases/rethinkdb-course-part-1/material/) ---
**Author:** ![benepo](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/benepo/32/248_2.png) [@benepo](https://www.rubin.community/u/benepo)\ **Post date:** [May 4, 2016, 7:18pm UTC](https://www.rubin.community/t/how-should-we-store-100-gb-data-sets-is-git-lfs-still-the-answer/724/9 "2016-05-04T19:18:22Z")
One more idea: if you’re considering mining the data set with _Big Data_ analysis tools like [Apache Spark](https://spark.apache.org/), you may want to use a format supported by [Sqoop](http://www.devx.com/Java/hadoop-sqoop-to-import-data-from-mysql.html): **ASCII** - CSV **Binary** - [Avro](https://avro.apache.org/docs/current/) - Sequence Files - [Parquet](http://ingest.tips/2014/11/10/parquet-support-arriving-in-sqoop/)