# Shared Gen3 data repositories ready for (some) use **URL:** \ **Category:** DM Notifications\ **Tags:** butler, dm-dev\ **Created:** [March 24, 2021, 4:41pm UTC](https://www.rubin.community/t/shared-gen3-data-repositories-ready-for-some-use/4845 "2021-03-24T16:41:24Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![jbosch](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbosch/32/12_2.png) [@jbosch](https://www.rubin.community/u/jbosch)\ **Post date:** [March 24, 2021, 4:41pm UTC](https://www.rubin.community/t/shared-gen3-data-repositories-ready-for-some-use/4845/1 "2021-03-24T16:41:24Z")
The long-term shared Gen3 data repositories at NCSA described by [RFC-741](https://jira.lsstcorp.org/browse/RFC-741) and [DMTN-167](https://github.com/lsst-dm/dmtn-167/) are now up and ready for users (at least I think so, and it’s time to test that assertion). They are not yet complete, but there is enough there already that some at least users should be able to switch their daily work to these data repositories. I also have plenty of documentation work to do before we can declare RFC-741 done, but I figure it’s better to open up these now than to block usage on documentation. All of these repos have `u/` subdirectories (corresponding to `u/` collections) that users can write to, with other directories read-only for everyone except me and (in one case) @madamow. I can open up other directories to others as we go; I certainly expect to for e.g. calibrations. And hopefully I’ll eventually have time to put some database-side guardrails on database-only entities; for now, just continue to be careful to delete or modify only collections that start with `u/`. The data repositories and their current status and contents are outlined below, but please feel free to poke around; I highly recommend ``` $ butler query-collections /repo/main --chains=tree | less ``` # `/repo/main` This data repository will hold essentially all data from all _real_ instruments (the only known exception is what’s in `/repo/ccso`). Right now it has essentially all of the HSC data at NCSA - PDR2 and a few special programs (a small amount of calibration data failed to ingest for reasons that have been diagnosed but not fixed). There are two suites of master calibrations (`HSC/calib/gen2/20180117` and `HSC/calib/gen2/20200115`), with the former marked as the default (it was in Gen2) and usable via just `HSC/calib` (aside: should the later calibration suite actually be the default?). There are also special collections for the heavily used RC2 subset; using `HSC/RC2/defaults` as your input collection should cover everything you need for regular DRP processing, and if you’re processing the whole thing, it should make passing a visit constraint on the command-line unnecessary. The converted `w_2021_02` and `w_2021_06` Gen2 RC2 runs are present as well - I’ll do `w_2021_10` shortly, now that (I think) it’s done. Eventually `/repo/main` will include data from LSST hardware and DECam as well; the former will be ingested after the filesystem reorganization scheduled for Thursday morning, and I’ll get started on ingesting DECam data later this week as well. # `/repo/dc2` This includes all of the DESC DC2 DR6 WFD raws and processing that will be used in DP0.1 (it’s a clone of the IDF repo - or rather the converse). That includes the original calibs used for the DESC processing. The dataset processed approximately monthly by DM is a subset of this, and there are two `TAGGED` collections that contain the raws for these important subsets: - `2.2i/raw/DP0`: raws for DP0 (currently everything, but this will stay the same even if we add more raws) - `2.2i/raw/test-med-1` (DM monthly processing subset) The full DESC-run DR6 WFD processing can be found in the `2.2i/runs/DP0.1` collection. # `/repo/teststand` This contains raw data from the simulated NCSA teststands. # `/repo/ccso` This is where alternate versions of LSST raws (written by the CCS with controller=‘O’) will land. At least I think that’s what it is; I’m confident that the people who care about this repository already know more about its contents than I do. So far it just contains instrument registrations. ---
**Author:** ![cwalter](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/cwalter/32/274_2.png) [@cwalter](https://www.rubin.community/u/cwalter)\ **Post date:** [March 25, 2021, 2:36am UTC](https://www.rubin.community/t/shared-gen3-data-repositories-ready-for-some-use/4845/2 "2021-03-25T02:36:19Z")
Are the /u/ repos protected from each other? Or can one user in principle wipe out someone else’s ? ---
**Author:** ![jbosch](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jbosch/32/12_2.png) [@jbosch](https://www.rubin.community/u/jbosch)\ **Post date:** [March 25, 2021, 3:38am UTC](https://www.rubin.community/t/shared-gen3-data-repositories-ready-for-some-use/4845/3 "2021-03-25T03:38:22Z")
The `u/` files and directories should be protected from each other, and that should prevent _most_ accidental deletion of database-only per-user things, because we’ll roll back database transactions when the associated filesystem deletions fail. But will still be possible to delete or modify other people’s database content in rare cases (mostly CHAINED, TAGGED, and CALIBRATION collections, which have no direct filesystem counterparts). And direct SQL write operations are _very_ dangerous (and very much disallowed at a policy level), because those don’t have the filesystem-transaction-consistency guardrails. ---
**Author:** ![raphaelshirley](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/raphaelshirley/32/1230_2.png) [@raphaelshirley](https://www.rubin.community/u/raphaelshirley)\ **Post date:** [March 31, 2021, 4:14pm UTC](https://www.rubin.community/t/shared-gen3-data-repositories-ready-for-some-use/4845/4 "2021-03-31T16:14:16Z")
Hi, I am working in the UK and don’t think I can get an NCSA account. Is there any plan to make the gen3 HSC data public or at least a small subset for testing? I might be able to ask for a special concession to get an NCSA account but I really just want to start looking at the Butler to prepare for using it with our own obs package. Best, Raphael. ---
**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\ **Post date:** [March 31, 2021, 4:50pm UTC](https://www.rubin.community/t/shared-gen3-data-repositories-ready-for-some-use/4845/5 "2021-03-31T16:50:55Z")
We do not yet, to my knowledge, have a package containing HSC data in Gen3 form. But there are (at least) two ways of converting some of the Gen2 HSC data we publish into Gen3. 1. Use [validation\_data\_hsc](https://github.com/lsst/validation_data_hsc) and the [conversion commands from the `faro` package](https://github.com/lsst/faro/blob/master/bin/hsc_gen2_to_gen3.sh#L5-L10). 2. Use [testdata\_ci\_hsc](https://github.com/lsst/testdata_ci_hsc/) and the conversion done by the `butler, instrument, curatedCalibrations, skymap, external, raws` targets in the [ci\_hsc\_gen3 `SConstruct` file](https://github.com/lsst/ci_hsc_gen3/blob/master/SConstruct).