Wget functionality for DP2 and future data releases

Support
1

Is there already or will there be the ability to download Rubin data that is still in internal release via a wget command?

I need to have a copy of the Rubin data (coadds and object catalogs) moved over to a non-IDAC data facility where only Rubin members will have access. Making a bulk data transfer is fine for DP1 but DP2 and later releases are too large for this to be considered. A wget option could work with a “on-the-fly” data download as needed. Other considerations are welcomed.

2

Hi @Rance, thanks for posting this question. As Forum moderator I’ve moved it into the Support category to ensure it gets a response.

The implementation of a bulk download service for Rubin data remains in development, and the exact functionality to be determined – but it is planned and will exist. A bit more information can be found in the technical note SQR-119, “Metrics-driven user restriction for the RSP”, which is focused on scaleable, sustainable design for such a service for the very large future Rubin datasets.

I think this answers your main question, so I’m going to mark this reply post as the solution. But please feel free to post new topics for new or related questions anytime.

3

Thank you very much @MelissaGraham! This is great to see. For the tracking of the user usage it would be really cool if the tool is some how also user accessible so that they can track how much they are downloading and apply their own throttle. But anyways, if I wanted to keep up to date on the development of this and the potential bulk download feature, is there a channel to follow or will this document just occasionally be updated?

4

Hi @Rance, the best way to keep up-to-date with new RSP features is to check this page, which is updated with each data release and each major change to RSP functionality: Updates — Rubin Science Platform. There are also typically announcements in the Data Services - Rubin Observatory LSST Community forum category for major changes and updates as well. And always, new Support topics are welcome if you ever can’t find what you’re looking for.