# Data Management **URL:** https://www.rubin.community/c/dm/10.md [Latest](https://www.rubin.community/latest.md) · [Categories](https://www.rubin.community/categories.md) · [Tags](https://www.rubin.community/tags.md) --- ## [About the Data Management category](https://www.rubin.community/t/about-the-data-management-category/51)
**Author:** [@jsick](https://www.rubin.community/u/jsick)\ **Replies:** 0
For LSST Data Management developers to discuss LSST Stack development. Posts made here will be forwarded to the dm-devel mailing list via out Community Mailbot tool. Not a DM developer, but looking for help with DM pro… --- ## [How to upload user tables into the RSP TAP schema from a JupyterLab notebook?](https://www.rubin.community/t/how-to-upload-user-tables-into-the-rsp-tap-schema-from-a-jupyterlab-notebook/12535)
**Author:** [@assodicuori](https://www.rubin.community/u/assodicuori)\ **Replies:** 1\ **Last updated:** [August 23, 2026, 6:09am UTC](https://www.rubin.community/t/how-to-upload-user-tables-into-the-rsp-tap-schema-from-a-jupyterlab-notebook/12535 "2026-08-23T06:09:28Z")
Hello everyone, I am working inside the Rubin Science Platform (RSP) JupyterLab environment with an official LSST account. Using a notebook, I have processed and extracted a localized catalog of cataclysmic variables fr… --- ## [Deploying Rubin-env 13.1](https://www.rubin.community/t/deploying-rubin-env-13-1/12320)
**Author:** [@ross](https://www.rubin.community/u/ross)\ **Replies:** 0\ **Last updated:** [July 22, 2026, 2:12am UTC](https://www.rubin.community/t/deploying-rubin-env-13-1/12320 "2026-07-22T02:12:27Z")
We’re pleased to announce the release of rubin-env 13.1. Key highlights: Astropy upgraded to \>=8.0.1,\<9 (biggest change in this release) NumPy bumped to 2.4 libgdal-core added (3.13), with supporting bump of libpq to … --- ## [Changes to Synthetic Source Injection Dataset Naming Conventions](https://www.rubin.community/t/changes-to-synthetic-source-injection-dataset-naming-conventions/11970)
**Author:** [@lskelvin](https://www.rubin.community/u/lskelvin)\ **Replies:** 0\ **Last updated:** [May 12, 2026, 3:59pm UTC](https://www.rubin.community/t/changes-to-synthetic-source-injection-dataset-naming-conventions/11970 "2026-05-12T15:59:07Z")
On Jira ticket DM-53324, a change has been implemented in the LSST Science Pipelines that modifies the recommended dataset naming conventions we use as part of our synthetic source injection pipelines. Until now, all da… --- ## [DMTN-337: Performance of Machine-Learned Reliability Scoring for Image Differencing](https://www.rubin.community/t/dmtn-337-performance-of-machine-learned-reliability-scoring-for-image-differencing/11950)
**Author:** [@taceroc](https://www.rubin.community/u/taceroc)\ **Replies:** 0\ **Last updated:** [May 8, 2026, 1:49pm UTC](https://www.rubin.community/t/dmtn-337-performance-of-machine-learned-reliability-scoring-for-image-differencing/11950 "2026-05-08T13:49:51Z")
The first release of the DMTN-337: Performance of Machine-Learned Reliability Scoring for Image Differencing is ready and available here: https://dmtn-337.lsst.io/ DMTN-337 reports on the status of the ML reliability (R… --- ## [Deploying Rubin-Env 13](https://www.rubin.community/t/deploying-rubin-env-13/11886)
**Author:** [@ross](https://www.rubin.community/u/ross)\ **Replies:** 0\ **Last updated:** [April 17, 2026, 5:42am UTC](https://www.rubin.community/t/deploying-rubin-env-13/11886 "2026-04-17T05:42:08Z")
Hello everyone, Today I come to inform you that we have a new rubin-env out for you to play with. Rubin env 13 focuses on security and updating some of the underlying infrastructure in our environment. Rubin env 13.0.0… --- ## [RFC-1111: Change Felis YAML format and data model to use column and table names instead of IDs](https://www.rubin.community/t/rfc-1111-change-felis-yaml-format-and-data-model-to-use-column-and-table-names-instead-of-ids/10859)
**Author:** [@jeremym](https://www.rubin.community/u/jeremym)\ **Replies:** 1\ **Last updated:** [March 18, 2026, 5:21pm UTC](https://www.rubin.community/t/rfc-1111-change-felis-yaml-format-and-data-model-to-use-column-and-table-names-instead-of-ids/10859 "2026-03-18T17:21:11Z")
Hello, I would like to bring attention to RFC-1111, a proposal related to the Felis YAML format Rubin uses to manage its Science Data Model schemas. These schemas are managed in the sdm\_schemas GitHub repository and def… --- ## [Patch with SQuaRE for phalanx deployers](https://www.rubin.community/t/patch-with-square-for-phalanx-deployers/11522)
**Author:** [@frossie](https://www.rubin.community/u/frossie)\ **Replies:** 0\ **Last updated:** [February 2, 2026, 5:02pm UTC](https://www.rubin.community/t/patch-with-square-for-phalanx-deployers/11522 "2026-02-02T17:02:15Z")
We are opening a monthly “office-hours” slot for external folks running production-grade phalanx clusters to drop in (on an as-needed basis) and sync their deployments while our developers are available to answer questio… --- ## [New service: Repertoire](https://www.rubin.community/t/new-service-repertoire/11512)
**Author:** [@frossie](https://www.rubin.community/u/frossie)\ **Replies:** 0\ **Last updated:** [January 31, 2026, 3:04am UTC](https://www.rubin.community/t/new-service-repertoire/11512 "2026-01-31T03:04:31Z")
Folks who are managing their own phalanx deployments should take note of our new discovery service called Repertoire. Repertoire is a data and service discovery layer that enables Phalanx applications to auto-discover s… --- ## [New column referencing feature in Felis](https://www.rubin.community/t/new-column-referencing-feature-in-felis/11393)
**Author:** [@jeremym](https://www.rubin.community/u/jeremym)\ **Replies:** 0\ **Last updated:** [December 16, 2025, 10:56pm UTC](https://www.rubin.community/t/new-column-referencing-feature-in-felis/11393 "2025-12-16T22:56:06Z")
Greetings from the Rubin Data Engineering team. We have added a new feature to Felis that allows referencing external column definitions from within a YAML schema. This was designed to improve management of shared table… --- ## [Advertisement of rust developer guidelines](https://www.rubin.community/t/advertisement-of-rust-developer-guidelines/11368)
**Author:** [@natelust](https://www.rubin.community/u/natelust)\ **Replies:** 0\ **Last updated:** [December 8, 2025, 9:55pm UTC](https://www.rubin.community/t/advertisement-of-rust-developer-guidelines/11368 "2025-12-08T21:55:37Z")
In RFC-1069 it was decided that the Rust programming language would be added as a development language for the science pipelines. The ground work for this is done, and a draft for the developer guide on using Rust for d… --- ## [Replacing dataset entry in butler database](https://www.rubin.community/t/replacing-dataset-entry-in-butler-database/9121)
**Author:** [@jjkavelaars](https://www.rubin.community/u/jjkavelaars)\ **Replies:** 4\ **Last updated:** [December 5, 2025, 10:03pm UTC](https://www.rubin.community/t/replacing-dataset-entry-in-butler-database/9121 "2025-12-05T22:03:15Z")
We have been running pipetask quantum backed butlers and merging datasets into our butler after processing. When transferring from graph output into the butler db, we get the following error: butler --log-level=ERROR -… --- ## [Science Pipeline Release 30.0.0 Status and Discussion](https://www.rubin.community/t/science-pipeline-release-30-0-0-status-and-discussion/11349)
**Author:** [@mwittgen](https://www.rubin.community/u/mwittgen)\ **Replies:** 1\ **Last updated:** [December 4, 2025, 11:33pm UTC](https://www.rubin.community/t/science-pipeline-release-30-0-0-status-and-discussion/11349 "2025-12-04T23:33:00Z")
We are starting the science pipelines release process for 30.0.0 aimed to support DP2 processing and DP2 release to the community. The process will be tracked on ticket DM-47297 and this post. The backport status can be… --- ## [Proposal: FGOC (Focal-Geometry and Curvature) Module for Early ISO Detection in LSST Prompt Processing](https://www.rubin.community/t/proposal-fgoc-focal-geometry-and-curvature-module-for-early-iso-detection-in-lsst-prompt-processing/11330)
**Author:** [@a8864666](https://www.rubin.community/u/a8864666)\ **Replies:** 2\ **Last updated:** [December 1, 2025, 6:09pm UTC](https://www.rubin.community/t/proposal-fgoc-focal-geometry-and-curvature-module-for-early-iso-detection-in-lsst-prompt-processing/11330 "2025-12-01T18:09:04Z")
Proposal: FGOC (Focal-Geometry and Curvature) Module for Early ISO Detection in LSST Prompt Processing Hi Rubin team, I have been studying short-arc geometric behavior with the goal of enabling earlier ISO / non-Kepleri… --- ## [New index management features added to Felis](https://www.rubin.community/t/new-index-management-features-added-to-felis/11281)
**Author:** [@jeremym](https://www.rubin.community/u/jeremym)\ **Replies:** 0\ **Last updated:** [November 18, 2025, 5:35pm UTC](https://www.rubin.community/t/new-index-management-features-added-to-felis/11281 "2025-11-18T17:35:06Z")
Greetings from the Rubin Data Engineering team. Some new index management features have been added to Felis in response to reports that loading data into tables with indexes in Postgres was performing slowly (GitHub #15… --- ## [Supported Kubernetes verions for phalanx deployers](https://www.rubin.community/t/supported-kubernetes-verions-for-phalanx-deployers/11223)
**Author:** [@frossie](https://www.rubin.community/u/frossie)\ **Replies:** 0\ **Last updated:** [November 5, 2025, 9:39pm UTC](https://www.rubin.community/t/supported-kubernetes-verions-for-phalanx-deployers/11223 "2025-11-05T21:39:21Z")
We understand that there are incompatibility issues between phalanx and older k8s versions. If you are a data center deploying RSP / phalanx be aware that we only test (and support) the following versions: From the ol… --- ## [Transitioning to eups.lsst.cloud on 11/12/2025](https://www.rubin.community/t/transitioning-to-eups-lsst-cloud-on-11-12-2025/11199)
**Author:** [@ross](https://www.rubin.community/u/ross)\ **Replies:** 0\ **Last updated:** [November 3, 2025, 6:26pm UTC](https://www.rubin.community/t/transitioning-to-eups-lsst-cloud-on-11-12-2025/11199 "2025-11-03T18:26:07Z")
As part of our ongoing efforts to improve our services, streamline our operations and move away from AWS, we have decided to permanently transition from eups.lsst.codes to eups.lsst.cloud next week. Over the past month,… --- ## [DM Monthly Status Report for 09 2025](https://www.rubin.community/t/dm-monthly-status-report-for-09-2025/11144)
**Author:** [@kervan](https://www.rubin.community/u/kervan)\ **Replies:** 0\ **Last updated:** [October 22, 2025, 3:20pm UTC](https://www.rubin.community/t/dm-monthly-status-report-for-09-2025/11144 "2025-10-22T15:20:07Z")
The DM monthly status report covering September activities has been posted to DocuShare, collection-16257. For convenience, the High-level Summary is pasted below. Direct link to the full report: DocuShare License Error … --- ## [Gaia DR2 reference catalog in LSST format](https://www.rubin.community/t/gaia-dr2-reference-catalog-in-lsst-format/3901)
**Author:** [@parejkoj](https://www.rubin.community/u/parejkoj)\ **Replies:** 19\ **Last updated:** [October 17, 2025, 4:52pm UTC](https://www.rubin.community/t/gaia-dr2-reference-catalog-in-lsst-format/3901 "2025-10-17T16:52:13Z")
Gaia DR2 refcat on lsst-dev: gaia\_dr2\_20200414 The complete Gaia DR2 catalog is now available on lsst-dev at /datasets/refcats/htm/v1/gaia\_dr2\_20200414 (see RFC-634). This catalog contains about 1.7 x 10^9 sources betwee… --- ## [New PredictedQuantumGraph class and file format](https://www.rubin.community/t/new-predictedquantumgraph-class-and-file-format/11033)
**Author:** [@jbosch](https://www.rubin.community/u/jbosch)\ **Replies:** 1\ **Last updated:** [October 2, 2025, 2:42pm UTC](https://www.rubin.community/t/new-predictedquantumgraph-class-and-file-format/11033 "2025-10-02T14:42:58Z")
I’ve just merged (in time for d\_2025\_10\_01) DM-51850, which adds a new lsst.pipe.base.quantum\_graph.PredictedQuantumGraph class and corresponding file format, which will soon be fully replacing lsst.pipe.base.QuantumGrap… --- ## [DM Monthly Status Report for 08 2025](https://www.rubin.community/t/dm-monthly-status-report-for-08-2025/10999)
**Author:** [@kervan](https://www.rubin.community/u/kervan)\ **Replies:** 1\ **Last updated:** [September 25, 2025, 5:23pm UTC](https://www.rubin.community/t/dm-monthly-status-report-for-08-2025/10999 "2025-09-25T17:23:11Z")
The DM monthly status report covering August activities has been posted to DocuShare, collection-16257. For convenience, the High-level Summary is pasted below. Direct link to the full report: DocuShare Authorization Err… --- ## [LSST Science Pipelines release 29.2.0 is now available](https://www.rubin.community/t/lsst-science-pipelines-release-29-2-0-is-now-available/10801)
**Author:** [@mwittgen](https://www.rubin.community/u/mwittgen)\ **Replies:** 1\ **Last updated:** [September 18, 2025, 8:41pm UTC](https://www.rubin.community/t/lsst-science-pipelines-release-29-2-0-is-now-available/10801 "2025-09-18T20:41:32Z")
Version 29.2.0 of the LSST Science Pipelines is now available. This is a minor feature release to address python binding memory leaks (see DM-38772) Updated documentation is available at https://pipelines.lsst.io/v/v2… --- ## [Deploying Rubin env 11](https://www.rubin.community/t/deploying-rubin-env-11/10969)
**Author:** [@ross](https://www.rubin.community/u/ross)\ **Replies:** 0\ **Last updated:** [September 16, 2025, 6:21pm UTC](https://www.rubin.community/t/deploying-rubin-env-11/10969 "2025-09-16T18:21:56Z")
I am excited to announce that we are rolling out another significant upgrade to our build infrastructure. This week, we will transition to the rubin-env 11.0.0 conda-forge metapackage. We will be rolling this out today (… --- ## [Ctrl\_bps\_htcondor will combine job's stderr and stdout](https://www.rubin.community/t/ctrl-bps-htcondor-will-combine-jobs-stderr-and-stdout/10943)
**Author:** [@mxk](https://www.rubin.community/u/mxk)\ **Replies:** 0\ **Last updated:** [September 11, 2025, 6:06pm UTC](https://www.rubin.community/t/ctrl-bps-htcondor-will-combine-jobs-stderr-and-stdout/10943 "2025-09-11T18:06:50Z")
Once DM-31879 is merged, BPS HTCondor plugin ( ctrl\_bps\_htcondor) will start redirecting both job’s stdout and stderr to the same file (\.out). In other words, BPS submissions using ctrl\_bps\_htcondor will no lo… --- ## [DM Monthly Status Report for 07 2025](https://www.rubin.community/t/dm-monthly-status-report-for-07-2025/10865)
**Author:** [@kervan](https://www.rubin.community/u/kervan)\ **Replies:** 0\ **Last updated:** [August 26, 2025, 5:30pm UTC](https://www.rubin.community/t/dm-monthly-status-report-for-07-2025/10865 "2025-08-26T17:30:31Z")
The DM monthly status report covering XXX activities has been posted to DocuShare, collection-16257. For convenience, the High-level Summary is pasted below. Direct link to the full report: DocuShare Authorization Error … --- ## [Third-party metadetect included as part of lsst\_distrib](https://www.rubin.community/t/third-party-metadetect-included-as-part-of-lsst-distrib/10806)
**Author:** [@arunkannawadi](https://www.rubin.community/u/arunkannawadi)\ **Replies:** 0\ **Last updated:** [August 13, 2025, 3:44pm UTC](https://www.rubin.community/t/third-party-metadetect-included-as-part-of-lsst-distrib/10806 "2025-08-13T15:44:11Z")
I just merged DM-52110 which distributes a DM-maintained fork of the third-party metadetect as part of lsst\_distrib (as a dependency of drp\_tasks) as discussed in RFC-1100. This would be available by default when setting… --- ## [Phalanx development now uses uv](https://www.rubin.community/t/phalanx-development-now-uses-uv/10765)
**Author:** [@rra](https://www.rubin.community/u/rra)\ **Replies:** 0\ **Last updated:** [August 6, 2025, 6:15pm UTC](https://www.rubin.community/t/phalanx-development-now-uses-uv/10765 "2025-08-06T18:15:39Z")
Hello everyone, Following very successful experiences with our other services, we are switching to uv to manage the Phalanx Python code. If you are doing Phalanx development, either as an environment administrator or an… --- ## [Deploying rubin-env 10.1.0](https://www.rubin.community/t/deploying-rubin-env-10-1-0/10763)
**Author:** [@mwittgen](https://www.rubin.community/u/mwittgen)\ **Replies:** 0\ **Last updated:** [August 5, 2025, 10:40pm UTC](https://www.rubin.community/t/deploying-rubin-env-10-1-0/10763 "2025-08-05T22:40:25Z")
The minor version update rubin-env 10.1.0 will be deployed effective with the following releases: Weekly: w\_2025\_32 Daily: d\_2025\_08\_06 Change: pybind11 has been updated from version 2 to version 3. While vers… --- ## [Pipeline execution code moved to pipe\_base](https://www.rubin.community/t/pipeline-execution-code-moved-to-pipe-base/10721)
**Author:** [@jbosch](https://www.rubin.community/u/jbosch)\ **Replies:** 0\ **Last updated:** [July 29, 2025, 10:24pm UTC](https://www.rubin.community/t/pipeline-execution-code-moved-to-pipe-base/10721 "2025-07-29T22:24:28Z")
I’ve just merged DM-48980, which implements the deprecation phase of RFC-1066 by moving a lot of code from ctrl\_mpexec to pipe\_base (with shims for backwards compatibility that emit deprecation warnings when imported or … --- ## [DM Monthly Status Report for 06 2025](https://www.rubin.community/t/dm-monthly-status-report-for-06-2025/10654)
**Author:** [@kervan](https://www.rubin.community/u/kervan)\ **Replies:** 0\ **Last updated:** [July 23, 2025, 7:47pm UTC](https://www.rubin.community/t/dm-monthly-status-report-for-06-2025/10654 "2025-07-23T19:47:57Z")
The DM monthly status report covering June activities has been posted to DocuShare, collection-16257. For convenience, the High-level Summary is pasted below. Direct link to the full report: DocuShare Authorization Error … [Next page](https://www.rubin.community/c/dm/10.md?page=1)