# \#middleware **URL:** https://www.rubin.community/tag/middleware/177.md [Latest](https://www.rubin.community/latest.md) · [Categories](https://www.rubin.community/categories.md) · [Tags](https://www.rubin.community/tags.md) --- ## [New PredictedQuantumGraph class and file format](https://www.rubin.community/t/new-predictedquantumgraph-class-and-file-format/11033)
**Author:** [@jbosch](https://www.rubin.community/u/jbosch)\ **Replies:** 1\ **Last updated:** [October 2, 2025, 2:42pm UTC](https://www.rubin.community/t/new-predictedquantumgraph-class-and-file-format/11033 "2025-10-02T14:42:58Z")
I’ve just merged (in time for d\_2025\_10\_01) DM-51850, which adds a new lsst.pipe.base.quantum\_graph.PredictedQuantumGraph class and corresponding file format, which will soon be fully replacing lsst.pipe.base.QuantumGrap… --- ## [More support for configurable PipelineTask connections](https://www.rubin.community/t/more-support-for-configurable-pipelinetask-connections/7630)
**Author:** [@jbosch](https://www.rubin.community/u/jbosch)\ **Replies:** 6\ **Last updated:** [November 20, 2023, 4:13pm UTC](https://www.rubin.community/t/more-support-for-configurable-pipelinetask-connections/7630 "2023-11-20T16:13:25Z")
After some misadventures involving git revert, I’ve just merged DM-38953, which provides a lot more support for PipelineTask connections that depend on configuration for more than just the dataset type name. This includ… --- ## [The QuantumGraph generation algorithm has been rewritten](https://www.rubin.community/t/the-quantumgraph-generation-algorithm-has-been-rewritten/7878)
**Author:** [@jbosch](https://www.rubin.community/u/jbosch)\ **Replies:** 0\ **Last updated:** [August 24, 2023, 6:08pm UTC](https://www.rubin.community/t/the-quantumgraph-generation-algorithm-has-been-rewritten/7878 "2023-08-24T18:08:50Z")
I’ve just merged DM-38498 (after w\_2023\_34 was tagged), which is a complete rewrite of the QuantumGraph generation algorithm. For the most part, you shouldn’t notice much of a difference, but there are a few important e… --- ## [New in-Python pipeline executor available](https://www.rubin.community/t/new-in-python-pipeline-executor-available/7569)
**Author:** [@kfindeisen](https://www.rubin.community/u/kfindeisen)\ **Replies:** 0\ **Last updated:** [April 18, 2023, 12:35am UTC](https://www.rubin.community/t/new-in-python-pipeline-executor-available/7569 "2023-04-18T00:35:27Z")
DM-36162 adds a third pipeline executor to the Science Pipelines. SeparablePipelineExecutor was designed for the Prompt Processing framework’s needs, but it’s flexible enough that other developers may find it useful. Co… --- ## [Simplifying and limiting deletion in Gen3 butler repos](https://www.rubin.community/t/simplifying-and-limiting-deletion-in-gen3-butler-repos/4808)
**Author:** [@jbosch](https://www.rubin.community/u/jbosch)\ **Replies:** 5\ **Last updated:** [June 14, 2022, 9:41pm UTC](https://www.rubin.community/t/simplifying-and-limiting-deletion-in-gen3-butler-repos/4808 "2022-06-14T21:41:31Z")
See the bottom of this post for a glossary that may help Gen3 non-experts understand it. The problems: Right now the Gen3 butler has two high-level methods, pruneCollection and pruneDatasets, which try to cover all o… --- ## [Change in Task metadata](https://www.rubin.community/t/change-in-task-metadata/6210)
**Author:** [@timj](https://www.rubin.community/u/timj)\ **Replies:** 6\ **Last updated:** [January 12, 2022, 9:51pm UTC](https://www.rubin.community/t/change-in-task-metadata/6210 "2022-01-12T21:51:47Z")
With the implementation of RFC-783 there has been a change in the way that the metadata attached to a Task is implemented. Previously metadata was stored in a PropertyList which was combined into a PropertySet. Now we ha… --- ## [Expanding and documenting Pipeline definitions](https://www.rubin.community/t/expanding-and-documenting-pipeline-definitions/6118)
**Author:** [@jbosch](https://www.rubin.community/u/jbosch)\ **Replies:** 8\ **Last updated:** [November 25, 2021, 3:07am UTC](https://www.rubin.community/t/expanding-and-documenting-pipeline-definitions/6118 "2021-11-25T03:07:05Z")
Following up on RFC-775 On RFC-775, @natelust and I proposed - without a lot of detail - having the new drp\_pipe package separate Pipeline source files (in recipes and ingredients directories) from their expanded forms (… --- ## [Changes to saving and loading QuantumGraphs](https://www.rubin.community/t/changes-to-saving-and-loading-quantumgraphs/4587)
**Author:** [@natelust](https://www.rubin.community/u/natelust)\ **Replies:** 0\ **Last updated:** [December 7, 2020, 8:29pm UTC](https://www.rubin.community/t/changes-to-saving-and-loading-quantumgraphs/4587 "2020-12-07T20:29:36Z")
DM-27784 and DM-27667 introduced a new system for saving and loading QuantumGraph objects created with the pipetask command. The file name of a QuantumGraph object must now end with the extension .qgraph. The binary rep… --- ## [Processing-output metadata and metrics in Gen3 queries and QuantumGraph generation](https://www.rubin.community/t/processing-output-metadata-and-metrics-in-gen3-queries-and-quantumgraph-generation/4524)
**Author:** [@jbosch](https://www.rubin.community/u/jbosch)\ **Replies:** 5\ **Last updated:** [November 10, 2020, 8:00pm UTC](https://www.rubin.community/t/processing-output-metadata-and-metrics-in-gen3-queries-and-quantumgraph-generation/4524 "2020-11-10T20:00:43Z")
We’ve had a few discussions on Slack about how to make exposure or visit (or similar) metadata values that are produced by processing (e.g. seeing derived from the PSF model, not DIMM data) available to the Gen3 butler q… --- ## [Query interfaces changes in Gen3 Butler](https://www.rubin.community/t/query-interfaces-changes-in-gen3-butler/4307)
**Author:** [@jbosch](https://www.rubin.community/u/jbosch)\ **Replies:** 0\ **Last updated:** [August 7, 2020, 7:34pm UTC](https://www.rubin.community/t/query-interfaces-changes-in-gen3-butler/4307 "2020-08-07T19:34:58Z")
I’ve just merged DM-25919, which both renames and modifies some of the Gen3 butler query methods that many of you are already familiar with. First, the bad news (breaking changes): Registry.queryDimensions has been r… --- ## [Deleting datasets and provenance in the Gen3 Butler](https://www.rubin.community/t/deleting-datasets-and-provenance-in-the-gen3-butler/4111)
**Author:** [@jbosch](https://www.rubin.community/u/jbosch)\ **Replies:** 5\ **Last updated:** [April 22, 2020, 1:54pm UTC](https://www.rubin.community/t/deleting-datasets-and-provenance-in-the-gen3-butler/4111 "2020-04-22T13:54:12Z")
This is a very technical post intended primarily for Gen3 middleware insiders. If you have an opinion but lack the background for all of it to make sense, please ask for clarification here or in #dm-middleware on Slack. … --- ## [Changes to collections, runs, and \`pipetask\` arguments in Gen3 middleware](https://www.rubin.community/t/changes-to-collections-runs-and-pipetask-arguments-in-gen3-middleware/4078)
**Author:** [@jbosch](https://www.rubin.community/u/jbosch)\ **Replies:** 0\ **Last updated:** [March 25, 2020, 2:52am UTC](https://www.rubin.community/t/changes-to-collections-runs-and-pipetask-arguments-in-gen3-middleware/4078 "2020-03-25T02:52:08Z")
I’ve just merged DM-21849, which implements almost all of RFC-663 (enough that I’m calling it Implemented; the remaining details will trickle in as Gen3 development continues). This changes how collections behave concep… --- ## [Gen3 Instruments, registries, and ingestion](https://www.rubin.community/t/gen3-instruments-registries-and-ingestion/3946)
**Author:** [@parejkoj](https://www.rubin.community/u/parejkoj)\ **Replies:** 6\ **Last updated:** [November 28, 2019, 6:17pm UTC](https://www.rubin.community/t/gen3-instruments-registries-and-ingestion/3946 "2019-11-28T18:17:44Z")
These are notes from a discussion @timj and I had today about the gen3 Instrument class, how Instruments are registered with a Butler, and how to handle ingesting “instrument signature data” (brighter-fatter, defects, tr… --- ## [DMTN-025: A survey of workflow management systems](https://www.rubin.community/t/dmtn-025-a-survey-of-workflow-management-systems/1124)
**Author:** [@robkooper](https://www.rubin.community/u/robkooper)\ **Replies:** 0\ **Last updated:** [September 6, 2016, 11:12pm UTC](https://www.rubin.community/t/dmtn-025-a-survey-of-workflow-management-systems/1124 "2016-09-06T23:12:14Z")
LSST Batch Production Services \[CONOPS\] are about executing “\[...\] campaigns on computing resources to produce the desired LSST data products \[...\]” where campaigns are defined as sets of pipelines (ordered ensembles of …