# ApTemplate missing finalVisitSummary dataset type **URL:** \ **Category:** Support\ **Created:** [June 26, 2024, 8:35pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784 "2024-06-26T20:35:08Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![ess20clsi](https://avatars.discourse-cdn.com/v4/letter/e/278dde/32.png) [@ess20clsi](https://www.rubin.community/u/ess20clsi)\ **Post date:** [June 26, 2024, 8:35pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/1 "2024-06-26T20:35:08Z")
Hi all, I am currently trying to run the AP Pipeline on some HSC data, with the ultimate goal of eventually doing image differencing. When I try to run the `ApTemplate.yaml` pipeline, I am getting the following error: ```auto lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback: Traceback (most recent call last): File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 199, in run if (qgraph := script.qgraph(pipelineObj=pipeline, **kwargs, show=show)) is None: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 210, in qgraph qgraph = f.makeGraph(pipelineObj, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 622, in makeGraph qgraph = graphBuilder.makeGraph( ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graphBuilder.py", line 1840, in makeGraph return scaffolding.makeQuantumGraph( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graphBuilder.py", line 1651, in makeQuantumGraph registryDatasetTypes=self._get_registry_dataset_types(registry), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graphBuilder.py", line 1682, in _get_registry_dataset_types raise MissingDatasetTypeError(f"Registry is missing an input dataset type {dstype}") lsst.daf.butler.registry._exceptions.MissingDatasetTypeError: "Registry is missing an input dataset type DatasetType('finalVisitSummary', {band, instrument, physical_filter, visit}, ExposureCatalog) ``` [This post](http://www.rubin.community/t/skipping-jointcal-results-at-makewarp-v26/8762) from a few days ago suggests running `updateVisitSummary` before running `makeWarp`, but I don’t really know how that plays into the pipeline. I’ve tried running everything up to `consolidateVisitSummary`, followed by the `updateVisitTask`, with the same “No global calibration during nightly validation” configs as suggested in the other post, and I get the following: ```auto lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback: Traceback (most recent call last): File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 199, in run if (qgraph := script.qgraph(pipelineObj=pipeline, **kwargs, show=show)) is None: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 210, in qgraph qgraph = f.makeGraph(pipelineObj, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 622, in makeGraph qgraph = graphBuilder.makeGraph( ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graphBuilder.py", line 1831, in makeGraph scaffolding.resolveDatasetRefs( File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graphBuilder.py", line 1357, in resolveDatasetRefs raise RuntimeError( RuntimeError: Dataset 'visitSummary_schema' (with no dimensions) could not be found in collections ('u/NH/processCCD/20240622T202946Z', 'HSC/raw/all', 'refcats/gen2', 'HSC/calib/run1', 'HSC/calib', 'u/NH/visitSummary/20240626T195758Z'). ``` However the `visitSummary_schema` dataset definitely exists within the `u/NH/visitSummary/20240626T195758Z` collection. If anyone has any ideas on how to solve the issue I would really appreciate the help! For reference, I am running V26 of the software. ---
**Author:** ![sfu](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sfu/32/3060_2.png) [@sfu](https://www.rubin.community/u/sfu)\ **Post date:** [June 29, 2024, 1:30am UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/2 "2024-06-29T01:30:26Z")
Hi, perhaps you can take a look at Lee’s notes. Those pipeline figures helped me understand how those tasks/data sets are connected. [Merian Data Processing Using the LSST Science Pipelines - HackMD](https://hackmd.io/@lsk/merian) ---
**Author:** ![rbliu](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rbliu/32/637_2.png) [@rbliu](https://www.rubin.community/u/rbliu)\ **Post date:** [June 29, 2024, 4:52pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/3 "2024-06-29T16:52:45Z")
Just a caveat about the Merian Survey Tutorial: there seems to be a missing `updateVisitSummary` in its `step 2d` – I had to add that task before running the following steps successfully. The order of these tasks should be like: ```auto Quanta Tasks ------ ---------------------------- 2 finalizeCharacterization 2 updateVisitSummary 18 writeRecalibratedSourceTable 18 transformSourceTable 2 consolidateSourceTable ``` So, I guess you could try to rerun these tasks and make sure that `updateVisitSummary` looks good. For the `visitSummary_schema` error, I also have no idea. I got everything worked after adding the configs to the pipeline yaml file. ```auto tasks: updateVisitSummary: class: lsst.drp.tasks.update_visit_summary.UpdateVisitSummaryTask config: wcs_provider: "input_summary" photo_calib_provider: "input_summary" background_provider: "input_summary" ``` ---
**Author:** ![galaxyumi331](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/galaxyumi331/32/2677_2.png) [@galaxyumi331](https://www.rubin.community/u/galaxyumi331)\ **Post date:** [July 1, 2024, 5:39pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/4 "2024-07-01T17:39:33Z")
Hi @ess20clsi, Have you had a chance to try suggestions by @sfu and @rbliu to see if these resolve your problem? ---
**Author:** ![ess20clsi](https://avatars.discourse-cdn.com/v4/letter/e/278dde/32.png) [@ess20clsi](https://www.rubin.community/u/ess20clsi)\ **Post date:** [July 4, 2024, 7:17pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/5 "2024-07-04T19:17:57Z")
Hi all, Sorry for taking so long to get back to this. Based on the quantum graphs in [Merian Data Processing Using the LSST Science Pipelines - HackMD](https://hackmd.io/@lsk/merian#5-Data-release-production) and from @rbliu’s response, I tried adding the `finalizeCharacterization` task to my pipeline, as it looks like that output dataset is needed to run `updateVisitSummary`. Unfortunately now I get the following: `RuntimeError: 1 dataset(s) of type 'src_schema' was/were present in a previous query, but could not be found now. This is either a logic bug in QuantumGraph generation or the input collections have been modified since QuantumGraph generation began.` Does anyone have any other ideas? ---
**Author:** ![kfindeisen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/kfindeisen/32/833_2.png) [@kfindeisen](https://www.rubin.community/u/kfindeisen)\ **Post date:** [July 10, 2024, 9:50pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/6 "2024-07-10T21:50:42Z")
Sorry for not seeing this sooner. First, note that [RFC-1018](https://rubinobs.atlassian.net/issues/RFC-1018) deprecates `ApTemplate` in favor of the DRP pipelines, so you may wish to use those to make your templates. Most of the above discussion is about the DRP pipelines anyway, and does not apply to `ApTemplate` (which was written using different assumptions about what processing is needed). If you do wish to continue with `ApTemplate`, the original bug you reported was fixed in `w_2023_38`, so I recommend using that version or a later weekly (it’s not fixed in 26.0.2, and release 27 is not quite ready yet). If you can’t switch to a newer version, you _might_ be able to patch your pipeline by setting `makeWarp.config.doApplyFinalizedPsf` to `False`. But there have been a lot of changes to the pipeline since release 26, so I can’t promise that that’s the only change needed. ---
**Author:** ![ess20clsi](https://avatars.discourse-cdn.com/v4/letter/e/278dde/32.png) [@ess20clsi](https://www.rubin.community/u/ess20clsi)\ **Post date:** [July 12, 2024, 6:33pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/7 "2024-07-12T18:33:49Z")
Thank you @kfindeisen, that is very helpful. I don’t need to continue with APTemplate, in fact it has been causing me other issues and so a different pipeline sounds perfect. I am trying to do difference imaging on some HSC data and the ap pipeline seemed like the best way to do that, but is there a different/better way? I am _very_ new to the LSST science pipeline, and have been struggling to figure out which tasks I should run in order to get a difference image. So you have any suggestions? ---
**Author:** ![kfindeisen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/kfindeisen/32/833_2.png) [@kfindeisen](https://www.rubin.community/u/kfindeisen)\ **Post date:** [July 12, 2024, 7:21pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/8 "2024-07-12T19:21:27Z")
The AP pipeline itself (`ApPipe.yaml`) is a perfectly fine way to do difference imaging, and should work\* (we run it on HSC data on a regular basis). The problem is getting the templates it needs (by default, `goodSeeing` coadds) in the first place. Unfortunately, I’ve never used [the DRP pipelines](https://github.com/lsst/drp_pipe/tree/main/pipelines) myself, so I can’t offer any advice re: which of its steps are needed. @mrawls or @elhoward might be able to help you there. \*Looking ahead, the one catch for `ApPipe` is that its source association code assumes access to a scratch database (APDB). Depending on whether you care about creating DiaObjects or whether you just want difference _images_, you would need to either: - create an SQLite APDB as described [in the `ap_pipe` docs](https://pipelines.lsst.io/v/v26_0_2/modules/lsst.ap.pipe/apdb.html) [link is for release 26; recent builds use [a different procedure](https://pipelines.lsst.io/v/daily/modules/lsst.ap.pipe/apdb.html)] - exclude the `diaPipe` task from `ApPipe`, but ensure you run `retrieveTemplate`, `subtractImages`, and possibly `detectAndMeasure` ---
**Author:** ![sgreenstreet](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sgreenstreet/32/1975_2.png) [@sgreenstreet](https://www.rubin.community/u/sgreenstreet)\ **Post date:** [July 15, 2024, 9:47pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/9 "2024-07-15T21:47:34Z")
Hi @ess20clsi - I wanted to follow-up to see if @kfindeisen’s response provided you with the solution you were looking for? ---
**Author:** ![ess20clsi](https://avatars.discourse-cdn.com/v4/letter/e/278dde/32.png) [@ess20clsi](https://www.rubin.community/u/ess20clsi)\ **Post date:** [July 15, 2024, 11:35pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/10 "2024-07-15T23:35:10Z")
Thank you so much, @kfindeisen! I really appreciate all your help. Ideally, I wouldn’t run _ALL_ of the DRP pipeline steps, as I don’t need many of the outputs it provides and I don’t want to spend time on steps I don’t need. As for `ApPipe`, I don’t need to create DiaObjects so I think it will be smooth sailing once I can create those templates. If anyone knows what steps/tasks in the DRP pipeline are needed for template construction, I would really appreciate the advice. In the mean time @sgreenstreet we can go ahead a mark @kfindeisen 's earlier response as the solution. ---
**Author:** ![parejkoj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/parejkoj/32/138_2.png) [@parejkoj](https://www.rubin.community/u/parejkoj)\ **Post date:** [July 18, 2024, 8:52pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/11 "2024-07-18T20:52:38Z")
@laurenam, @yusra , @lee could all offer suggestions about what steps of DRP to run to make the coadds you need. ---
**Author:** ![sgreenstreet](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sgreenstreet/32/1975_2.png) [@sgreenstreet](https://www.rubin.community/u/sgreenstreet)\ **Post date:** [July 18, 2024, 9:10pm UTC](https://www.rubin.community/t/aptemplate-missing-finalvisitsummary-dataset-type/8784/12 "2024-07-18T21:10:38Z")
Hi @ess20clsi - I also wanted to mention that [DP0.2 tutorial notebook 03c](https://github.com/rubin-dp0/tutorial-notebooks/blob/main/DP02_03c_Big_deepCoadd_Cutout/DP02_03c_Big_deepCoadd_Cutout.ipynb) demonstrates how to use the `GetTemplateTask` to create a single cutout image with contributions from multiple adjacent patches and tracts. In the LSST Science Pipelines, `GetTemplateTask` is used to create a template image from `deepCoadd` images for a given processed visit image, in order to perform Difference Image Analysis. This notebook may be a good resource for the steps needed for the template construction you’d like to do.