# In DECam data reduction, I faced a FileNotFoundError in "Ingest Raw Images" step **URL:** \ **Category:** Support\ **Tags:** butler, decam\ **Created:** [July 23, 2023, 10:54pm UTC](https://www.rubin.community/t/in-decam-data-reduction-i-faced-a-filenotfounderror-in-ingest-raw-images-step/7779 "2023-07-23T22:54:06Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Boseong](https://avatars.discourse-cdn.com/v4/letter/b/dbc845/32.png) [@Boseong](https://www.rubin.community/u/Boseong)\ **Post date:** [July 23, 2023, 10:54pm UTC](https://www.rubin.community/t/in-decam-data-reduction-i-faced-a-filenotfounderror-in-ingest-raw-images-step/7779/1 "2023-07-23T22:54:06Z")
Hello, I have initiated DECam data reduction by adhering to the guidelines provided in this tutorial: [The Gen3 workflow — LSST Science Pipelines](https://pipelines.lsst.io/modules/lsst.obs.decam/using-obs-decam.html) I faced an error when I ran this code : pipetask run -b REPO -i DECam/raw/all -o DECam/raw/crosstalk-sources -p $CP\_PIPE\_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml --register-dataset-types This is an error message: FileNotFoundError: Not enough datasets (0) found for non-optional connection overscan.camera (camera) with minimum=1 for quantum data ID {instrument: ‘DECam’, detector: 1, exposure: 516081, …}. I speculate that the error might have been due to the fact that I skipped executing line B and only ran line A. line A: butler ingest-raws REPO /path/to/raw/science/_.fits.fz --transfer link line B: butler ingest-raws REPO /path/to/raw/calibs/_.fits.fz --transfer link Since I only possess the original image and lack a bias, flat image, I was unable to run line B. \*As a trial, I ran the process with 3 files of r filter. Being my first attempt, I am uncertain if I have stated my query correctly. Any corrections or feedback would be greatly appreciated. Best regards, Boseong ---
**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\ **Post date:** [July 24, 2023, 12:14am UTC](https://www.rubin.community/t/in-decam-data-reduction-i-faced-a-filenotfounderror-in-ingest-raw-images-step/7779/2 "2023-07-24T00:14:19Z")
Did you run the `butler write-curated-calibrations` line? That will register the camera dataset. ---
**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\ **Post date:** [July 24, 2023, 1:26am UTC](https://www.rubin.community/t/in-decam-data-reduction-i-faced-a-filenotfounderror-in-ingest-raw-images-step/7779/3 "2023-07-24T01:26:06Z")
> [@Boseong](#): > > I was unable to run line B. You will also need the image calibrations (flats, bias etc) otherwise you will not be able to run single frame processing. ---
**Author:** ![Boseong](https://avatars.discourse-cdn.com/v4/letter/b/dbc845/32.png) [@Boseong](https://www.rubin.community/u/Boseong)\ **Post date:** [July 24, 2023, 5:22am UTC](https://www.rubin.community/t/in-decam-data-reduction-i-faced-a-filenotfounderror-in-ingest-raw-images-step/7779/4 "2023-07-24T05:22:05Z")
Thank you for replying. Yes, I realized that I had not included any calibration files, so I attempted to download these files. However, I could not find any file corresponding to the raw file in the NOIRLAB astro data archive. For example, In NOIRLAB astro data archive, I set a coordinate (104.611, -55.948) and filter(r DECam SDSS c0002 6415.0 1480.0) and instrument(ct4m, decam) only. Among the raw files, I found only object files. I don’t believe that flat and bias files are categorized under ‘instcal’. Therefore, it appears that there are no matching flat and bias files for the specific raw object file. Best regards, Boseong ---
**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\ **Post date:** [July 24, 2023, 2:42pm UTC](https://www.rubin.community/t/in-decam-data-reduction-i-faced-a-filenotfounderror-in-ingest-raw-images-step/7779/5 "2023-07-24T14:42:53Z")
You are going to need to either construct the flats and biases yourselves or else find some valid ones from somewhere. If you are bringing in external flats/biases you have to be careful that they match the form expected by the LSST science pipelines. Maybe @lskelvin can give you some pointers. See also this post and the discussion about Merian: > [@Adding new data to a butler gen3 \[DECam\]](http://www.rubin.community/t/adding-new-data-to-a-butler-gen3-decam/7728): > > Hi all, I already reduced a set of DECam images using the steps from the Merian Data Processing guide I now want to include new observations and sets of calibrations and am getting an error. I ingested the new raw data, and made the master bias (output to ‘DECam/calib/jhernandez/bias/’), but at the moment of running the step of generating crosstalk sources I am getting ValueError: Output CHAINED collection ‘DECam/calib/jhernandez/crosstalk’ exists, but it ends with a different sequence of i… ---
**Author:** ![Boseong](https://avatars.discourse-cdn.com/v4/letter/b/dbc845/32.png) [@Boseong](https://www.rubin.community/u/Boseong)\ **Post date:** [July 31, 2023, 10:57am UTC](https://www.rubin.community/t/in-decam-data-reduction-i-faced-a-filenotfounderror-in-ingest-raw-images-step/7779/6 "2023-07-31T10:57:16Z")
Thank you for providing such informative tutorial page! Fortunately, I resolved the issue using the information from this community page: [http://www.rubin.community/t/issues-with-ingestcalib-py-using-decam-images/5602/13?u](http://www.rubin.community/t/issues-with-ingestcalib-py-using-decam-images/5602/13?u) I only had to change two things: 1. As you advised, I used the following command to input the zero(bias) and flat(dome flat) data (which I had downloaded from an archive): butler ingest-raws REPO /path/to/raw/calibs/\*.fits.fz --transfer link 2. I appended “,/DECam/calib” right after “DECam/raw/all” in the following command: Before: pipetask run -b REPO -i DECam/raw/all -o DECam/raw/crosstalk-sources -p $CP\_PIPE\_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml --register-dataset-types After: pipetask run -b REPO -i DECam/raw/all,DECam/calib -o DECam/raw/crosstalk-sources -p $CP\_PIPE\_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml --register-dataset-types I will continue to follow the tutorial page that you provided. Thank you for your help. Best regards, Boseong