# Correct use of ingestImages.py for calibrations

**URL:** <https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069>\
**Category:** Support\
**Created:** [March 17, 2020, 10:23pm UTC](https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069 "2020-03-17T22:23:45Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![fraserw](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fraserw/32/877_2.png) [@fraserw](https://www.rubin.community/u/fraserw)\
**Post date:** [March 17, 2020, 10:23pm UTC](https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069/1 "2020-03-17T22:23:45Z")

</div>

I’ve been struggling to figure out the correct use of ingestImages.py to ingest HSC calibration data. I have raw images in $CI\_HSC\_DIR/raw and the ingest works perfectly with the command listed in the tutorial.

I have flats and biases in $CI\_HSC\_DIR/cals, but not sure what flags to provide for ingest to create the CALIBS directory. Obviously run as

> ingestImages.py DATA $CI\_HSC\_DIR/cals/\*.fits --mode=link

is wrong because processCcd.py throws the warning: HscMapper WARN: Unable to find calib root directory before failing catastrophically.

Thanks for the help!

---

<div class="post-metadata">

**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\
**Post date:** [March 18, 2020, 3:32pm UTC](https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069/2 "2020-03-18T15:32:41Z")

</div>

I think you want `ingestCalibs.py`, if the flats and biases are “master” coadded images produced by the HSC/LSST code and not “raw” telescope images, but an HSC expert should probably weigh in here.

---

<div class="post-metadata">

**Author:** ![fraserw](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fraserw/32/877_2.png) [@fraserw](https://www.rubin.community/u/fraserw)\
**Post date:** [March 18, 2020, 3:46pm UTC](https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069/3 "2020-03-18T15:46:12Z")

</div>

Thanks @ktl

What I have are the raw calib frames and not masters. As it is, I need the code to produce those masters, and I can’t ingest the frames in the first place.

---

<div class="post-metadata">

**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\
**Post date:** [March 18, 2020, 5:16pm UTC](https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069/4 "2020-03-18T17:16:25Z")

</div>

If these are raw calibration frames, then they should indeed be ingested with `ingestImages`. I’d have to see what the actual error message is; the `WARN` you mention is exactly that: merely a warning and not an actual problem.

Oh: the error is coming from `processCcd`? Then yes, you need to run master calibration generation code based on your raw calibrations (or get masters from somewhere else) and ingest them with `ingestCalibs`. Doing that calibration processing is far beyond me 🙂

---

<div class="post-metadata">

**Author:** ![fraserw](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fraserw/32/877_2.png) [@fraserw](https://www.rubin.community/u/fraserw)\
**Post date:** [March 18, 2020, 5:44pm UTC](https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069/5 "2020-03-18T17:44:46Z")

</div>

I see. Thanks. Yes I only have the raw calibration frames. I know Gemini automatically creates calibration masters on a ~monthly basis, but I am unaware of any service that would provide something different for the HSC.

FWIW I tried ingestCalibs on the raw frames, with no luck. The following error is thrown, suggesting that it is expecting fits files of a different format (data arrays in different locations in the fits file).

```auto
(lsst-scipipe-4d7b902) ubuntu@lsst-test:/media/NHproc$ ingestCalibs.py DATA $CI_HSC_DIR/CALIB/smoka/P*_data/*fits --mode=link --validity 60
root INFO: Loading config overrride file '/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_subaru/19.0.0+2/config/ingestCalibs.py'
HscMapper WARN: Unable to find calib root directory
CameraMapper INFO: Loading exposure registry from /media/NHproc/DATA/registry.sqlite3
Traceback (most recent call last):
  File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/bin/ingestCalibs.py", line 3, in <module>
    IngestCalibsTask.parseAndRun()
  File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingest.py", line 425, in parseAndRun
    task.run(args)
  File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingestCalibs.py", line 220, in run
    with self.register.openRegistry(calibRoot, create=args.create, dryrun=args.dryrun) as registry:
  File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingestCalibs.py", line 90, in openRegistry
    return RegisterTask.openRegistry(self, directory, create, dryrun, name)
  File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingest.py", line 294, in openRegistry
    registryName = os.path.join(directory, name)
  File "/home/ubuntu/lsst_stack/python/miniconda3-4.7.10/envs/lsst-scipipe-4d7b902/lib/python3.7/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

```

---

<div class="post-metadata">

**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\
**Post date:** [March 18, 2020, 5:46pm UTC](https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069/6 "2020-03-18T17:46:54Z")

</div>

Yes, `ingestCalibs` on the raw frames definitely won’t work properly. It fails in your case because the `CALIB` directory doesn’t exist. If you created it manually, it might appear to work but not actually do the right thing, or it might fail. In any case, apologies for the inscrutable error messages.

---

<div class="post-metadata">

**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\
**Post date:** [March 24, 2020, 12:55am UTC](https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069/7 "2020-03-24T00:55:49Z")

</div>

When were your science frames taken? I might have some pre-built calibs you could use without having to worry about building them yourself.

---

<div class="post-metadata">

**Author:** ![fraserw](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/fraserw/32/877_2.png) [@fraserw](https://www.rubin.community/u/fraserw)\
**Post date:** [March 24, 2020, 2:04am UTC](https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069/8 "2020-03-24T02:04:31Z")

</div>

The data I am currently playing with are June 2014, but we have multiple upcoming runs that I am trying to get ready for aka. familiarize myself with all parts of the pipeline.

@price Paul, do you have a machine automatically producing flats, biases, and such?

---

<div class="post-metadata">

**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\
**Post date:** [March 24, 2020, 2:26am UTC](https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069/9 "2020-03-24T02:26:45Z")

</div>

I can definitely cover June 2014 with existing calibs if you like, but it sounds like you want to learn. Which tutorial are you following?

I don’t have anything routinely generating calibs. I usually just put in a few days worth of effort once a year or two.

---

<div class="post-metadata">

**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\
**Post date:** [April 14, 2020, 4:09pm UTC](https://www.rubin.community/t/correct-use-of-ingestimages-py-for-calibrations/4069/10 "2020-04-14T16:09:19Z")

</div>

You might want to look at the log I kept from creating calibs earlier this year: [http://tigress-web.princeton.edu/~pprice/CALIB-20200115/calib-20200115.txt](http://tigress-web.princeton.edu/~pprice/CALIB-20200115/calib-20200115.txt)

It’s probably a larger scale production than you care about, but hopefully there will be some helpful guidance in there.