# New Tasks for Fake Source Insertion

**URL:** <https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722>\
**Category:** Data Management\
**Created:** [June 10, 2019, 7:42pm UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722 "2019-06-10T19:42:51Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![sophiereed](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sophiereed/32/777_2.png) [@sophiereed](https://www.rubin.community/u/sophiereed)\
**Post date:** [June 10, 2019, 7:42pm UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722/1 "2019-06-10T19:42:51Z")

</div>

There are now two new tasks for inserting fake sources. One for coadds (`insertFakes.py`) and one for single epoch images (`processCcdWithFakes.py`). Both tasks will look for a `deepCoadd_fakeSourceCat` as the standard input to draw the fakes from. I have been using one of these for the COSMOS patch downloaded from the UW sims database, the column names of which are the defaults set in the config. The required parameters are:

> **raColName:** The column name for the RA coordinate, default: raJ2000, in radians  
> **decColName:** The column name for the Dec. coordinate, default: decJ200, in radians  
> **diskHLR:** The column that contains the disk half light radius, default: DiskHalfLightRadius, in arcsecs  
> **bulgeHLR:** The column that contains the bulge half light radius, default: BulgeHalfLightRadius, in arcsecs  
> **magVar:** The column name for the magnitude calculated taking variability into account, default: magVar (eg. imagVar), in AB mags  
> **nDisk:** The column that contains the sersic index of the disk component, default: disk\_n  
> **nBulge:** The column that contains the sersic index of the bulge component, default: bulge\_n  
> **aDisk:** The column that contains the semi major axis length of the disk component, default: a\_d  
> **aBulge:** The column that contains the semi major axis length of the bulge component, default: a\_b  
> **bDisk:** The column that contains the semi minor axis length of the disk component, default: b\_d  
> **bBulge:** The column that contains the semi minor axis length of the bulge component, default: b\_b  
> **paDisk:** The column that contains the PA of the disk component, default: pa\_disk, in degrees  
> **paBulge:** The column that contains the PA of the bulge component, default: pa\_bulge, in degrees  
> **sourceType:** If set to galaxy will use the previous config options to insert a galaxy, if set to star will add a star using the PSF from the image. **Update:** This can be a string/bytes of “star”/“galaxy” or 0 (star)/ 1 galaxy.

These can be specified via the command line as config options or in a config file. The config option `fakeType` allows the user to specify the path to a fits file that contains the above columns (`fakeType=/path/to/fits/file.fits`). This config option can also be used to specify whether a static (`fakeType=static`) or snapshot (`fakeType=snapshot`) catalogue is used. The snapshot catalogues for each calexp in a visit have magnitudes dependent on the MJD of the calexp they were generated for. They can be used for variability studies. The default is static, which contains only one magnitude which is independent of the date of the exposure. Stars and galaxies to be input are supplied via the same file. For stars all the parameters other than position, magnitude and sourceType can be set to anything as they will not be used.

The additions of galaxies is done using galsim as such it inherits so restrictions from galsim. There are restrictions on the minimum and maximum Sersic index that can be used; 0.3 \<= nBulge/nDisk \<= 6.2.

**Adding into single epoch images.**

You need a directory with the outputs from singleFrameDriver and jointcal/fgcm (if you want to use the improved calibrations). Then run:

`> processCcdWithFakes.py /datasets/hsc/repo/ --rerun where/your/inputs/are:where/you/want/the/outputs --id visit=1228 ccd=0..8^10..103 filter=HSC-I tract=9813`

Note that unlike `processCcd` or `singleFrameDriver` the id argument requires the tract key.  
If you want to use any of the external calibration options (jointcal or fgcm for photometry and jointcal for astrometry) then you can do that by setting some config options. If you want to use jointcal instead then replace “fgcm” in the example below with “jointcal”.

```
config.makeCoaddTempExp.doApplyExternalPhotoCalib=True
config.assembleCoadd.externalPhotoCalibName="fgcm"
config.assembleCoadd.assembleStaticSkyModel.doApplyExternalPhotoCalib=True
config.assembleCoadd.assembleStaticSkyModel.externalPhotoCalibName="fgcm"

```

This task also runs the processing steps from `processCcd` to produced a `fakes_calexp` and a `fakes_src` which are equivalent to the normal `calexp` and `src` but contain the fake sources and, for the calexp, the mask plane `FAKE` which covers the area that the image of the fake source was added to. This mask plane is propagated into two flags in the `fakes_src` (and subsequent coadd catalogues) `base_PixelFlags_flag_fake` for objects where any of their footprint overlaps the inserted fake and `base_PixelFlags_flag_fakeCenter` for any object whose centre pixel overlaps the fake.

There is also a driver script to run this that works like singleFrameDriver; processCcdWithFakesDriver. It takes the same arguments and configs as processCcdWithFakes.

Subsequent processing steps are run as normal except for the specification of config options.

`skyCorrection.py: -c calexpType=fakes_calexp`

For example:

`> skyCorrection.py /datasets/hsc/repo --rerun where/you/put/the/output/from/processCcdWithFakes --id visit=1228 filter=HSC-I tract=9813 -c calexpType=fakes_calexp`

For coaddDriver.py a config file containing the following config options is required:

```
config.hasFakes=True
config.makeCoaddTempExp.useMeasMosaic=False
config.detectCoaddSources.hasFakes=True
config.makeCoaddTempExp.hasFakes=True
config.assembleCoadd.hasFakes=True
config.calexpType=fakes_calexp

```

For example:  
`> coaddDriver.py /datasets/hsc/repo --rerun where/you/put/the/output/from/skyCorrection --id tract=9813 patch=2,2 filter=HSC-I --selectId visit=1228 filter=HSC-I tract=9813 -C cdConfigFakes.py`

Where cdConfigFakes.py is the file containing the required config options listed above. coaddDriver produces `fakes_deepCoadd_calexp` when run in this way (rather than `deepCoadd_calexp`).

For multiBandDriver.py the options below need to be in a config file:

```
config.hasFakes=True
config.measureCoaddSources.measurement.plugins['base_PixelFlags'].masksFpAnywhere.append('FAKE')
config.measureCoaddSources.measurement.plugins['base_PixelFlags'].masksFpCenter.append('FAKE')
config.forcedPhotCoadd.measurement.plugins['base_PixelFlags'].masksFpAnywhere.append('FAKE')
config.forcedPhotCoadd.measurement.plugins['base_PixelFlags'].masksFpCenter.append('FAKE')
config.deblendCoaddSources.hasFakes=True
config.measureCoaddSources.hasFakes=True
config.forcedPhotCoadd.hasFakes=True

```

For example:

`> multiBandDriver.py /datasets/hsc/repo --rerun where/you/put/the/outputs/from/coaddDriver --id tract=9813 filter=HSC-I patch=2,2 -C mbdConfigFakes.py`

Where mbdConfigFakes.py contains the config options specified above.

After these steps you should have a coadd with fakes in it and measurement performed on it. Following the examples above, this will only be for one visit in the i band.

**Adding into coadds.**

This requires everything up to the production of `deepCoadds` to have been done. `deepCoadds` are produced during coaddDriver.py which can be run with `doDetection=False` to produce these but not run all the subsequent processing steps. This is faster.

The first thing to run is:

`> insertFakes.py /datasets/hsc/repo --rerun where/the/deepCoadds/are:where/you/want/the/outputs/to/go --id tract=9813 patch=2,2 filter=HSC-G^HSC-R^HSC-I^HSC-Z^HSC-Y`

This adds fakes to the `deepCoadds` (producing `fakes_deepCoadd`) and has the same options listed at the top of these instructions. Inserting variable sources into coadds doesn’t make much sense so it isn’t really supported for this step. Specifying your own catalogue name still works though.

After fakes have been added, multiBandDriver can be run with `doDetection=True` to perform the the processing steps not done by coaddDriver previously. This will produce `fakes_deepCoadd_calexp` rather than the `deepCoadd_calexp` normally produced.

multiBandDriver needs the config options below included in a config file.

```
config.measureCoaddSources.propagateFlags.flags={'calib_psf_candidate': 0.2, 'calib_psf_used': 0.2, 'calib_psf_reserved': 0.2, 'calib_photometry_used': 0.2, 'calib_photometry_reserved': 0.2}
config.hasFakes=True
config.measureCoaddSources.measurement.plugins['base_PixelFlags'].masksFpAnywhere.append('FAKE')
config.measureCoaddSources.measurement.plugins['base_PixelFlags'].masksFpCenter.append('FAKE')
config.forcedPhotCoadd.measurement.plugins['base_PixelFlags'].masksFpAnywhere.append('FAKE')
config.forcedPhotCoadd.measurement.plugins['base_PixelFlags'].masksFpCenter.append('FAKE')
config.deblendCoaddSources.hasFakes=True
config.measureCoaddSources.hasFakes=True
config.forcedPhotCoadd.hasFakes=True
config.doDetection=True
config.detectCoaddSources.hasFakes=True

```

For example:  
`> multiBandDriver.py /datasets/hsc/repo --rerun where/you/put/the/outputs/from/coaddDriver --id tract=9813 filter=HSC-I patch=2,2 -C mbdConfigFakesCoadd.py`

Where the needed config options are in mbdConfigFakesCoadd.py.

**Adding Sources Selectively**  
If you wish to add a subselection of the sources from your given catalog then you can do so by specifying the config option `sourceSelectionColName`. For example:

`-c fakeType="/my/favourite/source/catalogue/full/of/things.fits" -c sourceSelectionColName="fakeQuasars"`

This will only add the objects that are set to True in this column in the input catalogue.

**Adding Images Directly**  
If you want to add images directly then you need to set the config options:  
`-c insertImages = True` and `-c fakeType="/catalogue/with/image/details.fits"`.

The catalogue given needs to have the following columns:

> **band + imFilename:** The filename of the image to be inserted at the specified position, e.g. gimFilename  
> **sourceType:** Whether it is a star or galaxy  
> **band + magVar:** The column name for the magnitude calculated taking variability into account, e.g. imagVar, in AB mags  
> **raJ2000:** The R.A to insert the centre of the image at.  
> **decJ2000:** The declination to insert the centre of the image at.

This will add the images specified at the locations specified by the ra and dec given. At the moment you cannot add both images and sources from a catalogue to the same image.

**Sample Data**  
Sample fakes data can be found in /datasets/hsc/repo/rerun/rivate/sreed/hscRerun/cosmosFakes/fakesAddedVisit for data with fakes added into the calexps. Please take care not to overwrite things in this repo if using it for testing purposes.

---

<div class="post-metadata">

**Author:** ![parejkoj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/parejkoj/32/138_2.png) [@parejkoj](https://www.rubin.community/u/parejkoj)\
**Post date:** [June 12, 2019, 9:01pm UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722/2 "2019-06-12T21:01:19Z")

</div>

Would it be possible to rename `insertFakes` and `processFakes` to something that better represents what they do? Assuming I’m understanding the above text correctly, maybe something like `insertFakesInCoadds` and `insertFakesInExposures`, respectively? The existing names were not helpful in the UW group understanding the difference during today’s `SourceDetection` brownbag.

---

<div class="post-metadata">

**Author:** ![swinbank](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/swinbank/32/24_2.png) [@swinbank](https://www.rubin.community/u/swinbank)\
**Post date:** [June 28, 2019, 10:54pm UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722/3 "2019-06-28T22:54:14Z")

</div>

I happened to just stumble across [https://github.com/lsst/pipe\_tasks/pull/301](https://github.com/lsst/pipe_tasks/pull/301).

---

<div class="post-metadata">

**Author:** ![sophiereed](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sophiereed/32/777_2.png) [@sophiereed](https://www.rubin.community/u/sophiereed)\
**Post date:** [July 12, 2019, 4:09pm UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722/4 "2019-07-12T16:09:20Z")

</div>

This has been renamed and the instructions updated.

---

<div class="post-metadata">

**Author:** ![jeffcarlin](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jeffcarlin/32/491_2.png) [@jeffcarlin](https://www.rubin.community/u/jeffcarlin)\
**Post date:** [January 20, 2020, 9:47pm UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722/5 "2020-01-20T21:47:27Z")

</div>

Hi @sophiereed - are there also tools for generating fake source catalogs to be inserted, or is that left to the user to implement?

---

<div class="post-metadata">

**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\
**Post date:** [January 21, 2020, 2:56am UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722/6 "2020-01-21T02:56:34Z")

</div>

No, I asked too. I might be implementing something like that soon.

---

<div class="post-metadata">

**Author:** ![sophiereed](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sophiereed/32/777_2.png) [@sophiereed](https://www.rubin.community/u/sophiereed)\
**Post date:** [January 21, 2020, 3:15pm UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722/7 "2020-01-21T15:15:30Z")

</div>

I have some code that makes fake source catalogues but it isn’t official and needs a little TLC. The TLC is on my to do list. It was lower down but could be moved up.

---

<div class="post-metadata">

**Author:** ![jeffcarlin](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jeffcarlin/32/491_2.png) [@jeffcarlin](https://www.rubin.community/u/jeffcarlin)\
**Post date:** [January 21, 2020, 4:54pm UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722/8 "2020-01-21T16:54:19Z")

</div>

No rush - just checking. For now I’ll see if reformatting fake source catalogs I already have (for SynPipe runs) will work with the new code.

---

<div class="post-metadata">

**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\
**Post date:** [March 2, 2020, 7:38pm UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722/9 "2020-03-02T19:38:28Z")

</div>

Note that the `FAKE` mask plane gets set for the entire postage stamp of the fake sources (maybe 40 pixels, square, for PSFs), not just the significant or central pixels. This means that the `base_PixelFlags_flag_fake*` flags are not as useful as they might be. For identifying fake sources in the output catalogs, it seems to me that it’s better to rely on cross-correlating with the input catalog than to use flags in the output catalogs.

---

<div class="post-metadata">

**Author:** ![sophiereed](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sophiereed/32/777_2.png) [@sophiereed](https://www.rubin.community/u/sophiereed)\
**Post date:** [March 4, 2020, 3:55pm UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722/10 "2020-03-04T15:55:51Z")

</div>

This was by design, I wanted to know all the objects that might be affected by the fake being inserted on top of them. Having a better way of identifying the fake sources than just matching to the input catalogue would be great but I am not certain the best way to go about it. I don’t want to miss catastrophic failures for example.

---

<div class="post-metadata">

**Author:** ![laura.inno](https://avatars.discourse-cdn.com/v4/letter/l/a88e4f/32.png) [@laura.inno](https://www.rubin.community/u/laura.inno)\
**Post date:** [February 8, 2022, 12:24pm UTC](https://www.rubin.community/t/new-tasks-for-fake-source-insertion/3722/11 "2022-02-08T12:24:38Z")

</div>

Hello! Is there a notebook or some instructions on how to use these functions and how to prepare a catalog of fake sources to be injected?