# Pan-STARRS reference catalog in LSST format **URL:** \ **Category:** DM Notifications\ **Tags:** refcat, panstarrs\ **Created:** [January 21, 2017, 1:56am UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572 "2017-01-21T01:56:44Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [January 21, 2017, 1:56am UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572/1 "2017-01-21T01:56:44Z")
The Pan-STARRS 3pi stack data (images and catalogs) were [released to the world](http://panstarrs.stsci.edu) as DR1 on 2016 December 19. This provides a high-quality network of calibration sources over three quarters of the sky to a modest depth. Telescopes both large and small (depths overlapping i ~ 17 to 21 mag) observing fields north of Dec=-30 deg in the optical will have good PS1 calibration sources in the field, observed through the same column as the target sources, allowing simple relative calibration for astrometry, photometry and spectroscopy. To allow us (and others) to take advantage of this, I have produced a point source catalog in the LSST reference catalog format and also in the [astrometry.net](http://astrometry.net) format we have used in the past. Details on the contents, construction and use of the catalog are available from the [`README.txt` file](https://lsst-web.ncsa.illinois.edu/~pprice/ps1_pv3_3pi_20170110/README.txt) included in the catalog, which I will summarise here. ## Overview This reference catalog contains astrometry and grizy photometry for 2,990,470,528 point sources at Dec \> -30 deg to i ~ 22.5 mag. Users of this catalog should include the following acknowledgement in publications: > The Pan-STARRS1 Surveys (PS1) have been made possible through contributions of the Institute for Astronomy, the University of Hawaii, the Pan-STARRS Project Office, the Max-Planck Society and its participating institutes, the Max Planck Institute for Astronomy, Heidelberg and the Max Planck Institute for Extraterrestrial Physics, Garching, The Johns Hopkins University, Durham University, the University of Edinburgh, Queen’s University Belfast, the Harvard-Smithsonian Center for Astrophysics, the Las Cumbres Observatory Global Telescope Network Incorporated, the National Central University of Taiwan, the Space Telescope Science Institute, the National Aeronautics and Space Administration under Grant No. NNX08AR22G issued through the Planetary Science Division of the NASA Science Mission Directorate, the National Science Foundation under Grant No. AST-1238877, the University of Maryland, and Eotvos Lorand University (ELTE), the Los Alamos National Laboratory, and the Gordon and Betty Moore Foundation. Relevant papers for information and citation include: - [Chambers et al., “The Pan-STARRS1 Surveys”, 2016arXiv161205560C](http://adsabs.harvard.edu/abs/2016arXiv161205560C) - [Magnier et al., “The Pan-STARRS Data Processing System”, 2016arXiv161205240M](http://adsabs.harvard.edu/abs/2016arXiv161205240M) - [Waters et al., “Pan-STARRS Pixel Processing: Detrending, Warping, Stacking”, 2016arXiv161205245W](http://adsabs.harvard.edu/abs/2016arXiv161205245W) - [Magnier et al., “Pan-STARRS Pixel Analysis: Source Detection and Characterization”, 2016arXiv161205244M](http://adsabs.harvard.edu/abs/2016arXiv161205244M) - [Magnier et al., “Pan-STARRS Photometric and Astrometric Calibration”, 2016arXiv161205242M](http://adsabs.harvard.edu/abs/2016arXiv161205242M) - [Flewelling et al., “The Pan-STARRS1 Database and Data Products”, 2016arXiv161205243F](http://adsabs.harvard.edu/abs/2016arXiv161205243F) - [Tonry et al., “The Pan-STARRS1 Photometric System”, 2012ApJ…750…99T](http://adsabs.harvard.edu/abs/2012ApJ...750...99T) - [Schlafly et al., “Photometric Calibration of the First 1.5 Years of the Pan-STARRS1 Survey”, 2012ApJ…756..158S](http://adsabs.harvard.edu/abs/2012ApJ...756..158S) ## LSST-style format The LSST-style format uses FITS files divided by HTM. You can download the LSST-style catalog [here](https://lsst-web.ncsa.illinois.edu/~pprice/ps1_pv3_3pi_20170110/). Be warned that it’s rather large: 423 GB. You can download a subset of the catalog by grabbing the appropriate FITS files; see the section in the [`README.txt`](https://lsst-web.ncsa.illinois.edu/~pprice/ps1_pv3_3pi_20170110/README.txt) entitled “Field selections” for details. You need to install it in your data repo: ``` ln -s /path/to/ps1_pv3_3pi_20170110/ /path/to/DATA_REPO/ref_cats ``` Then you need to adjust your configuration. Here’s a configuration for `processCcd.py` (either to put in your obs package as `config/processCcd.py`, or to specify on the command-line with `--configfile`). See also the section below on “Common configuration” for `colorterms` and `filterMap`. ``` from lsst.meas.algorithms import LoadIndexedReferenceObjectsTask config.calibrate.astromRefObjLoader.retarget(LoadIndexedReferenceObjectsTask) config.calibrate.astromRefObjLoader.ref_dataset_name = "ps1_pv3_3pi_20170110" config.calibrate.photoRefObjLoader.retarget(LoadIndexedReferenceObjectsTask) config.calibrate.photoRefObjLoader.ref_dataset_name = "ps1_pv3_3pi_20170110" config.calibrate.photoCal.photoCatName = "ps1_pv3_3pi_20170110" ``` ## Astrometry.net-style format The astrometry.net-style format uses custom FITS files divided by HEALPix and indexed using tools provided by [astrometry.net](http://astrometry.net). You can download the astrometry.net-style catalog [here](https://lsst-web.ncsa.illinois.edu/~pprice/ps1_pv3_3pi_20170110-and/). This format is a bit smaller than the LSST-style format (it does not include as much detail, e.g., proper motions), but is still quite hefty at 396 GB. If you want a subset of the catalog, you will need to figure out the appropriate HEALPix using the `get-healpix` tool from [astrometry.net](http://astrometry.net) (I’m not entirely sure whether it’s the “XY”, “RING” or “NESTED” scheme you want) and grab the `*_and__[012].fits` files. Or you can grab the appropriate files from the LSST-style format and generate your own astrometry.net-style catalog using the included scripts; see the [`README.txt`](https://lsst-web.ncsa.illinois.edu/~pprice/ps1_pv3_3pi_20170110/README.txt) under “[Astrometry.net](http://Astrometry.net) format” for details. The downloaded catalog doesn’t need to be installed in the data repo, but it should be declared to eups: ``` eups declare astrometry_net_data ps1_pv3_3pi_20170110-and -r /path/to/ps1_pv3_3pi_20170110-and ``` Then, to use it, you need to: ``` setup -j astrometry_net_data ps1_pv3_3pi_20170110-and ``` No configuration changes are currently required, because astrometry.net-style catalogs are currently the default (this will likely change soon), but do see the section below on “Common configuration” for `colorterms` and `filterMap`. ## Common configuration Regardless of which style of catalog you use, you need to set the `colorterms` and `filterMap` appropriately for your camera. Here is an example configuration for `processCcd.py`: ``` # These colorterms are for HSC, included as an example colorterms = config.calibrate.photoCal.colorterms from lsst.pipe.tasks.colorterms import ColortermDict, Colorterm colorterms.data["ps1*"] = ColortermDict(data={ 'g': Colorterm(primary="g", secondary="r", c0=0.00730066, c1=0.06508481, c2=-0.01510570), 'r': Colorterm(primary="r", secondary="i", c0=0.00279757, c1=0.02093734, c2=-0.01877566), 'r2': Colorterm(primary="r", secondary="i", c0=0.00117690, c1=0.00003996, c2=-0.01667794), 'i': Colorterm(primary="i", secondary="z", c0=0.00166891, c1=-0.13944659, c2=-0.03034094), 'i2': Colorterm(primary="i", secondary="z", c0=0.00180361, c1=-0.18483562, c2=-0.02675511), 'z': Colorterm(primary="z", secondary="y", c0=-0.00907517, c1=-0.28840221, c2=-0.00316369), 'y': Colorterm(primary="y", secondary="z", c0=-0.00156858, c1=0.14747401, c2=0.02880125), }) # For the HSC r2 and i2 filters, use the r and i values from the catalog for refObjLoader in (config.calibrate.astromRefObjLoader, config.calibrate.photoRefObjLoader, config.charImage.refObjLoader, ): refObjLoader.filterMap['r2'] = 'r' refObjLoader.filterMap['i2'] = 'i' ``` * * * EDIT Aug 2020: NCSA/LSST doesn’t seem to be running their web server at the moment. If you can’t access it, there’s a copy of the catalog available from [http://tigress-web.princeton.edu/~pprice/ps1\_pv3\_3pi\_20170110/](http://tigress-web.princeton.edu/~pprice/ps1_pv3_3pi_20170110/) . ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [January 23, 2017, 3:58pm UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572/2 "2017-01-23T15:58:17Z")
Rick White (STScI) sent around this summary of the PS1 archive usage since the data release: > I thought you would be interested to hear about the usage of the PS1 public archive at STScI. It has been 1 month since the PS1 data release on December 19. In that time: > 15.4 TB of images have been downloaded to 15,500 different IP addresses. > 3.2 million image cutouts (“postage stamps”) have been extracted. > 2.5 million Virtual Observatory catalog queries have been done by 70 different IP addresses, returning information on 580 million objects. > 644,000 database queries have been executed through our CasJobs interface by 99 different users, returning more than 7.6 billion rows of data. The total data volume downloaded from CasJobs is 4.4 TB. > 16,100 catalog searches have been done using the MAST web search form. > For comparison, the image data volume in the current public release is about 100 TB, and the catalog includes approximately 10 billion objects in a 20 TB database. > Despite the heavy usage, I believe there have been no negative impacts on other STScI archive usage or on our internet access. Our IT infrastructure has shown a lot of resiliency in handling the load with little difficulty. > This was only possible through the efforts of the PS1 team, and the huge popularity of the data is a testament to the value of the survey! ---
**Author:** ![jrmullaney](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jrmullaney/32/502_2.png) [@jrmullaney](https://www.rubin.community/u/jrmullaney)\ **Post date:** [October 7, 2019, 4:36pm UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572/3 "2019-10-07T16:36:05Z")
I realise that this is a fairly old thread now, but I thought it may be the best place to put this for interested parties to find… We’ve been using the Pan-STARRS catalogue for about a year now, and it’s producing excellent results (thanks @price). However, because it is the _whole_ PS1 catalogue and because we require the full survey coverage, we’ve found it a bit “non-portable”; indeed, I’ve always refrained from putting it on my laptop due to its size. Since we don’t need such a deep reference catalogue for our purposes, we’ve created a smaller LSST-format catalogue by extracting only sources brighter than 19th mag (g-band) which, after gzip compression, is “only” 29GB (it’s roughly double that after decompression). If a shallower reference is suitable for your purposes, and you wish to use this smaller catalogue, you can grab it from: [http://astro.shef.ac.uk/lsstps1/](http://astro.shef.ac.uk/lsstps1/) To help prevent bot-downloads, we’ve put it behind a simple login: username: lsstps1 password: UoSlsst Download it, unzip and untar it, and use it in the same way @price suggests in his earlier post. While we’ve done some (limited) testing and found it reproduces the results of the larger catalogue exactly, we haven’t tested it to exhaustion, and so accept no responsibility/liability etc. Clearly, if you use this smaller catalogue then the same acknowledgements apply as in @price’s original post. Thanks again to @price for the far greater effort of generating and making the original LSST-format catalogue freely available in the first place. ---
**Author:** ![lskelvin](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/lskelvin/32/2537_2.png) [@lskelvin](https://www.rubin.community/u/lskelvin)\ **Post date:** [May 30, 2021, 6:33pm UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572/4 "2021-05-30T18:33:57Z")
As above, I realize that this is a fairly old thread, but I wanted to post this question here in case it’s useful to anyone else. I’m trying to follow these instructions in a `gen3` world, and I have a few questions about setting up my ref cat for use by a small group of users on a local machine. Specifically, I’m not sure of the best way to ingest the `ps1_pv3_3pi_20170110` catalogue and have it referenced as a collection. I have the files [linked above](http://tigress-web.princeton.edu/~pprice/ps1_pv3_3pi_20170110/) downloaded, and have them sym-linked into my butler repository, but I’m unsure of the best way to register these data as a ref cat collection. Are there any `gen3` tools that will help set this up? ---
**Author:** ![parejkoj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/parejkoj/32/138_2.png) [@parejkoj](https://www.rubin.community/u/parejkoj)\ **Post date:** [May 31, 2021, 8:39pm UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572/5 "2021-05-31T20:39:30Z")
We have no gen3 refcat ingest tools; refcats have to be converted from a gen2 repo. Follow [DM-29543](https://jira.lsstcorp.org/browse/DM-29543) for upates. ---
**Author:** ![Paula](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/paula/32/1441_2.png) [@Paula](https://www.rubin.community/u/Paula)\ **Post date:** [September 2, 2021, 3:30pm UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572/6 "2021-09-02T15:30:35Z")
Hi! I want to share that I am trying to download the content from the catalog [available](http://tigress-web.princeton.edu/~pprice/ps1_pv3_3pi_20170110/), but not entirely. Taking a look at the README.txt from testdata\_ci\_hsc/ps1\_pv3\_3pi\_20170110 of the tutorial of [Getting Started with the LSST Science Pipelines](https://pipelines.lsst.io/getting-started/) It proposed a method of selecting fields: ``` Field selections ---------------- The 423 GB size of this catalog may be prohibitive for individual use. In that case, it's possible to copy individual files required for particular regions of interest. Just grab the .fits files (plus config.py and this README.txt file), where the list of shards is calculated from this code: from lsst.meas.algorithms.htmIndexer import HtmIndexer from lsst.afw.coord import IcrsCoord from lsst.afw.geom import degrees def getShards(ra, dec, radius): htm = HtmIndexer(depth=7) shards, onBoundary = htm.get_pixel_ids(IcrsCoord(ra*degrees, dec*degrees), radius*degrees) return shards ``` Nowadays I checked that the attributes `htm.get_pixel_ids` and `IcrsCoord` are no longer part if the lsst pipeline software, so I was able to accommodate the `getShards` function to the current attributes available: ``` from lsst.meas.algorithms.htmIndexer import HtmIndexer import lsst.geom as geom def getShards(lon,lat, radius): # lon goes from 0 to 2pi htm = HtmIndexer(depth=7) shards, onBoundary = htm.getShardIds(geom.SpherePoint(lon*geom.degrees, lat*geom.degrees), radius*geom.degrees) return shards ``` when I print the results of `getShards(3,0,1)` I get: > [131108 131111 253976 253979 131080 131081 131083 131109 131110 131116 > 131118 131119 131124 131126 131127 253956 253958 253959 253977 253978 > 253980 253981 253983 254008 254009 254011] It looks like it’s working nice! Happy to receive any comments 🙂 ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [September 2, 2021, 3:48pm UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572/7 "2021-09-02T15:48:27Z")
> [@Paula](#): > > It looks like it’s working nice! Happy to receive any comments Well done, and thank you! ---
**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\ **Post date:** [September 2, 2021, 4:05pm UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572/8 "2021-09-02T16:05:51Z")
> [@Paula](#): > > tutorial of [Getting Started with the LSST Science Pipelines](https://pipelines.lsst.io/getting-started/) I Yesterday we released the Gen3 version of this tutorial and it can be found at > **[Getting started with the LSST Science Pipelines — LSST Science Pipelines](https://pipelines.lsst.io/v/weekly/getting-started/index.html)** ---
**Author:** ![parejkoj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/parejkoj/32/138_2.png) [@parejkoj](https://www.rubin.community/u/parejkoj)\ **Post date:** [October 21, 2021, 7:46pm UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572/9 "2021-10-21T19:46:34Z")
I’ve generated the gen3 refcat ingest index `.ecsv` files on lsst-devl. See this post for more details: [Gaia DR2 reference catalog in LSST format - #18 by parejkoj](http://www.rubin.community/t/gaia-dr2-reference-catalog-in-lsst-format/3901/18) ---
**Author:** ![ameisner](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ameisner/32/2392_2.png) [@ameisner](https://www.rubin.community/u/ameisner)\ **Post date:** [April 12, 2022, 1:29am UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572/10 "2022-04-12T01:29:02Z")
@price – I noticed that this set of reference catalogs includes shards covering nearly all of the southern sky, even below the Dec ~ -30 deg cutoff of Pan-STARRS. Do you know what survey(s) the sources in these southern reference catalogs come from? For instance: [https://tigress-web.princeton.edu/~pprice/ps1\_pv3\_3pi\_20170110/168357.fits](https://tigress-web.princeton.edu/~pprice/ps1_pv3_3pi_20170110/168357.fits) Has a few thousand objects, with Dec ~ -77 deg. Thanks very much… ---
**Author:** ![price](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/price/32/488_2.png) [@price](https://www.rubin.community/u/price)\ **Post date:** [April 12, 2022, 12:46pm UTC](https://www.rubin.community/t/pan-starrs-reference-catalog-in-lsst-format/1572/11 "2022-04-12T12:46:57Z")
It’s been a while since I generated that catalog, and even longer since I was involved with Pan-STARRS. And the cluster serving the data is down right now so I can’t look at that example. My guess is that it’s Gaia, which was used to seed the catalog and has somehow slipped through, but I don’t know if that’s right or even makes sense.