# Noirlab Data Lab access in Jupyter Notebooks **URL:** \ **Category:** Support\ **Created:** [July 30, 2024, 11:06pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926 "2024-07-30T23:06:06Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![sdcorle1](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sdcorle1/32/3052_2.png) [@sdcorle1](https://www.rubin.community/u/sdcorle1)\ **Post date:** [July 30, 2024, 11:06pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/1 "2024-07-30T23:06:06Z")
This may be the wrong place to raise this issue. I am attempting to use code extracted from a Noirlab notebook to build a new notebook that would access astrometry data for processing. The Noirlab code uses Data Lab (dl) methods and I attempted to pip install dl. I get a message “Defaulting to user installation because normal site-packages is not writeable” that prevents the install. Should I be putting this issue into GitHub? ---
**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\ **Post date:** [July 30, 2024, 11:08pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/2 "2024-07-30T23:08:08Z")
> [@sdcorle1](#): > > Defaulting to user installation because normal site-packages is not writeable That message is telling you it is trying to install it locally. It’s not telling you it failed. ---
**Author:** ![sdcorle1](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sdcorle1/32/3052_2.png) [@sdcorle1](https://www.rubin.community/u/sdcorle1)\ **Post date:** [July 30, 2024, 11:13pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/3 "2024-07-30T23:13:26Z")
Thanks for the quick response. After the pip install, I try to import the methods associated with dl, but the response is: "—\> 15 from dl import authClient as ac, queryClient as qc 17 # Get helpers for various convenience function 18 from dl.helpers.utils import convert ModuleNotFoundError: No module named ‘dl’ " It was this message that caused me to believe that the install had failed. I also looked for a dl folder in the python site-packages folder and couldn’t find anything. ---
**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\ **Post date:** [July 30, 2024, 11:25pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/4 "2024-07-30T23:25:04Z")
What did the pip install say after the warning? ---
**Author:** ![frossie](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/frossie/32/9_2.png) [@frossie](https://www.rubin.community/u/frossie)\ **Post date:** [July 31, 2024, 3:38am UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/5 "2024-07-31T03:38:04Z")
1. To be clear, are you using `pip install --user` ? 2. Also are you sure there is a pypi package called dl that is the one you want? Have you tried it from eg. your own laptop and it works? I may be blind but dl on pypi seems to be this: [GitHub - mrdanbrooks/dl: dl - rm with forgivness](https://github.com/mrdanbrooks/dl) which is not what you are looking for ---
**Author:** ![sdcorle1](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sdcorle1/32/3052_2.png) [@sdcorle1](https://www.rubin.community/u/sdcorle1)\ **Post date:** [July 31, 2024, 3:53am UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/6 "2024-07-31T03:53:49Z")
Have a look at the Imports and Setup section of this notebook: [notebooks-latest/01\_GettingStartedWithDataLab/02\_GettingStartedWithDataLab.ipynb at master · astro-datalab/notebooks-latest · GitHub](https://github.com/astro-datalab/notebooks-latest/blob/master/01_GettingStartedWithDataLab/02_GettingStartedWithDataLab.ipynb) There is no install in this code, but the import from dl is there and it must work in the Noirlab environment. ---
**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\ **Post date:** [July 31, 2024, 3:45pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/7 "2024-07-31T15:45:26Z")
> [@frossie](#): > > `pip install --user` ? Modern pip (very recent it seems) is clever enough to not need `--user`. The warning message in the initial report was pip saying that it couldn’t install into the read-only location so is falling back to user install. That isn’t the error. What I don’t know is what pip said after that warning. (and yes, `pip install dl` will try to install [Client Challenge](https://pypi.org/project/dl/)) ---
**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\ **Post date:** [July 31, 2024, 3:46pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/8 "2024-07-31T15:46:39Z")
Data lab python packages seem to be at [noaodatalab · PyPI](https://pypi.org/project/noaodatalab/) ---
**Author:** ![sdcorle1](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sdcorle1/32/3052_2.png) [@sdcorle1](https://www.rubin.community/u/sdcorle1)\ **Post date:** [July 31, 2024, 4:02pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/9 "2024-07-31T16:02:25Z")
I confirmed that the code works fine if I sign into the Noirlab notebook environment. My objective is to bring access to GAIA data into the Rubin notebook environment. Maybe I am barking up the wrong tree. If you wanted to do this, i.e. have GAIA and Rubin data available in the same working environment, do you have any suggestions? ---
**Author:** ![sdcorle1](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sdcorle1/32/3052_2.png) [@sdcorle1](https://www.rubin.community/u/sdcorle1)\ **Post date:** [July 31, 2024, 4:07pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/10 "2024-07-31T16:07:10Z")
## I think that you wanted to know this: ModuleNotFoundError Traceback (most recent call last) Cell In[1], line 15 10 get\_ipython().run\_line\_magic(‘matplotlib’, ‘inline’) 12 # Data Lab and related imports 13 14 # You’ll need at least these for authenticating and for issuing database queries —\> 15 from dl import authClient as ac, queryClient as qc 17 # Get helpers for various convenience function 18 from dl.helpers.utils import convert ModuleNotFoundError: No module named ‘dl’ ---
**Author:** ![sdcorle1](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sdcorle1/32/3052_2.png) [@sdcorle1](https://www.rubin.community/u/sdcorle1)\ **Post date:** [July 31, 2024, 4:09pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/11 "2024-07-31T16:09:29Z")
Or, was it this: Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: dl in /home/sdcorle1/.local/lib/python3.11/site-packages (0.1.0) Collecting argparse\>=1.0 (from dl) Using cached argparse-1.4.0-py2.py3-none-any.whl.metadata (2.8 kB) Using cached argparse-1.4.0-py2.py3-none-any.whl (23 kB) DEPRECATION: nb-black 1.0.7 has a non-standard dependency specifier black\>=‘19.3’; python\_version \>= “3.6”. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of nb-black or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at [Deprecate legacy versions and version specifiers · Issue #12063 · pypa/pip · GitHub](https://github.com/pypa/pip/issues/12063) Installing collected packages: argparse Successfully installed argparse-1.4.0 Note: you may need to restart the kernel to use updated packages. ---
**Author:** ![sdcorle1](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sdcorle1/32/3052_2.png) [@sdcorle1](https://www.rubin.community/u/sdcorle1)\ **Post date:** [July 31, 2024, 4:17pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/12 "2024-07-31T16:17:36Z")
Sorry, I was out of sync on the messaging with you. The pip install noaodatalab did the trick. Thanks for very much for your patience. Problem solved. ---
**Author:** ![gpdf](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/gpdf/32/65_2.png) [@gpdf](https://www.rubin.community/u/gpdf)\ **Post date:** [August 1, 2024, 6:03am UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/13 "2024-08-01T06:03:51Z")
In the operations era, we will have a then-current Gaia data release directly available in the RSP on our own database and TAP servers, so that fast cross-queries can be done. For now… I’m not sure whether DataLab offers value-added services for the Gaia data, but what I generally do is just use PyVO to do queries against the ESA Gaia TAP server, [https://gea.esac.esa.int/tap-server/tap](https://gea.esac.esa.int/tap-server/tap) . Note also that it’s easy to access Gaia from the RSP Portal Aspect (though you didn’t ask about that): Select “General TAP” from the sidebar: ![image](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/2X/7/749ac7082a3e704fc51900d88a23753e3fb93baa.png) and then select Gaia on the resulting screen: ![image](https://us1.discourse-cdn.com/flex002/uploads/rubin/original/2X/b/b5d4cb919fd90d46dd80acdaee119df7f1326d04.png) This will issue queries directly against the ESA TAP service. ---
**Author:** ![galaxyumi331](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/galaxyumi331/32/2677_2.png) [@galaxyumi331](https://www.rubin.community/u/galaxyumi331)\ **Post date:** [August 1, 2024, 7:50pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/14 "2024-08-01T19:50:44Z")
@sdcorle1, If you want to explore the EAS Gaia TAP service option using PyVO in your NB following Gregory’s suggestion, [here](https://dp0-3.lsst.io/tutorials-dp0-3/rendered_nb_03_06.html) is a tutorial you can have a look, Sec. 4 is particularly relevant. ---
**Author:** ![sdcorle1](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sdcorle1/32/3052_2.png) [@sdcorle1](https://www.rubin.community/u/sdcorle1)\ **Post date:** [August 2, 2024, 4:12pm UTC](https://www.rubin.community/t/noirlab-data-lab-access-in-jupyter-notebooks/8926/15 "2024-08-02T16:12:08Z")
@gpdf @galaxyumi331 Thanks to both of you for these excellent suggestions. The news about the operational environment is great. From my early explorations, I am using the dl code to access the GAIA data. I’ll take a look to see what the TAP service can provide. TAP sounds like the better long term solution, especially when the GAIA data is available in the same environment. I have another question that derives from having GAIA and LSST data in the same TAP environment. While looking around in the GAIA documentation, I discovered that they have created a spreadsheet that cross-references exoplanet candidates to GAIA source ID’s. The source ID’s then lead to parallax and proper motion parameters. I ran a query using IN to compare the GAIA source table to a small subset of the spreadsheet and with a WHERE parameter that looked for RA and DEC within the DC2 boundaries. This query resulted in one candidate exoplanet within the boundaries. (I’m not greedy so this one is enough for my current purpose. ) I took the RA and DEC from the candidate exoplanet to the Rubin environment and tried to match against LSST source records. I got four reasonable matches. I know that the LSST DP0.2 data is synthetic. Nonetheless, this brought up my new question. When there is GAIA data in the operational LSST data environment, are you aware of any techniques that might be used to cross reference from one data source to the other?