# Developing Software with the DM Stack **URL:** \ **Category:** Data Management\ **Tags:** dm-dev, lsstsw\ **Created:** [August 24, 2015, 8:11pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126 "2015-08-24T20:11:47Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [August 24, 2015, 8:11pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/1 "2015-08-24T20:11:47Z")
This page describes a method of installing and maintaining a DM software stack that allows you to easily keep up to date with the current master of packages and to push your own changes. It was updated 2015-11-04 to support git-lfs packages such as afwdata. ## Initial Installation - Set up a git credential manager, e.g. as per afwdata’s [README.md](https://github.com/lsst/afwdata/blob/master/README.md). - (alternately: edit the file lsstsw/etc/repos.yaml to change all instances of [https://github.com/lsst](https://github.com/lsst), except afwdata,to [git@github.com](mailto:git@github.com):lsst. This works fine if you don’t plan to push to afwdata, and doesn’t require using the credential manager.) - Clone the [lsstsw](https://github.com/lsst/lsstsw) package into the location of your choice: `git clone git@github.com:lsst/lsstsw.git` - Install lsstsw and rebuild lsst\_apps using these [instructions](https://developer.lsst.io/build-ci/lsstsw.html). Note that this pulls over all of afwdata, which is a lot of data. If you already have a cloned copy you may be able to avoid this by making a symlink to it from `lsstsw/build/afwdata`. - Eups tag the current build as “current” using command: `eups tags --clone b`_NNN_ `current` where `b`_NNN_ is the eups tag that lsstsw used for the packages installed by the `rebuild` command. ## Development Cycle To work on one or more packages, start by making a fresh build (if you have not gotten one recently) using rebuild: - `rebuild lsst_apps` If this complains that it cannot clone some packages then you will have to update repos.yaml; the simplest way to do this to specify the `-u` flag in the rebuild command. - `eups tags --clone bNNN current` as described above Next declare the git clones of the packages you will be working on to eups. The git clones are in `$LSSTSW/build` and I suggest you declare them with version `git` and a eups tag of your username (since that tag is permitted by eups without altering the eups configuration). For example, user `rowen` could issue the command `eups declare -r $LSSTSW/build/afw afw git -t rowen`. Now edit your code and rebuild manually, using `scons` as needed. Having eups-tagged your packages, you can set them up using: `setup -r . -t` _your\_username_. **Warning: the rebuild command will erase any changes you have not pushed**. Do not run the `rebuild` command while developing and testing your code unless you really need it. Before you run `rebuild` carefully verify that you have **pushed** (not just committed) all your changes. When you are done with your work: - Clean your commit history using `git rebase -i $HEAD~`_N_ where _N_ is the number of commits. - Get your changes reviewed. - Clean up your code as required by the review and clean up the commit history again. - Merge your changes to master and push all changes. - Remove your username tag from the packages. For example to remove your username tag from afw: `eups undeclare afw git -t` _your\_username_. - Build a fresh copy of the stack using `rebuild lsst_apps` followed by `eups tags --clone b`_NNN_ `current`. ## Caveats - Be careful not to overwrite your work with a “rebuild” command. Always commit and **push** before rebuilding, or do your work on clones that are not in `lsstsw/build`. - Be sure to clone tags if you want your new build to be current, because otherwise the default “setup” command won’t act as you expect. - Be careful when trying to run packages in the build directory (rather than installed versions) because “rebuild” only builds what it has to, so some of those directories may not have been built. - You should usually specify “-u” with the rebuild command to update your repos.yaml file. Otherwise you risk the build failing with a complaint about missing packages. ---
**Author:** ![jsick](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/jsick/32/672_2.png) [@jsick](https://www.rubin.community/u/jsick)\ **Post date:** [August 31, 2015, 12:18am UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/2 "2015-08-31T00:18:53Z")
How does this ‘lsstsw’ method of curating a development stack differ in practice from using `lsst-build` (described in [https://confluence.lsstcorp.org/display/LDMDG/The+LSST+Software+Build+Tool](https://confluence.lsstcorp.org/display/LDMDG/The+LSST+Software+Build+Tool))? Should a stack developer _always_ use the lsstsw workflow your describe, or are there cases where `lsst-build` should be used directly? ---
**Author:** ![timj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/timj/32/10_2.png) [@timj](https://www.rubin.community/u/timj)\ **Post date:** [August 31, 2015, 3:56pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/3 "2015-08-31T15:56:20Z")
> [@jsick](#): > > differ in practice from using lsst-build Well, I use `lsst-build` directly but maybe I’m just special. I think it was because I couldn’t find the right documentation when I was trying to find out what to do so I simply looked around the `lsstsw` checkout and the documentation in the code that I could most easily follow was in the `lsst-build` command… I should add that it took me a few times before I realised that this scheme trashes your entire working tree and you have to push to the remote repo. I was somewhat confused the first few times when I had done work on a ticket branch and asked `lsstsw` to build it, only to find that my branch had disappeared (luckily I knew the SHA1 so I could find it again). ---
**Author:** ![RHL](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rhl/32/25_2.png) [@RHL](https://www.rubin.community/u/RHL)\ **Post date:** [September 10, 2015, 9:59pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/4 "2015-09-10T21:59:30Z")
I use a much less magic approach (magic is of course in the eye of the beholder). When I want to get some work done, I build a recent weekly (I’ll use a binary when I can) : ``` eups distrib install lsst_apps -t w_2015_36 ``` Then if I want to work on _e.g._ afw, ``` cd LSST/afw ``` (where I have a git clone) ``` git pull setup -r . -j scons -Q opt=3 -j 6 ``` and I’m good to go. If I need to work on more than a product or two, I tag them: ``` eups list -t rhl ``` (and then `eups undeclare -t rhl NAME` any that I don’t still need) ``` eups declare -r . -t rhl ``` followed by ``` setup -t rhl -t w_lsst_36 lsst_apps ``` I should be able to make w\_lsst\_36 current with a `eups tag --clone=w_lsst_36 current` command, but there seem to be problems in the table file that the versions are incorrect – it’s on @mjuric’s list. If I could, then either ``` setup -t rhl -T w_lsst_36 lsst_apps ``` or a simple ``` setup -t rhl lsst_apps ``` would work. ---
**Author:** ![natepease](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/natepease/32/2086_2.png) [@natepease](https://www.rubin.community/u/natepease)\ **Post date:** [October 9, 2015, 4:49pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/5 "2015-10-09T16:49:03Z")
> [@rowen](#): > > Now edit your code and rebuild manually, using scons as needed. Having eups-tagged your packages, you can set them up using: eups setup -r . -t your\_username. Is this line correct? or is this the correct command? `setup -r . -t your_username` (leaves out the `eups` before `setup`). - when I leave off eups the command works, otherwise it says “eups setup: Unrecognized command: setup” ---
**Author:** ![ktl](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/ktl/32/1373_2.png) [@ktl](https://www.rubin.community/u/ktl)\ **Post date:** [October 9, 2015, 5:18pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/6 "2015-10-09T17:18:16Z")
> [@npease](#): > > Is this line correct? Just testing you 🙂 . Fixed, and thanks. ---
**Author:** ![natepease](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/natepease/32/2086_2.png) [@natepease](https://www.rubin.community/u/natepease)\ **Post date:** [October 9, 2015, 5:34pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/7 "2015-10-09T17:34:38Z")
thanks! I’m glad to know I’m at least somewhat sane 😉 ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [October 23, 2015, 3:59pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/8 "2015-10-23T15:59:29Z")
The work flow @RHL advocates also works, but I feel is less suitable for active daily development because `lsstsw` (or `lsstbuild`) provides direct support for the following: - Pull in a needed fix, either from master or one or more ticket branches - Rebuild a set of interrelated packages after making a fix to one of them Using `eups distrib install` is something developers should know how to use, but I don’t recommend it for daily development because it is too limited and offers too great a chance of incompatibility between the master you are working on and the last weekly release. See also [What development workflow should be advocated? [w/ Poll]](https://community.lsst.org/t/what-development-workflow-should-be-advocated-w-poll/297) for an extensive discussion. ---
**Author:** ![RHL](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rhl/32/25_2.png) [@RHL](https://www.rubin.community/u/RHL)\ **Post date:** [October 26, 2015, 4:09pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/9 "2015-10-26T16:09:03Z")
Does your proposal work with a shared stack? ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [October 26, 2015, 5:36pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/10 "2015-10-26T17:36:46Z")
I doubt using `lsstsw` to build shared stacks for development is a good idea, though it can be done. The biggest issue is that each developer would have to keep packages being worked on out of the main stack area, so that if one developer runs `rebuild` it does not overwrite the work of other developers. That loses one of the main advantages to `lsstsw`: being able to build all related packages on one or more tickets. Still, with careful coordination it can work. For a shared stack that is intended for users instead of developers `lsstsw` seems even less appropriate, since users should use official releases of the software, if possible. ---
**Author:** ![RHL](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rhl/32/25_2.png) [@RHL](https://www.rubin.community/u/RHL)\ **Post date:** [October 26, 2015, 7:32pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/11 "2015-10-26T19:32:58Z")
That was what I thought. I believe that we need a workflow that works with a mostly shared stack. This doesn’t mean that this way of working will be good for all developers/situations, but if it isn’t I think we need to document and support two options. ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [October 30, 2015, 11:49pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/12 "2015-10-30T23:49:39Z")
It’s a bit tricky to do it all. Developers often require cutting edge software, and users are usually better served by official (or semi-official) releases. It is hard for me to imagine that developers and users would usefully share a stack. I can imagine developers being able to share lsstsw stacks if we tweak lsstsw to install in a shared location but offer each developer a personal “build” directory. I have no idea if that would provide enough benefit to be worth doing. It would save some space, but it would also get very cluttered and that clutter would be hard to clean out. And a cluttered stack is hard to manage; eups slows down and even if that were fixed, the output of `list` commands becomes large enough to be a problem. ---
**Author:** ![RHL](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rhl/32/25_2.png) [@RHL](https://www.rubin.community/u/RHL)\ **Post date:** [November 2, 2015, 2:16pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/13 "2015-11-02T14:16:30Z")
> [@rowen](#): > > It is hard for me to imagine that developers and users would usefully share a stack. Why? A specific version (or tagged version) is well defined in a shared stack whether or not some developer has other versions installed. ---
**Author:** ![parejkoj](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/parejkoj/32/138_2.png) [@parejkoj](https://www.rubin.community/u/parejkoj)\ **Post date:** [April 21, 2016, 11:44pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/14 "2016-04-21T23:44:17Z")
Russell just requested that I post this here for his memory. When I need to clean a bunch of old builds from my lsstsw, I do the following: ```auto cd ~/lsst/lsstsw/stack/ rm -rf DarwinX86/* rm -rf ups_db/* ``` Then I `rebuild lsst_apps`, and replace the `afwdata/BLAH` directory with a symlink to my `~/lsst/afwdata` checkout (which is kept up to date because it’s symlinked to `~/lsst/lsstsw/stack/afwdata`). This lets you clean up an overly-large eups list and free a bunch of disk space, without having to reinstall _everything_ or damaging your miniconda install/environments. ---
**Author:** ![rowen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/rowen/32/52_2.png) [@rowen](https://www.rubin.community/u/rowen)\ **Post date:** [June 29, 2016, 7:50pm UTC](https://www.rubin.community/t/developing-software-with-the-dm-stack/126/15 "2016-06-29T19:50:23Z")
> [@parejkoj](#): > > Then I rebuild lsst\_apps, and replace the afwdata/BLAH directory with a symlink to my ~/lsst/afwdata checkout (which is kept up to date because it’s symlinked to ~/lsst/lsstsw/stack/afwdata). One minor addendum: also do the same for `testdata_cfht`. None of the other testdata packages is large enough to be a concern at this time.