# Ap\_pipe in parallel jobs

**URL:** <https://www.rubin.community/t/ap-pipe-in-parallel-jobs/4239>\
**Category:** Support\
**Created:** [July 1, 2020, 2:18pm UTC](https://www.rubin.community/t/ap-pipe-in-parallel-jobs/4239 "2020-07-01T14:18:01Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mchbib](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/mchbib/32/998_2.png) [@mchbib](https://www.rubin.community/u/mchbib)\
**Post date:** [July 1, 2020, 2:18pm UTC](https://www.rubin.community/t/ap-pipe-in-parallel-jobs/4239/1 "2020-07-01T14:18:01Z")

</div>

Hello all, I want to run ap\_pipe for different bunches of visits for a given night (in 3 jobs separately), just to reduce the time. What is the best way to do this as I would like to get, by the end, one common association.db file that gathers all visits of the 3 jobs? Thank you

---

<div class="post-metadata">

**Author:** ![kfindeisen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/kfindeisen/32/833_2.png) [@kfindeisen](https://www.rubin.community/u/kfindeisen)\
**Post date:** [July 1, 2020, 5:24pm UTC](https://www.rubin.community/t/ap-pipe-in-parallel-jobs/4239/2 "2020-07-01T17:24:24Z")

</div>

Hello, thank you for your question. `ap_pipe.py` is designed so that multiple calls can update the same database, so there’s no problem there.

There _may_ be a problem with running the jobs in parallel, however; one possibility is that sources detected in visits processed simultaneously would be assigned to different objects at the same sky position. @cmorrison would be better able to tell you how well the database handles concurrent access.

---

<div class="post-metadata">

**Author:** ![mchbib](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/mchbib/32/998_2.png) [@mchbib](https://www.rubin.community/u/mchbib)\
**Post date:** [July 2, 2020, 3:36pm UTC](https://www.rubin.community/t/ap-pipe-in-parallel-jobs/4239/3 "2020-07-02T15:36:49Z")

</div>

Hello, thank you for your answer. So, there is no guaranty to have safe association with parallel jobs. Is there another way to reduce the time needed by ap\_pipe? It took almost 16 hours for one visit/103 ccds. Thank you

---

<div class="post-metadata">

**Author:** ![kfindeisen](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/kfindeisen/32/833_2.png) [@kfindeisen](https://www.rubin.community/u/kfindeisen)\
**Post date:** [July 2, 2020, 10:50pm UTC](https://www.rubin.community/t/ap-pipe-in-parallel-jobs/4239/4 "2020-07-02T22:50:35Z")

</div>

Maybe you could clarify something. When you said “in 3 jobs separately”, did you mean you run `ap_pipe.py` three times? That is what I assumed at first, and as I said I am not sure what might happen in that case.

On the other hand, we _have_ done parallel runs internal to the program using the `--processes`/`-j` command-line arguments, and those function without corruption (with the caveat, as @mrawls mentioned, that sources may be processed in a different order from that in which they were observed). If you are not yet using that feature, it might give you the speed-up you need.

---

<div class="post-metadata">

**Author:** ![mchbib](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/mchbib/32/998_2.png) [@mchbib](https://www.rubin.community/u/mchbib)\
**Post date:** [July 3, 2020, 3:49am UTC](https://www.rubin.community/t/ap-pipe-in-parallel-jobs/4239/5 "2020-07-03T03:49:51Z")

</div>

Hello, yes this is what I meant with 3 jobs separately (run it three times). Thank you for explaining how you performed parallel jobs. I was not using the --process option before, it is what I need then to reduce the ap\_pipe process time ! Thank you very much!