Hi Chris,
Thanks a lot for the answer.
Do you know what is the -j option parallized over ?
I could maybe still use -j 16, but use a slurm job that has N calls of ap_pipe for my N visits.
I am not fully sure if that is supposed to be different, or if you are saying that the memory pb is probably not linked to the “multiple visits” problem, but rather due to a given visit that crashes due for instance to the number of sources detected (maybe a single defective one with many artifacts for instance)?
I am not trying to drop alert packages, I actually had to use w_2020_23 instead of v20.0.0, wich I use for the rest of my work, because v20 drops large .avro files (as mentioned here: Run ap_pipe on existing calexps/wcs), and it cannot be turned off.
Is that what you are talking about?
Do you have an easy way to get the “run down by task” of memory usage?
I always run with --profile ap_pipe_profile.dat option, so maybe the information is in there?
Best,
Ben