22:19:28drmeisterThere was a potential infinite loop of handling the error and then restarting the calculation and it generating the error again. It would very occasionally knock itself out of that cycle.
22:19:39drmeisterI set it up so it only tries 3 times and then gives up.
22:20:24drmeisterI have an MPI build in apptainer. I'm not sure how to test it though.
23:15:58drmeisterThings are going well now with load balancing.