libera/#clasp - IRC Chatlog
Search
14:17:11
drmeister
I have a problem in the search that gets smaller and smaller the more searching I do.
14:17:43
drmeister
It's a bit difficult to describe but imagine I'm generating puzzle pieces that must connect to other puzzle pieces.
14:17:47
yitzi
I seem to recall, that is how the "channels" in lparallel work. I think that there is a bug in lparallel in that the next job in the queue won't start if you don't retrieve the result waiting on the channel. But that shouldn't be a problem for PMAP. I ran into this issue for the TIRUN app when we are sketching the ligands.
14:18:27
drmeister
With a short search - of say 20 - about 1% of the puzzle pieces don't fit a following piece.
14:18:53
drmeister
With a search of 200 - it's about 0.4% of the puzzle pieces don't fit a following piece.
14:26:45
drmeister
The measurement for each node is not very good - or lparallel is doing crazy things.
14:27:56
drmeister
I am assuming I need to watch the trend - and the trend looks like there is still a long tail.
14:41:02
yitzi
There are some examples of using futures in https://github.com/cando-developers/cando/blob/0fc1fa09ee22521403bd46e1b8298f82ae2d94f5/src/lisp/cando-widgets/molecule-select.lisp
14:42:59
yitzi
drmeister: yes...that was me. You could also keep it simple https://lparallel.org/kernel/
14:46:26
yitzi
drmeister: I am pretty sure you just add all the tasks and then just idle while waiting for the results ... which are just indicative that the job completed.
15:24:16
drmeister
So you just open a channel and submit-task's to it and they automatically go the the *kernel* and then you call receive-result for each task?
16:30:58
drmeister
I added per-node/per-thread logging and my attempt at load balancing is absolute shite.
16:31:37
drmeister
I was sorting the jobs based on the number of atoms - figuring more atoms take more time.
16:32:34
drmeister
That's not at all the case - the amount of time varies hugely. Now I suspect that some non-linear optimizations are getting trapped and I'm letting them wander too long.
19:37:10
stassats
i would have made a queue of jobs from which each thread repeatedly gets a job (or a batch of jobs, if each individual one is very small)
20:04:50
drmeister
It's not a burning issue - it looks like I can push MPI into the future a bit because I think I solved the issue with the tail. I had an almost infinite loop of error /error handling.
20:24:06
yitzi
If it is not already in the container then just add that to the apt-get install in the def file
22:18:46
drmeister
It was a handler that recognized 3 or 4 linear atoms (a problem for non-linear optimization) and that caught the error and tried to shake up the 3 or 4 linear atoms. It doesn't work very well probably because the rest of the structure forces the atoms back into a linear arrangement.
22:19:28
drmeister
There was a potential infinite loop of handling the error and then restarting the calculation and it generating the error again. It would very occasionally knock itself out of that cycle.