Search
Saturday, 12th of January 2019, 13:00:31 UTC
13:00:56
kpoeck
Regarding compile-file-parrallel
13:01:09
kpoeck
Looking at https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cmp/compile-file-parallel.lsp#L153
13:01:29
kpoeck
Regarding compile-file-parallel
13:01:55
kpoeck
Threads are being created for every form in the to be compiled file
13:02:30
kpoeck
So if the file being created has 1000 forms, 1000 threads are being created
13:03:12
kpoeck
That might explain why joining the threads takes some time
13:03:52
kpoeck
In https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cmp/compile-file-parallel.lsp#L184
19:33:55
drmeister
kpoeck: Do 1000 threads get created?
19:34:36
drmeister
I'm surprised - I thought that ast compilation would slow things down enough so that maybe 20-30 threads were created at a time.
20:31:03
kpoeck_
Compiling asdf 254 threads are getting xreated and later joined
20:31:58
kpoeck_
They end so fast that there don‘t seem to be more than 10 threads at a time
20:32:39
kpoeck_
Still 254 different threads are getting created and have to be synchronized
20:36:23
kpoeck_
There is a cl-threadpool, perhaps it is usable
21:22:56
drmeister
I see - you have data on asdf.
21:23:16
drmeister
No - that's not what you are saying - there are no more than 10 threads at a time - that's been my experience.
21:23:27
drmeister
Is it expensive to synchronize threads?
21:23:53
Bike
seems unlikely that it would be more expensive than compiling.
21:25:25
drmeister
kpoeck_: I've been looking at compile-file-parallel - I see two things slowing it down (1) invocation of compile during AST generation (2) A still mysterious lock contention between the AST->native code threads and AST generation.
21:26:05
drmeister
I put it aside for a bit while I'm working on chemistry code and thinking about it.
21:28:04
kpoeck_
I assumed creating that many threads is expensive, but i committed the sin of not measuring it
21:28:45
drmeister
"They who are without sin shall cast the first stone"
21:29:00
drmeister
(no stones are thrown)
21:36:50
drmeister
Oh my goodness - there are a lot of pull requests.
21:37:05
drmeister
How far are we from compiling Mcclim?
21:37:56
kpoeck_
If you accept 2 requests stock mcclim works
21:39:46
kpoeck_
But please review 628, since you and Bike had comments
21:40:38
kpoeck_
I see 660 is accepted :-)
21:47:09
jackdaniel
6 more and clasp will burst into flames ;-)
21:56:25
drmeister
kpoeck_: I merged a change and then noticed something - FYI...
21:56:32
drmeister
https://www.irccloud.com/pastebin/4SOqa3Kb/
21:56:53
drmeister
Since we did the x.characterp() test we don't need to use gc::As<Character_sp>(x)
21:57:05
drmeister
That basically tests if it's a character twice.
21:57:11
drmeister
I'd use code like this...
21:57:25
drmeister
https://www.irccloud.com/pastebin/9oe6zECA/
21:58:06
drmeister
gc::As_unsafe<Character_sp>(x) is for when you KNOW x is a Character_sp and you just want the compiler to treat it like one - it's a zero cost transformation.
21:58:16
drmeister
I'll make the change.
21:59:36
drmeister
Thank you for the pull requests - I'm just about done merging them.
22:07:39
drmeister
Ok, everything has been merged - thank you very much!
22:15:52
drmeister
And so now we should be able to compile the stock Mcclim - I'm building and testing now.
22:48:39
kpoeck_
I have some things in ~/quicklisp/local-projects/..
22:49:14
kpoeck_
Not everything is in the last quicklisp release
22:52:15
kpoeck_
Writing a wiki page now
23:03:58
kpoeck_
Wrote the page: how to load mcclim in cladp
23:06:32
kpoeck_
https://github.com/clasp-developers/clasp/wiki/How-to-load-mcclim-in-clasp
23:10:23
drmeister
What is the name of of beach's HIR viewer?
23:15:32
kpoeck_
So let me add that also to the page
23:26:28
drmeister
Ah - thank you ver much.
23:45:13
kpoeck_
Joining 254 threads for parallel compile of asdf took 33 seconds
23:50:03
kpoeck_
Is that the contention you are talking about?
Sunday, 13th of January 2019, 1:00:31 UTC