freenode/#clasp - IRC Chatlog
Search
15:50:11
drmeister
Bike: I think hoisting the AST's needs to be done in serial mode as well. I was doing it in parallel - but that is leading to some form compilation in the threads and that is leading to problems.
15:59:04
drmeister
Now I'm worried that this is going to interact poorly with the literal compilation.
16:07:48
drmeister
I don't have all the details in my head though - I may be wrong. I'll try with hoisted asts.
16:09:43
drmeister
Hoisting ast's though - that involves form compilation for load-time-values - right? I see it in the backtrace so I'm pretty sure it's a general thing.
16:10:34
drmeister
They get compiled in the top level environment - so I if I did do it in threads - the environment isn't a problem.
16:12:54
drmeister
Does that mean something other than that they are compiled in the top level environment?
16:16:41
drmeister
I have a lot of compiler functions that take an environment as a parameter - I always think of the top level environment being what I get when I pass NIL as the environment
16:18:02
drmeister
For Cleavir I define a *clasp-env* that I pass in place of NIL - that's also the top level environment.
16:20:17
drmeister
I may need to switch back to hoisting in threads - my current problem may be a simple issue of the threads that translate ast's needing to communicate back to the main thread about problems like unknown global function references.
17:47:00
drmeister
stassats: In slime I get 'sldb-quit called outside of sldb buffer' if I try to quit out of an sldb buffer created by a thread. Any ideas where to start fixing that?
17:51:59
drmeister
I guess launching threads from within clasp - the dynamic environment of the parent thread doesn't carry over to the child threads.
17:54:04
drmeister
You are using slime and you run something that launches a child thread and that child thread signals an error and a sldb buffer opens up.
17:55:29
drmeister
I mean - I know we aren't supposed to write code with errors. But the hypothetical situation where you do incorporate an error in the code.
17:57:12
drmeister
But restarts are maintained in dynamic variables aren't they? I'd have to initialize the child thread special bindings with the ones that store restarts - no?
17:57:45
drmeister
I'm talking without doing any research into what special variables I'm referring to.
18:00:44
drmeister
I have this stuff... https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/clos/conditions.lsp#L43
18:04:40
drmeister
I thought if I initialize the special-variable *restart-clusters* of a child thread with the value from the parent thread that the child would see the restarts.
19:03:53
drmeister
stassats: Thank you for that advice - I can recover now from child threads dying.
19:06:10
drmeister
It compiles the AST's serially and then launches threads to do the AST->HIR->MIR->LLVM-IR->native-code in parallel.
19:06:51
drmeister
There is clearly some competition going on by the threads - but I don't know what. I'm going to try to swap in tcmalloc for the regular malloc.
19:08:04
drmeister
We still have to implement a bytecode interpreter for literals - we have it figured out - we just need to implement it. That will improve things as well.
19:08:31
drmeister
I came up with a scheme where the C++ byte code interpreter will be automatically generated by the Common Lisp code.
19:12:14
drmeister
I've got one really ugly thing in this parallel compile-file business - it's to initialize child threads special variables with the value from the parent. It has a bad code smell to it but I'm not sure what else I could have here.
19:17:15
drmeister
Hmm that might work. My understanding is that the forms get evaluated in the child thread - does what you suggest work with that?
19:18:24
drmeister
The bordeaux-threads documentation on *default-special-bindings* isn't clear about where the forms get evaluated.
19:23:17
drmeister
Bah! I get an assertion failure in the parallel compile-file when compiling babel.
19:24:20
drmeister
There's more to this than I have time to deal with right now. I thought I could get it working quickly and benefit from faster compiles but I'm getting anxious about how much time this is taking and how deep I seem to be getting into the compiler rabbit hole.
19:24:56
Bike
if you want to take a break i could maaybe look at it, probably after the loading machine
19:25:48
drmeister
Well, I thought it would be straightforward to go from the AST on in parallel - but there is stuff going on with load-time-values.
19:27:07
drmeister
https://github.com/Bike/SICL/blob/master/Code/Cleavir/HIR-transformations/segregate-lexicals.lisp#L212
19:27:25
drmeister
https://github.com/Bike/SICL/blob/master/Code/Cleavir/HIR-transformations/segregate-lexicals.lisp#L176
19:34:42
drmeister
I implemented your suggestion above and I'll try it running in serial and see if the same problem shows up. It would be illuminating to know if it's some kind of out of order thing or a general problem with the way the compilation is broken up now.
19:35:31
drmeister
I'll try serial compilation, then threaded but one thread at a time and then all threads going nuts.
19:36:54
drmeister
What was that thing that you mentioned about telling asdf that systems are loaded when they are loaded a different way than how asdf would load them?
20:04:38
drmeister
Huh - that's surprising - compile-file-parallel works fine if I run the ast->hir->...->native-code in the parent thread - so breaking up the compile-file this way is not the problem.