freenode/#clasp - IRC Chatlog
Search
14:26:10
wuehlmaus
actually the city where cracauer` lived before he flew to the states decades ago so we both do know eachother
14:29:09
kpoeck
I did a comparison between ast and cst builds, latest everything from this morning, no private patches
14:30:46
kpoeck
Full builds comparison for every between ast and cst build is here: http://kpoeck.github.io/Compile-ast-cst.xlsx
14:32:03
kpoeck
Flamegraph for ast executing cl-bench.misc:run-compiler here: https://kpoeck.github.io/ast-cl-bench-compiler.svg
14:32:57
kpoeck
Flamegraph for cst executing cl-bench.misc:run-compiler here: https://kpoeck.github.io/cst-cl-bench-compiler.svg
14:35:08
kpoeck
In this benchmark, there are no additional inline declarations, so will chase another that has inline declarations
14:36:25
drmeister
That's the code I'm profiling - I'm not sure if it's the best but it's a place to start.
14:59:21
drmeister
::notify Bike Do we have a problem in predlib.lsp? There is the data structure *elementary-types* that is not thread safe. It's an alist and that may be why we have not run into serious problems where two threads modify it at the same time. Also, macroexpansion spends a lot of time in canonical-type - that updates *elementary-types*.
15:00:23
drmeister
When we need a fix that propagates to other people I fork the broken github project, fix my local copy and add it to the pile of github repos that we pull in explicitly.
15:03:44
drmeister
I don't know where to make that change - can you make a recommendation or make the change?
15:04:58
drmeister
We don't have a umask function in clasp - clasp/src/core/unixfsys.cc would be the place to add it.
15:10:58
cracauer`
if I rename esrap-20190521-git to local-projects/esrap, will that still be picked up and not retrieve the esrap-20190521-git version?
15:11:34
drmeister
kpoeck: That took a while to figure out. Clasp backtraces are very deep - you need to collect them all because if you collect 3999 and it's 4000 deep then you lose the bottom most one and that backtrace is shifted relative to all the others and the flame graphs break.
15:12:12
drmeister
But if I try to render flame graphs with 4000 deep backtraces - that takes forever on chrome - it doesn't work.
15:12:57
drmeister
Here I'm describing the backtraces/stack as main is at the bottom and the most recent function called is at the top.
15:14:15
drmeister
cracauer`: My experience is that anything in local-projects overrides everything and esrap won't be retrieved by quicklisp.
15:14:42
cracauer`
so it gets the module name from inside the module, not by the module's directory name?
15:15:27
cracauer`
Chris, do you remember why we have 2 locations for quicklisp modules for a full cando+jupyter build?
15:16:28
drmeister
kpoeck: That's what all the sbcl scripts in src/profiler do - they post-process the dtrace output by adding symbols, cutting out uninteresting backtrace frames and cropping the backtraces so that we can see the most important information in the flamegraphs
15:18:25
drmeister
We moved to /opt/clasp/lib/quicklisp when you made the excellent suggestion that we get our sh*t together and control and version all of these packages and put them in /opt/clasp
15:19:13
kpoeck
cracauer: local-projects takes preference and the name of the directory is not relevant
15:19:27
drmeister
Then for some reason, there was a really good reason recently to move quicklisp into clasp/src/lisp/modules/quicklisp/... like asdf is in clasp/src/lisp/modules/asdf/ - but I don't remember why.
15:22:43
drmeister
Do you recall why we did this? I'm sure I didn't do it unilaterally, we must have talked about this.
15:23:00
kpoeck
drmeister: Here is my flamegraph for compiling math.lisp from cl-bench: https://kpoeck.github.io/ast-cl-bench-math.svg
15:24:35
drmeister
kpoeck: I have a better cleanup script for you that will remove the frames that contain TOP-COMPILE-FILE.
15:25:45
drmeister
kpoeck: I just pushed a change to stacks.lisp that should improve the flame graphs.
15:29:19
drmeister
cracauer`: Within clasp I added the "quicklisp:" hostname a while ago. That is what cando uses to find quicklisp. Once it does (load "quicklisp:setup.lisp") the quicklisp location is set.
15:30:15
drmeister
Yes, on my laptop it takes 55 min to build cando/cclasp and another 45 min to build the quicklisp part not including any jupyterlab stuff.
15:30:57
drmeister
I think the quicklisp: hostname is set using CLASP_QUICKLISP_DIRECTORY hang on...
15:31:32
cracauer`
that's correct, the question is why redicrets it to clasp/src/lisp/modules/quicklisp during some parts of the process?
15:31:35
drmeister
The code in bundle.cc is run at startup to locate all of the directories that clasp needs.
15:32:19
drmeister
Ok - that is unexpected news. I don't know how that can happen. Do you know how to reproduce that?
15:38:06
drmeister
cracauer`: On a different topic - are you familiar with this idea? https://en.wikipedia.org/wiki/Read-copy-update
15:41:29
drmeister
I've been trying to figure it out for a while. We have some CS faculty who I could ask but I haven't been able to ask them yet.
15:44:23
drmeister
I'm trying to figure out how to apply it to something complicated like a hash-table.
15:44:45
drmeister
I'd need to get every thread in a state where there are no readers of the hash-table to make updates.
15:46:56
drmeister
To do an update all threads would have to go into a non-reader state and then I can copy the alist into the hash-table.
15:47:39
drmeister
The tricky part is rehashes - especially with location dependency and moving garbage collection. There a gethash can trigger a rehash.
15:48:09
drmeister
Does this all sound like I'm applying the idea of read-copy-update or am I out to lunch here?
15:49:26
drmeister
We had a lot of problems when we used locks on a thread-safe hash-table for find-class. The exclusive access lock bounced from CPU to CPU and everything slowed way, way down.
15:50:01
drmeister
I think I'm looking at a similar challenge with *elementary-types* in predlib.lsp.
15:50:50
drmeister
I could make it a hash-table - but then I have the problems that I had with find-class.
15:51:12
drmeister
I've been looking at RCU as a solution to these sorts of synchronization problems.
15:52:04
drmeister
Not really, because the head of the alist is an aligned pointer - I'm guessing that's why we haven't run into problems with it.
15:53:19
drmeister
Writing to it is atomic or almost atomic- isn't it? There is a very small time window where two threads can screw things up.
15:54:04
stassats
there's no such thing as "almost" when it comes to atomic, and adding new elements to an alist is certainly not atomic
15:54:36
drmeister
Exactly , we probably aren't modifying *elementary-types* very often - but if I switch it to a hash-table - then it makes it more likely that we will have conflicts between threads that I need to deal with by redesigning things.
15:56:36
drmeister
I'm trying to understand why we haven't run into a problem with it. I'm guessing it's because we don't update it much and that it is an alist. Two things together that have let us get by without dealing with this. No? Yes?
15:57:11
stassats
the consequences of concurrently modifying a hash-table are more dire, an alist will just lose values, which can't be considered a good outcome either
15:57:37
drmeister
But if I change it to a hash-table to hopefully speed up reads - then I really should deal with it. I could just use a thread safe hash-table - if it's not written to often - then there won't be a problem with lock contention.
15:59:47
drmeister
It's spending 20% of the time in canonical-type. That looks up and possibly modifies *elementary-types*
16:12:30
drmeister
Yeesh, the code requires a lot of changes to change *elementary-types* to anything other than an alist.
16:28:57
drmeister
What we know right now is if we comment out that (declare ignore ...) that everything works. When I build with that declare in place - then the quicklisp load of esrap goes nuts and prints endless empty lines.
16:32:02
scymtym
SLIME can handle it to some extend but i have seen inaccurate results in complicated cases
16:38:07
scymtym
the unusual bit in the code in question is shadowing the global macro WITH-CACHED-RESULT via MACROLET
16:41:15
drmeister
I can't get anywhere with slime macroexpansion so I uncommented that declare and now...
16:42:39
drmeister
https://github.com/quicklisp/quicklisp-client/blob/master/dists/quicklisp/software/esrap-20190521-git/src/evaluator.lisp#L106
16:46:33
scymtym
just as a sanity check: does anything change if you rename the BODY variable in lines 73 and 75?
16:49:49
scymtym
you could also try putting a PRINT around the body of NAMED-LAMBDA/CACHE-VARIANTS to see what it expands to (should be expanded four times)
17:01:28
drmeister
scymtym: I can't get insight - I starting to suspect it's a problem in our AST interpreter - because it's got to be happening when the WITH-CACHED-RESULT macrolet function is being evaluated.
17:05:56
drmeister
Hmm, there nothing in the ast-interpreter that deals specifically with macrolet. macrolets are an environment thing - declares should be gone by the time the ast-interpreter hits them.
17:06:44
scymtym
no worries. i wouldn't have been surprised if that section of the code had a problem
17:12:11
drmeister
Grrr, I'll probably need Bike's help - I can't even figure out where this is evaluated from
17:19:14
drmeister
::notify Bike Could you check the discussion here. There is an issue compiling esrap that might be due to Cleavir not handling a (declare (ignore ...)) properly in a macrolet in a backquote in a macrolet. It's esrap evaluator.lisp line 74
18:08:42
drmeister
Indeed there are. This issue in particular: https://github.com/robert-strandh/SICL/issues/141
18:17:19
drmeister
Given that it's non-conforming behavior in cleavir I'm loath to ask scymtym to make a change for clasp.
18:20:23
drmeister
cracauer`: I can reproduce the problem when I run cando in slime and (ql:quickload "cl-jupyter")
19:03:16
drmeister
cracauer`: I found the source of the problem. There is a compiler macro on CONCATENATE that is messed up.
19:03:46
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cmp/opt-sequence.lsp#L137
19:07:58
drmeister
How the hell do we get all the way to the last quicklisp package at the end of the cando build before this rears its ugly head?
19:11:09
pfdietz
There's an offshoot of ansi-tests that's useful for testing compiler macros and other compile-time optimizations on builtins, maybe that would help.
19:13:08
pfdietz
They put together calls to various built-ins with random arguments, random type declarations, and see if things work.
19:18:45
pfdietz
I need to get that test sute back into a more usable state. It was off being toyed with by others for some years.
22:04:43
v0|d
cracauer`: a question. say you have syntax/lexer/parser for a lang, additionally have an interpreter for it. say you decided to compile it (ie to some ir). What is the correct way to prove that the compiler obeys the interpreter?