freenode/#clasp - IRC Chatlog
Search
6:32:17
karlosz
::notify Bike do you mind reviewing the pull request i sent in to sicl about inlining i sent in a while ago? some feedback on the general approach would be nice because its been sort of a long standing issue in the compiler for a bit
14:26:10
wuehlmaus
actually the city where cracauer` lived before he flew to the states decades ago so we both do know eachother
14:29:09
kpoeck
I did a comparison between ast and cst builds, latest everything from this morning, no private patches
14:30:46
kpoeck
Full builds comparison for every between ast and cst build is here: http://kpoeck.github.io/Compile-ast-cst.xlsx
14:32:03
kpoeck
Flamegraph for ast executing cl-bench.misc:run-compiler here: https://kpoeck.github.io/ast-cl-bench-compiler.svg
14:32:57
kpoeck
Flamegraph for cst executing cl-bench.misc:run-compiler here: https://kpoeck.github.io/cst-cl-bench-compiler.svg
14:35:08
kpoeck
In this benchmark, there are no additional inline declarations, so will chase another that has inline declarations
14:36:25
drmeister
That's the code I'm profiling - I'm not sure if it's the best but it's a place to start.
14:59:21
drmeister
::notify Bike Do we have a problem in predlib.lsp? There is the data structure *elementary-types* that is not thread safe. It's an alist and that may be why we have not run into serious problems where two threads modify it at the same time. Also, macroexpansion spends a lot of time in canonical-type - that updates *elementary-types*.
15:00:23
drmeister
When we need a fix that propagates to other people I fork the broken github project, fix my local copy and add it to the pile of github repos that we pull in explicitly.
15:03:44
drmeister
I don't know where to make that change - can you make a recommendation or make the change?
15:04:58
drmeister
We don't have a umask function in clasp - clasp/src/core/unixfsys.cc would be the place to add it.
15:10:58
cracauer`
if I rename esrap-20190521-git to local-projects/esrap, will that still be picked up and not retrieve the esrap-20190521-git version?
15:11:34
drmeister
kpoeck: That took a while to figure out. Clasp backtraces are very deep - you need to collect them all because if you collect 3999 and it's 4000 deep then you lose the bottom most one and that backtrace is shifted relative to all the others and the flame graphs break.
15:12:12
drmeister
But if I try to render flame graphs with 4000 deep backtraces - that takes forever on chrome - it doesn't work.
15:12:57
drmeister
Here I'm describing the backtraces/stack as main is at the bottom and the most recent function called is at the top.
15:14:15
drmeister
cracauer`: My experience is that anything in local-projects overrides everything and esrap won't be retrieved by quicklisp.
15:14:42
cracauer`
so it gets the module name from inside the module, not by the module's directory name?
15:15:27
cracauer`
Chris, do you remember why we have 2 locations for quicklisp modules for a full cando+jupyter build?
15:16:28
drmeister
kpoeck: That's what all the sbcl scripts in src/profiler do - they post-process the dtrace output by adding symbols, cutting out uninteresting backtrace frames and cropping the backtraces so that we can see the most important information in the flamegraphs
15:18:25
drmeister
We moved to /opt/clasp/lib/quicklisp when you made the excellent suggestion that we get our sh*t together and control and version all of these packages and put them in /opt/clasp
15:19:13
kpoeck
cracauer: local-projects takes preference and the name of the directory is not relevant
15:19:27
drmeister
Then for some reason, there was a really good reason recently to move quicklisp into clasp/src/lisp/modules/quicklisp/... like asdf is in clasp/src/lisp/modules/asdf/ - but I don't remember why.
15:22:43
drmeister
Do you recall why we did this? I'm sure I didn't do it unilaterally, we must have talked about this.
15:23:00
kpoeck
drmeister: Here is my flamegraph for compiling math.lisp from cl-bench: https://kpoeck.github.io/ast-cl-bench-math.svg
15:24:35
drmeister
kpoeck: I have a better cleanup script for you that will remove the frames that contain TOP-COMPILE-FILE.
15:25:45
drmeister
kpoeck: I just pushed a change to stacks.lisp that should improve the flame graphs.
15:29:19
drmeister
cracauer`: Within clasp I added the "quicklisp:" hostname a while ago. That is what cando uses to find quicklisp. Once it does (load "quicklisp:setup.lisp") the quicklisp location is set.
15:30:15
drmeister
Yes, on my laptop it takes 55 min to build cando/cclasp and another 45 min to build the quicklisp part not including any jupyterlab stuff.
15:30:57
drmeister
I think the quicklisp: hostname is set using CLASP_QUICKLISP_DIRECTORY hang on...
15:31:32
cracauer`
that's correct, the question is why redicrets it to clasp/src/lisp/modules/quicklisp during some parts of the process?
15:31:35
drmeister
The code in bundle.cc is run at startup to locate all of the directories that clasp needs.
15:32:19
drmeister
Ok - that is unexpected news. I don't know how that can happen. Do you know how to reproduce that?
15:38:06
drmeister
cracauer`: On a different topic - are you familiar with this idea? https://en.wikipedia.org/wiki/Read-copy-update
15:41:29
drmeister
I've been trying to figure it out for a while. We have some CS faculty who I could ask but I haven't been able to ask them yet.
15:44:23
drmeister
I'm trying to figure out how to apply it to something complicated like a hash-table.
15:44:45
drmeister
I'd need to get every thread in a state where there are no readers of the hash-table to make updates.
15:46:56
drmeister
To do an update all threads would have to go into a non-reader state and then I can copy the alist into the hash-table.
15:47:39
drmeister
The tricky part is rehashes - especially with location dependency and moving garbage collection. There a gethash can trigger a rehash.
15:48:09
drmeister
Does this all sound like I'm applying the idea of read-copy-update or am I out to lunch here?
15:49:26
drmeister
We had a lot of problems when we used locks on a thread-safe hash-table for find-class. The exclusive access lock bounced from CPU to CPU and everything slowed way, way down.
15:50:01
drmeister
I think I'm looking at a similar challenge with *elementary-types* in predlib.lsp.
15:50:50
drmeister
I could make it a hash-table - but then I have the problems that I had with find-class.
15:51:12
drmeister
I've been looking at RCU as a solution to these sorts of synchronization problems.
15:52:04
drmeister
Not really, because the head of the alist is an aligned pointer - I'm guessing that's why we haven't run into problems with it.
15:53:19
drmeister
Writing to it is atomic or almost atomic- isn't it? There is a very small time window where two threads can screw things up.
15:54:04
stassats
there's no such thing as "almost" when it comes to atomic, and adding new elements to an alist is certainly not atomic
15:54:36
drmeister
Exactly , we probably aren't modifying *elementary-types* very often - but if I switch it to a hash-table - then it makes it more likely that we will have conflicts between threads that I need to deal with by redesigning things.
15:56:36
drmeister
I'm trying to understand why we haven't run into a problem with it. I'm guessing it's because we don't update it much and that it is an alist. Two things together that have let us get by without dealing with this. No? Yes?
15:57:11
stassats
the consequences of concurrently modifying a hash-table are more dire, an alist will just lose values, which can't be considered a good outcome either
15:57:37
drmeister
But if I change it to a hash-table to hopefully speed up reads - then I really should deal with it. I could just use a thread safe hash-table - if it's not written to often - then there won't be a problem with lock contention.
15:59:47
drmeister
It's spending 20% of the time in canonical-type. That looks up and possibly modifies *elementary-types*
16:12:30
drmeister
Yeesh, the code requires a lot of changes to change *elementary-types* to anything other than an alist.
16:28:57
drmeister
What we know right now is if we comment out that (declare ignore ...) that everything works. When I build with that declare in place - then the quicklisp load of esrap goes nuts and prints endless empty lines.
16:32:02
scymtym
SLIME can handle it to some extend but i have seen inaccurate results in complicated cases