freenode/#clasp - IRC Chatlog
Search
15:18:25
drmeister
We moved to /opt/clasp/lib/quicklisp when you made the excellent suggestion that we get our sh*t together and control and version all of these packages and put them in /opt/clasp
15:19:13
kpoeck
cracauer: local-projects takes preference and the name of the directory is not relevant
15:19:27
drmeister
Then for some reason, there was a really good reason recently to move quicklisp into clasp/src/lisp/modules/quicklisp/... like asdf is in clasp/src/lisp/modules/asdf/ - but I don't remember why.
15:22:43
drmeister
Do you recall why we did this? I'm sure I didn't do it unilaterally, we must have talked about this.
15:23:00
kpoeck
drmeister: Here is my flamegraph for compiling math.lisp from cl-bench: https://kpoeck.github.io/ast-cl-bench-math.svg
15:24:35
drmeister
kpoeck: I have a better cleanup script for you that will remove the frames that contain TOP-COMPILE-FILE.
15:25:45
drmeister
kpoeck: I just pushed a change to stacks.lisp that should improve the flame graphs.
15:29:19
drmeister
cracauer`: Within clasp I added the "quicklisp:" hostname a while ago. That is what cando uses to find quicklisp. Once it does (load "quicklisp:setup.lisp") the quicklisp location is set.
15:30:15
drmeister
Yes, on my laptop it takes 55 min to build cando/cclasp and another 45 min to build the quicklisp part not including any jupyterlab stuff.
15:30:57
drmeister
I think the quicklisp: hostname is set using CLASP_QUICKLISP_DIRECTORY hang on...
15:31:32
cracauer`
that's correct, the question is why redicrets it to clasp/src/lisp/modules/quicklisp during some parts of the process?
15:31:35
drmeister
The code in bundle.cc is run at startup to locate all of the directories that clasp needs.
15:32:19
drmeister
Ok - that is unexpected news. I don't know how that can happen. Do you know how to reproduce that?
15:38:06
drmeister
cracauer`: On a different topic - are you familiar with this idea? https://en.wikipedia.org/wiki/Read-copy-update
15:41:29
drmeister
I've been trying to figure it out for a while. We have some CS faculty who I could ask but I haven't been able to ask them yet.
15:44:23
drmeister
I'm trying to figure out how to apply it to something complicated like a hash-table.
15:44:45
drmeister
I'd need to get every thread in a state where there are no readers of the hash-table to make updates.
15:46:56
drmeister
To do an update all threads would have to go into a non-reader state and then I can copy the alist into the hash-table.
15:47:39
drmeister
The tricky part is rehashes - especially with location dependency and moving garbage collection. There a gethash can trigger a rehash.
15:48:09
drmeister
Does this all sound like I'm applying the idea of read-copy-update or am I out to lunch here?
15:49:26
drmeister
We had a lot of problems when we used locks on a thread-safe hash-table for find-class. The exclusive access lock bounced from CPU to CPU and everything slowed way, way down.
15:50:01
drmeister
I think I'm looking at a similar challenge with *elementary-types* in predlib.lsp.
15:50:50
drmeister
I could make it a hash-table - but then I have the problems that I had with find-class.
15:51:12
drmeister
I've been looking at RCU as a solution to these sorts of synchronization problems.
15:52:04
drmeister
Not really, because the head of the alist is an aligned pointer - I'm guessing that's why we haven't run into problems with it.
15:53:19
drmeister
Writing to it is atomic or almost atomic- isn't it? There is a very small time window where two threads can screw things up.
15:54:04
stassats
there's no such thing as "almost" when it comes to atomic, and adding new elements to an alist is certainly not atomic
15:54:36
drmeister
Exactly , we probably aren't modifying *elementary-types* very often - but if I switch it to a hash-table - then it makes it more likely that we will have conflicts between threads that I need to deal with by redesigning things.
15:56:36
drmeister
I'm trying to understand why we haven't run into a problem with it. I'm guessing it's because we don't update it much and that it is an alist. Two things together that have let us get by without dealing with this. No? Yes?
15:57:11
stassats
the consequences of concurrently modifying a hash-table are more dire, an alist will just lose values, which can't be considered a good outcome either
15:57:37
drmeister
But if I change it to a hash-table to hopefully speed up reads - then I really should deal with it. I could just use a thread safe hash-table - if it's not written to often - then there won't be a problem with lock contention.
15:59:47
drmeister
It's spending 20% of the time in canonical-type. That looks up and possibly modifies *elementary-types*
16:12:30
drmeister
Yeesh, the code requires a lot of changes to change *elementary-types* to anything other than an alist.
16:28:57
drmeister
What we know right now is if we comment out that (declare ignore ...) that everything works. When I build with that declare in place - then the quicklisp load of esrap goes nuts and prints endless empty lines.
16:32:02
scymtym
SLIME can handle it to some extend but i have seen inaccurate results in complicated cases
16:38:07
scymtym
the unusual bit in the code in question is shadowing the global macro WITH-CACHED-RESULT via MACROLET
16:41:15
drmeister
I can't get anywhere with slime macroexpansion so I uncommented that declare and now...
16:42:39
drmeister
https://github.com/quicklisp/quicklisp-client/blob/master/dists/quicklisp/software/esrap-20190521-git/src/evaluator.lisp#L106
16:46:33
scymtym
just as a sanity check: does anything change if you rename the BODY variable in lines 73 and 75?
16:49:49
scymtym
you could also try putting a PRINT around the body of NAMED-LAMBDA/CACHE-VARIANTS to see what it expands to (should be expanded four times)
17:01:28
drmeister
scymtym: I can't get insight - I starting to suspect it's a problem in our AST interpreter - because it's got to be happening when the WITH-CACHED-RESULT macrolet function is being evaluated.
17:05:56
drmeister
Hmm, there nothing in the ast-interpreter that deals specifically with macrolet. macrolets are an environment thing - declares should be gone by the time the ast-interpreter hits them.
17:06:44
scymtym
no worries. i wouldn't have been surprised if that section of the code had a problem
17:12:11
drmeister
Grrr, I'll probably need Bike's help - I can't even figure out where this is evaluated from
17:19:14
drmeister
::notify Bike Could you check the discussion here. There is an issue compiling esrap that might be due to Cleavir not handling a (declare (ignore ...)) properly in a macrolet in a backquote in a macrolet. It's esrap evaluator.lisp line 74
18:08:42
drmeister
Indeed there are. This issue in particular: https://github.com/robert-strandh/SICL/issues/141
18:17:19
drmeister
Given that it's non-conforming behavior in cleavir I'm loath to ask scymtym to make a change for clasp.
18:20:23
drmeister
cracauer`: I can reproduce the problem when I run cando in slime and (ql:quickload "cl-jupyter")
19:03:16
drmeister
cracauer`: I found the source of the problem. There is a compiler macro on CONCATENATE that is messed up.
19:03:46
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cmp/opt-sequence.lsp#L137
19:07:58
drmeister
How the hell do we get all the way to the last quicklisp package at the end of the cando build before this rears its ugly head?
19:11:09
pfdietz
There's an offshoot of ansi-tests that's useful for testing compiler macros and other compile-time optimizations on builtins, maybe that would help.
19:13:08
pfdietz
They put together calls to various built-ins with random arguments, random type declarations, and see if things work.
19:18:45
pfdietz
I need to get that test sute back into a more usable state. It was off being toyed with by others for some years.
22:04:43
v0|d
cracauer`: a question. say you have syntax/lexer/parser for a lang, additionally have an interpreter for it. say you decided to compile it (ie to some ir). What is the correct way to prove that the compiler obeys the interpreter?