freenode/#clasp - IRC Chatlog

15:11:34 drmeister kpoeck: That took a while to figure out. Clasp backtraces are very deep - you need to collect them all because if you collect 3999 and it's 4000 deep then you lose the bottom most one and that backtrace is shifted relative to all the others and the flame graphs break.

15:12:12 drmeister But if I try to render flame graphs with 4000 deep backtraces - that takes forever on chrome - it doesn't work.

15:12:30 drmeister So I collect up to 4000 frames but I truncate the frames at 400 from the bottom.

15:12:57 drmeister Here I'm describing the backtraces/stack as main is at the bottom and the most recent function called is at the top.

15:13:02 drmeister Does that answer the question?

15:13:28 kpoeck Yes

15:14:15 drmeister cracauer`: My experience is that anything in local-projects overrides everything and esrap won't be retrieved by quicklisp.

15:14:42 cracauer` so it gets the module name from inside the module, not by the module's directory name?

15:15:27 cracauer` Chris, do you remember why we have 2 locations for quicklisp modules for a full cando+jupyter build?

15:16:28 drmeister kpoeck: That's what all the sbcl scripts in src/profiler do - they post-process the dtrace output by adding symbols, cutting out uninteresting backtrace frames and cropping the backtraces so that we can see the most important information in the flamegraphs

15:17:11 drmeister cracauer`: Here's my recollection... it's fuzzy.

15:17:25 drmeister We started with ~/quicklisp/...

15:18:25 drmeister We moved to /opt/clasp/lib/quicklisp when you made the excellent suggestion that we get our sh*t together and control and version all of these packages and put them in /opt/clasp

15:19:13 kpoeck cracauer: local-projects takes preference and the name of the directory is not relevant

15:19:25 kpoeck Name is taken from the asd file

15:19:27 drmeister Then for some reason, there was a really good reason recently to move quicklisp into clasp/src/lisp/modules/quicklisp/... like asdf is in clasp/src/lisp/modules/asdf/ - but I don't remember why.

15:21:39 drmeister cracauer`: I'm wracking my brains why.

15:21:53 drmeister Would it be ok to just stick with clasp/src/lisp/modules/quicklisp ?

15:22:43 drmeister Do you recall why we did this? I'm sure I didn't do it unilaterally, we must have talked about this.

15:23:00 kpoeck drmeister: Here is my flamegraph for compiling math.lisp from cl-bench: https://kpoeck.github.io/ast-cl-bench-math.svg

15:23:28 kpoeck I assume all the llvm-work is than in the upper part that is cut-off

15:24:35 drmeister kpoeck: I have a better cleanup script for you that will remove the frames that contain TOP-COMPILE-FILE.

15:24:39 drmeister Hang on.

15:25:45 drmeister kpoeck: I just pushed a change to stacks.lisp that should improve the flame graphs.

15:28:08 kpoeck ok, pulling

15:28:27 cracauer` I'm symlinking the two quicklisp locations for now.

15:28:46 cracauer` we can unify later

15:28:56 cracauer` right now the build is too slow to test cosmetic fixes

15:29:19 drmeister cracauer`: Within clasp I added the "quicklisp:" hostname a while ago. That is what cando uses to find quicklisp. Once it does (load "quicklisp:setup.lisp") the quicklisp location is set.

15:30:15 drmeister Yes, on my laptop it takes 55 min to build cando/cclasp and another 45 min to build the quicklisp part not including any jupyterlab stuff.

15:30:17 cracauer` yeah, but CLASP_QUICKLISP_DIRECTORY points to /opt/clasp/lib/quicklisp

15:30:32 cracauer` so the build process seems to change that for some jupyter things

15:30:39 drmeister The quicklisp part is single threaded, single process.

15:30:57 drmeister I think the quicklisp: hostname is set using CLASP_QUICKLISP_DIRECTORY hang on...

15:31:13 drmeister Yes: https://github.com/clasp-developers/clasp/blob/dev/src/core/bundle.cc#L272

15:31:32 cracauer` that's correct, the question is why redicrets it to clasp/src/lisp/modules/quicklisp during some parts of the process?

15:31:35 drmeister The code in bundle.cc is run at startup to locate all of the directories that clasp needs.

15:32:19 drmeister Ok - that is unexpected news. I don't know how that can happen. Do you know how to reproduce that?

15:38:06 drmeister cracauer`: On a different topic - are you familiar with this idea? https://en.wikipedia.org/wiki/Read-copy-update

15:38:26 drmeister It's a multithread synchronization approach that doesn't require locks.

15:41:29 drmeister I've been trying to figure it out for a while. We have some CS faculty who I could ask but I haven't been able to ask them yet.

15:43:47 stassats that doesn't appear to be a difficult concept

15:44:23 drmeister I'm trying to figure out how to apply it to something complicated like a hash-table.

15:44:45 drmeister I'd need to get every thread in a state where there are no readers of the hash-table to make updates.

15:45:03 stassats copying a hash-table is probably not a great idea

15:45:21 drmeister I was thinking of a hash-table that has an a-list in front of it.

15:45:59 drmeister A gethash first checks the alist (atomic head) and then the hash table.

15:46:24 drmeister (setf gethash) would write to the alist.

15:46:56 drmeister To do an update all threads would have to go into a non-reader state and then I can copy the alist into the hash-table.

15:47:39 drmeister The tricky part is rehashes - especially with location dependency and moving garbage collection. There a gethash can trigger a rehash.

15:48:09 drmeister Does this all sound like I'm applying the idea of read-copy-update or am I out to lunch here?

15:48:23 stassats what are you doing with your hash-tables that simple locking is not enough?

15:49:26 drmeister We had a lot of problems when we used locks on a thread-safe hash-table for find-class. The exclusive access lock bounced from CPU to CPU and everything slowed way, way down.

15:50:01 drmeister I think I'm looking at a similar challenge with *elementary-types* in predlib.lsp.

15:50:35 drmeister I think it's slowing macroexpansion down because *elementary-types* is an alist.

15:50:50 drmeister I could make it a hash-table - but then I have the problems that I had with find-class.

15:51:10 stassats and an alist doesn't have concurrency problems or something?

15:51:12 drmeister I've been looking at RCU as a solution to these sorts of synchronization problems.

15:52:04 drmeister Not really, because the head of the alist is an aligned pointer - I'm guessing that's why we haven't run into problems with it.

15:52:35 stassats an aligned pointer? what does that matter?

15:52:51 stassats and then again, how often are you modifying *elementary-types*?

15:53:19 drmeister Writing to it is atomic or almost atomic- isn't it? There is a very small time window where two threads can screw things up.

15:54:04 stassats there's no such thing as "almost" when it comes to atomic, and adding new elements to an alist is certainly not atomic

15:54:36 drmeister Exactly , we probably aren't modifying *elementary-types* very often - but if I switch it to a hash-table - then it makes it more likely that we will have conflicts between threads that I need to deal with by redesigning things.

15:55:21 drmeister Pushing something to the head of an alist is pretty fast.

15:56:36 drmeister I'm trying to understand why we haven't run into a problem with it. I'm guessing it's because we don't update it much and that it is an alist. Two things together that have let us get by without dealing with this. No? Yes?

15:57:11 stassats the consequences of concurrently modifying a hash-table are more dire, an alist will just lose values, which can't be considered a good outcome either

15:57:37 drmeister But if I change it to a hash-table to hopefully speed up reads - then I really should deal with it. I could just use a thread safe hash-table - if it's not written to often - then there won't be a problem with lock contention.

15:58:34 drmeister I mean - look at this flame graph

15:58:34 drmeister https://usercontent.irccloud-cdn.com/file/jJhtNp4M/out-21015macro.svg

15:58:48 drmeister https://www.irccloud.com/pastebin/zXsWTz3e/

15:58:51 drmeister That

15:58:58 drmeister 's the code that generated it.

15:59:38 stassats i can't make heads or tails in these flame graphs

15:59:47 drmeister It's spending 20% of the time in canonical-type. That looks up and possibly modifies *elementary-types*

16:00:23 drmeister They are the hottest thing in profiling.

16:01:30 kpoeck it also spends 50% in gc

16:01:58 stassats an alist can be made thread-safe with just a cas-loop, there's that

16:03:55 drmeister I'll get more data and try some stuff.

16:04:12 drmeister kpoeck: Yes - that too.

16:05:09 drmeister Hey karlosz_ I generated a flame graph for a macroexpansion.

16:05:26 drmeister This code... https://www.irccloud.com/pastebin/MhQfRsos/

16:05:38 drmeister Generates this:

16:05:39 drmeister https://usercontent.irccloud-cdn.com/file/jJhtNp4M/out-21015macro.svg

16:12:30 drmeister Yeesh, the code requires a lot of changes to change *elementary-types* to anything other than an alist.

16:12:35 drmeister Not today.

16:27:29 scymtym drmeister: can you tell me more about the esrap problem?

16:27:42 drmeister Ah yes - I'm in a position to do that now.

16:27:49 drmeister Give me a sec and I'll do some macroexpansions.

16:28:57 drmeister What we know right now is if we comment out that (declare ignore ...) that everything works. When I build with that declare in place - then the quicklisp load of esrap goes nuts and prints endless empty lines.

16:30:27 scymtym in which CL implementation?

16:30:40 drmeister Oh sh*t - can slime macroexpansion handle macrolet?

16:30:43 drmeister Clasp.

16:32:02 scymtym SLIME can handle it to some extend but i have seen inaccurate results in complicated cases

16:34:05 scymtym does the problem happen during macroexpansion or when compiling the expansion?

16:38:07 scymtym the unusual bit in the code in question is shadowing the global macro WITH-CACHED-RESULT via MACROLET

16:38:53 drmeister It's hard to tell where the problem happens. It's during compilation of esrap.

16:39:25 drmeister It may be a problem in clasp/cleavir - I can't rule that out.

16:40:38 scymtym it is strange that the declaration in particular throws it off

16:41:15 drmeister I can't get anywhere with slime macroexpansion so I uncommented that declare and now...

16:41:17 drmeister https://www.irccloud.com/pastebin/OGQpUTWQ/

16:41:36 drmeister It's treating the (declare (ignore ...)) as regular code.

16:42:38 drmeister Here is where it reports the first error

16:42:39 drmeister https://github.com/quicklisp/quicklisp-client/blob/master/dists/quicklisp/software/esrap-20190521-git/src/evaluator.lisp#L106

16:44:53 scymtym one error for each time the local version of WITH-CACHED-RESULT appears

16:46:33 scymtym just as a sanity check: does anything change if you rename the BODY variable in lines 73 and 75?

16:47:49 drmeister I'll check in a moment - I did this...

16:47:52 drmeister https://www.irccloud.com/pastebin/D5gXEWH9/

16:48:07 drmeister That will print a backtrace when with-cached-result is evaluated.

16:49:49 scymtym you could also try putting a PRINT around the body of NAMED-LAMBDA/CACHE-VARIANTS to see what it expands to (should be expanded four times)

16:49:56 drmeister https://gist.github.com/drmeister/6b73a92d4a1937f526e21b14777753ea

16:50:21 drmeister Oh - that's a good suggestion - give me a sec.

16:52:35 cracauer` this is now holding up jupyter:

16:52:37 cracauer` https://pastebin.com/QVe2b4c5

16:52:44 cracauer` reproduced on 2 installations

16:53:14 drmeister cracauer`: Hmm, that's new - but I haven't built jupyter in a while.

16:53:18 drmeister https://www.irccloud.com/pastebin/KZ9zW0Uh/

16:53:31 cracauer` more backtrace:

16:53:32 drmeister cracauer`: Give me a sec

16:53:33 cracauer` https://pastebin.com/zfjFBvXZ

16:53:56 cracauer` Chris, if you want to see for yourself, you can start from the deploy script

16:54:11 cracauer` I pushed everything to make it works up until that point.

16:54:27 cracauer` Just say `./main.sh 1 2 3` in the deploy toplevel dir

16:54:36 cracauer` Well, on Linux I know it works.

16:54:43 cracauer` My macs are still macports-disabled.

16:58:53 scymtym drmeister: that expansion looks ok to me

17:01:28 drmeister scymtym: I can't get insight - I starting to suspect it's a problem in our AST interpreter - because it's got to be happening when the WITH-CACHED-RESULT macrolet function is being evaluated.

17:03:35 scymtym drmeister: sounds plausible

17:04:17 drmeister I apologize for making any suggestion that it might be a problem with esrap.

17:05:56 drmeister Hmm, there nothing in the ast-interpreter that deals specifically with macrolet. macrolets are an environment thing - declares should be gone by the time the ast-interpreter hits them.

17:06:44 scymtym no worries. i wouldn't have been surprised if that section of the code had a problem

17:12:11 drmeister Grrr, I'll probably need Bike's help - I can't even figure out where this is evaluated from

17:15:30 drmeister It's deep in Cleavir.

17:19:14 drmeister ::notify Bike Could you check the discussion here. There is an issue compiling esrap that might be due to Cleavir not handling a (declare (ignore ...)) properly in a macrolet in a backquote in a macrolet. It's esrap evaluator.lisp line 74

17:19:14 Colleen drmeister: Got it. I'll let Bike know as soon as possible.

17:24:52 drmeister cracauer`: It's crashing on a sanity check in myjson.lisp.

17:24:59 drmeister It's here:

17:25:11 drmeister https://github.com/drmeister/cl-jupyter/blob/master/src/myjson.lisp#L257

17:25:31 drmeister That's indicated by this: (myjson.lisp-^257^TOP-COMPILE-FILE)

17:25:47 drmeister That means a toplevel form in myslon.lisp line 257 is on the stack.

17:25:57 drmeister https://github.com/drmeister/cl-jupyter/blob/master/src/myjson.lisp#L257

18:06:29 kpoeck_ Drmeister: there are issues with cleavir and macrolet

18:06:42 drmeister Oh - there are?

18:08:42 drmeister Indeed there are. This issue in particular: https://github.com/robert-strandh/SICL/issues/141

18:09:12 kpoeck_ This is issue 781 in clasp

18:09:45 kpoeck_ Caused by the issue you just quoted

18:09:53 kpoeck_ From cleavir

18:10:13 drmeister There we go - thank you very much!

18:13:02 kpoeck_ #-cst (declare ...)

18:17:19 drmeister Given that it's non-conforming behavior in cleavir I'm loath to ask scymtym to make a change for clasp.

18:19:57 kpoeck_ I meant in the local copy, not upstream

18:20:23 drmeister cracauer`: I can reproduce the problem when I run cando in slime and (ql:quickload "cl-jupyter")

19:03:16 drmeister cracauer`: I found the source of the problem. There is a compiler macro on CONCATENATE that is messed up.

19:03:21 drmeister https://www.irccloud.com/pastebin/Ra0f0jPj/

19:03:46 drmeister https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cmp/opt-sequence.lsp#L137

19:04:55 drmeister https://www.irccloud.com/pastebin/oF5zqQrC/

19:05:00 drmeister Why the heck?

19:07:58 drmeister How the hell do we get all the way to the last quicklisp package at the end of the cando build before this rears its ugly head?

19:11:09 pfdietz There's an offshoot of ansi-tests that's useful for testing compiler macros and other compile-time optimizations on builtins, maybe that would help.

19:11:12 drmeister I pushed a fix.

19:11:28 drmeister pfdietz: That sounds very helpful.

19:11:46 pfdietz Let me see if I can dig up a link.

19:12:27 pfdietz https://github.com/pfdietz/ansi-test

19:12:34 pfdietz Look in the directory random/

19:12:44 pfdietz There are a bunch of "random type prop tests".

19:13:02 drmeister Don't they automatically get tested?

19:13:08 pfdietz They put together calls to various built-ins with random arguments, random type declarations, and see if things work.

19:13:31 pfdietz I don't remember if those are included.

19:13:36 pfdietz (in the usual run)

19:14:12 pfdietz concatenate tests are in random-type-prop-test-13.lsp

19:18:45 pfdietz I need to get that test sute back into a more usable state. It was off being toyed with by others for some years.