freenode/#clasp - IRC Chatlog
Search
12:41:14
kpoeck
Some exiting progress of Bike and karlosz regarding Cleavir (Version 1). For months I coudn't compile maxima, but now that works again due to improved inlining
12:42:44
kpoeck
It seems to discussions of Bike and karlosz are very productive (although hard to follow)
12:44:14
kpoeck
Regarding performance compiling "generated-encodings.lsp". I redid a flamegraph on macosx, https://kpoeck.github.io/out-37324.svg
12:45:15
kpoeck
Whatever the reason is, I only get helpfull flamegraph if I use "CLASP_BUILD_MODE = 'object'"
12:46:24
kpoeck
drmeister said it should work for CLASP_BUILD_MODE = 'faso', but it doesn't really work for me, svg are generated, but the content is reduced (the lisp part seem to be missing)
12:48:07
kpoeck
I also tried metering with Swank-metering. The results says that most of the compilation time is spent in CLASP-CLEAVIR::SIMPLE-EVAL-CST
12:50:04
Bike
equivalent to saying most compilation time is in cst-eval, aka evaluating things in eval-when, i think
12:53:49
kpoeck
(time (ext::process-encodings-file)) -> Time real(2.997 secs) run(2.997 secs) consed(384374048 bytes) unwinds(69414)
12:59:30
drmeister
kpoeck: There is another option that you need to provide to the build to get symbols in backtraces.
13:00:41
drmeister
That is because the unix 'backtrace(...)' function doesn't have access to any info for JITted frames and interpreted frames.
13:01:31
drmeister
So DEBUG_JIT_LOG_SYMBOLS writes a file to /tmp/perf-<pid>.map and the flame graph tool uses it to fill in missing symbol information.
13:02:00
drmeister
It's an extra file that clasp generates and so only developers will need to turn it on.
13:03:56
drmeister
I ran into a problem building cando - it hangs when it's installing it - I'm trying to figure out why. (sigh)
13:06:05
drmeister
To integrate cando into clasp someone would clone https://github.com/cando-developers/cando.git extensions/cando
13:06:58
yitzi
I know...the clasp stuff used to be on separate RUN lines. I just left the cando line there as a reminder to do it at some point.
13:06:59
drmeister
./waf build_cboehm needs cando installed in the clasp directory hierarchy before you build everything - it will build the extension into clasp automatically.
13:08:29
yitzi
Have you committed your changes? Maybe I can try after I get back from my morning walk if you haven't figured it out yet.
13:10:27
drmeister
This is what I've done so far - I added the lines to install cando. It's hanging on: RUN ./waf install_cboehm
13:11:20
drmeister
I think it's trying to build the quicklisp code for cando but 'waf' - our build system that we all love to hate suppresses the output and so we have no idea where it's hanging. Thanks 'waf'.
13:15:52
yitzi
It could? No idea really. The installed .clasprc references the quicklisp installed by SBCL earlier in the dockerfile
13:16:07
drmeister
Also cando doesn't need any quicklisp local packages at that stage - it supplies it's own quicklisp systems and downloads others from the standard quicklisp repositories.
14:15:42
kpoeck
Bike cleavir-env:cst-eval seems to be called for every top-level form, see https://gist.github.com/kpoeck/c7124d975e99806570262041940b9f9c
14:16:24
Bike
that's not every toplevel form, it's the eval-whens in the toplevel forms in the file you picked
14:19:52
Bike
all of thse hopefully don't even get to the ast interpreter, sincce they're just progn, quote, and function calls
14:20:37
kpoeck
Didn't take implicit eval-when into account, so now I finally understand what you said
14:56:50
drmeister
It looks like we haven't submitted a pull request for bordeaux-threads. I'm going to submit one.
14:59:26
Bike
alright, if i have the ast interpreter check for interpretability ahead of time, the compile time drops from "100s-120s" according to karlosz to like 80
14:59:50
Bike
for asdf. still more than the compiler, so it's still stupid. if i can get the compiler to work on an ast directly that ought to be fixable
15:10:38
kpoeck
drmeister I honestly believe you made a pull request to bordeaux-threads with things there are already in there
15:14:16
kpoeck
drmeister your pull request only touches the file bordeaux-threads.asd adding clauses that are already there
15:14:48
kpoeck
Please look at https://github.com/sionescu/bordeaux-threads/blob/master/bordeaux-threads.asd
15:14:56
drmeister
Yeah - I'm looking at it and I don't get why github thinks those changes need to be made.
15:19:16
Bike
i don't understand. the clasp change is in sionescu's repo, right? and that's what quicklisp uses.
15:19:50
kpoeck
But in quicklisp for every single component one can specify whether a release is taken, or current git
15:20:47
drmeister
Excellent - something good came out of this. As kpoeck said - Stelian said one minute ago that he'd do a release.
15:21:19
yitzi
drmeister: so moving the clone of bordeaux-threads earlier in the Docker should fix it?
15:21:45
kpoeck
Look at https://github.com/quicklisp/quicklisp-projects/blob/master/projects/bordeaux-threads/source.txt
15:22:01
kpoeck
There it says latest-github-release https://github.com/sionescu/bordeaux-threads.git
15:22:20
drmeister
yitzi: The issue is that bordeaux-threads on quicklisp is out of date and so we always have to install a fork of it. I. GET. BIT. BY. THIS. EVERYTIME. I. INSTALL. CANDO. ANYWHERE.
15:24:35
Bike
"release" being like a specific thing sionescu has to do. quicklisp isn't just using the latest commit in the repo.
15:25:27
kpoeck
Shinmera software si configured to use the lates commit on git, I like that better e,g. https://github.com/quicklisp/quicklisp-projects/blob/master/projects/dissect/source.txt
15:26:01
drmeister
Ok. Now I understand. How do we figure out what the latest release looks like other than wiping out my local-projects/bordeaux-threads and pulling one from quicklisp.
15:26:19
drmeister
It's still going to take days or weeks for this to resolve - even if he releases - right?
15:28:18
kpoeck
in my local project of bordeau-threads i did git remote add upstream https://github.com/sionescu/bordeaux-threads.git
15:29:43
drmeister
yitzi: Right now I'm putting everything after building cando because that takes a couple of hours.
15:30:23
kpoeck
Unfortunately no, works fine on macosx and your linux box, but is freaking slow on the buildbot
15:33:11
drmeister
Yes - I'm working on a Dockerfile to build cando with jupyter notebook/jupyterlab
15:34:23
yitzi
drmeister: I've started rebuilding on my end by moving the clone of bordeaux-threads earlier to see if that works.
16:28:28
drmeister
yitzi: Excellent - thank you. I tend to knock these together to avoid long delays and then rearrange them once it does what I want.
16:51:45
kpoeck
drmeister Bike as an immediate solution to the "generated-encodings.lsp" problem I could lazily load the data. That should make the build much faster, but would be a 3 seconds delay on first use
19:37:02
kpoeck
drmeister could you please review https://github.com/slime/slime/pull/562 for slime? Adds profiling for clasp in slime
19:37:40
drmeister
yitzi: I'm back - I'm slowly trying to get cando to build in the docker image. I've had trouble with the quicklisp compilation step - I'm trying to figure out what is going on.
19:38:55
drmeister
There were two problems. One - it can't find /usr/local/bin/cclasp-boehm that is surprising.
19:39:44
drmeister
I've made a change to clasp that forces that step at the end of install that didn't show output to actually show output - I'm testing it in the docker container.
19:40:04
drmeister
It's really, really, really annoying that waf suppresses output - it drives me insane.
19:40:47
drmeister
Huh - ok - I thought I saw it trying to run /usr/local/bin/ccando-boehm maybe I have the wrong path.
19:42:01
drmeister
kpoeck: How do I review the slime pull request? Can I pull it into my local slime?
19:42:29
drmeister
yitzi: I've moved to a faster machine and I gave docker 18 cores. Hopefully I can get through this faster.
19:46:04
kpoeck
drmeister git remote add karsten https://github.com/kpoeck/slime.git git pull karsten master
19:46:15
selwyn
seems to me that bordeaux-threads can be put into quickclasp? you would only have to install quickclasp when installing cando, which is one line of common lisp. i could do this myself if it helps
19:47:04
drmeister
selwyn: A new release of bordeaux-threads is about to be made. I think using quickclasp for this will push the problem into the future.
20:05:45
karlosz
generated-encodings.lisp started freezing up for me as well on mac, so i suspect it might actually be a problem with the wscript.config maybe
20:07:28
karlosz
i'm thinking it would be better right now to not inline the arithmetic operators if there is no associated type information with any of the operands
20:07:54
karlosz
since i don't really see what inlining adds in that case, besides bloating the instruction cache and giving llvm much more to chew on
20:08:15
karlosz
this would be a first step to having something like a monomorphized approach to inlining
20:09:18
karlosz
as is, i don't think the compiler-macro machinery is able to detect the presence of types though, so making the decision would have to be at the HIR level, but by then the possibility to inline is gone, correct?
20:20:05
karlosz
breaking down into binary operations is standard, that's a good normalization to make
20:20:32
karlosz
you only need the full &rest shebang if you're calling indirectly like with mapcar #'+ or something
20:25:32
drmeister
kpoeck: But how does the profiling get timing data? What is the resolution of the clock?