freenode/#clasp - IRC Chatlog
Search
16:11:45
cracauer
Right now it works as users would expect when cando/jupyter people use ./build.sh and other people use ./waf directly
16:14:53
cracauer
3: whether quicklisp compilation (if any) goes into a static directory (not ~/.cache)
16:15:39
cracauer
when it is acting as a distribution builder is uses /opt/clasp/lib/quicklisp and no relocation of fasl files
16:16:33
cracauer
even if you use it as a distribution builder you can request homedir and .cache use by putting CLASP_QUICKLISP_HOMEDIR=1
16:17:22
cracauer
well, I think for now we are good. People who don't want cando and jupyter just use waf, which works as before
16:17:56
cracauer
people who want cando and/or jupyter use build.sh which pulls in both by default, and gets configured via a file
16:18:58
cracauer
As far as defaults go, I think just renaming build.sh to a name indicating that it is for pulling in more stuff will do
16:21:26
cracauer
I guess I need to enforce that the proper env variables are set when the leap executable is being called, though.
16:22:10
drmeister
Can we set things from within Clasp - so we don't have to rely on environment variables?
16:24:01
cracauer
It's what chrome, firefox etc use, too. It is just much easier to debug and if needed to adjust (compared to being inside the binary)
16:25:52
cracauer
for the jupyter builds this is already somewhere in jupyter startup. That's how the docker jupyter has been working for a long while.
18:52:39
drmeister
cracauer: Cool - so can I pull the deploy script and build an /opt/clasp directory on my system?
19:04:50
cracauer
oh and we don't have a shellscript wrapper to set env variables around the lisp executables yet
19:16:48
drmeister
an engineer at Apple or Google (I dunno) - but he's contributing patches to get lldb to work with the gdb/jit API that I haven't gotten to work yet.
19:17:21
drmeister
He's got some demo code that I need to build because it should work. We could check if that also means that profiling its jitted code on linux works.
21:28:03
drmeister
cracauer: Do you know what the expression in esrap that miscompiled with that declare looks like? I didn't have time to debug it and I should provide some more details to scymtym.
21:28:53
drmeister
I did see it crash - it printed blank lines in an endless loop. I did comment out the (declare ignore ...) and my build is working because of it.
22:10:00
drmeister
Sorry - it's been pretty consistent for the past couple of months since we switched to the CST compiler. Bike made some changes to the GF dispatch (he implemented a little generic function virtual machine gfvm).
22:14:08
drmeister
I'm repeating myself - but Nintendo switched to clang and llvm in the past two years.
22:21:59
drmeister
Ha ha - certainly not. They were excited about a 3% increase in performance with BotW when they switched to clang from their proprietary C++ compiler.
22:26:06
karlosz
3% is quite a lot from just changing C compilers... did they use gcc before or something?
22:27:02
drmeister
karlosz: They said they had proprietary tools. They did complain that they didn't like losing access to lexical variables in debugging.
22:27:15
cracauer
I'm going nuts. I have 4 different machines testing different variants and wanting modules and quicklisp locations. I had to start all of them over fro scratch
22:28:03
drmeister
Are we going to have to do some genetic programming to figure out the optimal configuration?
22:30:19
karlosz
especially optimizations which merge instructions and cant merge the line information cleanly
22:31:06
cracauer
that's not what I meant. 27 or 54 minutes is the same damage. The problem is broken trees.
22:37:36
drmeister
We gotta get ASDF to do parallel compilation. It's just way too slow compiling files one after the other.
22:53:51
karlosz
even better: has anyone made flamegraphs for generate-ast vs cst compiler to compare?
22:54:50
drmeister
I'm talking with an engineer about the gdb/jit interface on linux - I suspect but don't know that if I get that working that perf should work as well.
22:57:38
karlosz
anytime anyone complains about the compiler speed i get an itch to look at it again - just because i know it doesn't have to be so
23:11:02
karlosz
well, CST-TO-AST is taking a long time. if i remember correctly, CSTs represent non macro expanded code
23:12:36
karlosz
macroexpansion happens somewhere in between the creation of the original cst and the final ast level
23:12:46
cracauer
I was able to edit esrap-20190521-git/src/evaluator.lisp between it being retried but before compilation bombed.
23:14:13
drmeister
If I raise the crop level to 800 frames I can't render it. Chrome is working very, very slowly.
23:15:11
karlosz
nothing immediately algorithmic jumps out at me from the flamegraph, since it seems like its just walking a tree. so it looks like clos could be slow? it really shouldnt take so much time to convert csts to asts
23:15:53
Bike
is it macro EXPANSION that takes time, or building the macroexpanders themselves, because the latter has been a recurring thing
23:19:21
karlosz
getting rid of call-with-variable-bound is also going to help a lot with clearing up those frames
23:19:48
karlosz
but this is good. i never saw this when i profiled cleavir with clisp because i use clisp's own macroexpander
23:21:21
karlosz
its also a good candidate for the bottleneck precisely because we haven't seen the same issue with generate-ast, which doesnt have cst-to-ast at all
23:24:07
karlosz
cst-to-ast seems to be more than 50% of compile time for setf.lsp if im interpreting it correctly
23:31:25
karlosz
its bizarre especially because im pretty sure this was not in previous flame graphs
23:32:15
karlosz
with the previous flamegraphs my-hir-transformations was the only thing you could even see
23:39:54
drmeister
Bike: link-inline-remove-builtins is 14% - it's good that we are getting rid of it.
23:42:53
karlosz
but i mean yeah, i think we know now at least that cst-to-ast as a whole takes >50% of the compilation time for macro-heavy setf.lsp, so that's something
23:48:27
karlosz
yeah isolating macro functions and getting a minimal test case to profile would be good
23:51:28
karlosz
another general suggestion: this experiment has made me think that the cst-compiler isnt actually slower than generate-ast for the majority of files in the self build process
23:52:01
karlosz
only a few select files in the cst-compiler really blow up in compile time, due to it handling certain things differently
23:52:24
karlosz
so one thing to do would be to add compilation times to each compilation file built during the build process
23:52:44
karlosz
and compare the build times for each individual file to see which files slow down the most
23:53:14
karlosz
my hypothesis is that for most files, there isnt actually much slow down, but we will see a couple of outliers where cst takes much more time
23:54:18
karlosz
but i think the macro expanding thing would be a good first construct to investigate
0:20:48
drmeister
Sounds good - I have an emergency with a broken vacuum cleaner - I have to run to buy some screws.
0:34:35
kpoeck
my pick on adding compilation times to the build is in https://github.com/clasp-developers/clasp/pull/788
0:39:10
kpoeck
tomorrow - late here - i can do what karlosz prosed, compare compilation-times in the build for ast and cst
0:42:23
Bike
drmeister: could you look at https://github.com/clasp-developers/clasp/pull/772 ? i want to merge it but don't understand the problem