freenode/#clasp - IRC Chatlog
Search
10:02:14
Colleen
attila_lendvai: drmeister said at 2018.07.05 02:44:52: I'd rather keep it - C++ code size isn't really an issue and faster is better in my eyes.
10:02:14
Colleen
attila_lendvai: frgo said at 2018.07.08 17:28:56: I would like to align with you on how we define a protocol for integrating extensions into the build process automatically.
13:07:15
Colleen
Bike: Kevslinger said 2 hours, 53 minutes ago: this is a reminder for Kevin to ask you a question.
13:07:15
minion
Bike, memo from makomo: the "const-size" version of once-only implemented using the list version: https://plaster.tymoon.eu/view/850#850. no EVAL! i guess the most general way to go about it then is to always write a version which works with an arbitrary number of forms and then derive the const-size version from it.
14:18:23
Bike
turns out we've been masking undefined function style warnings if there are no other warnings
16:03:25
Bike
printing undefined function warnings results in a lot of undefined function warnings, but they're almost entirely spurious
16:14:38
beach
If so, I don't think it takes into account compilation units. It even complains about directly recursive functions.
16:17:58
Bike
there are handlers on the cleavir conditions anyway, because a call to an unknown function being a compile-time error wouldn't really work.
16:25:19
drmeister
I've reached a conclusion - Clasp's compiler is slow because LLVM is slow. Lowering llvm-IR to native code without optimization is expensive - it takes seconds.
16:25:52
drmeister
beach: The rat appears to have escaped by chewing his way out of the wall to the outside world.
16:27:32
drmeister
It's a little difficult to describe - but my dad, when he was in his 80's cut a hole in the side of the house and put a piece of molding on it to cover the hole and allow an electrical wire to pass from inside the house to the outside (not to code!). The cavity in the wall had a hole going into the pantry and this large hole with a piece of molding over it.
16:28:04
drmeister
I had trapped the rat in the cavity in the wall for two days while I waited for it to get hungry and I would trap it when it tried to reenter the pantry.
16:30:33
drmeister
But there was no evidence that it was still in the wall. So I proceeded closing things up.
16:33:39
drmeister
Looking from the outside of the house in to the cavity in the wall through to the hole in the pantry that the rat was using to enter the pantry https://usercontent.irccloud-cdn.com/file/xnjfDumc/1531326730.JPG
16:35:19
beach
Is it that they have some quadratic algorithms and you are compiling bigger chunks that usual.
16:37:21
drmeister
All I know is I turn off all optimizations and I now properly (I think) track all time spent in llvm - and LLVM comes out to 50% to 80% of the time it takes to take a form to a function pointer.
16:38:17
drmeister
I'm going to add these times to the output when clasp compiles files - so that we can see this every time.
16:38:35
frgo
We are calling llvm as a library - we don't start a process always, or do we? (Sorry - might be a very dumb question but ...)
16:39:06
drmeister
We are calling llvm as a library - there are no processes or interprocedural communication going on.
16:39:31
drmeister
This is time spent selecting instructions, assigning registers - all that good stuff.
16:41:51
drmeister
I added a cmp:with-track-llvm-time macro and I wrapped it around every llvm call that takes significant time.
16:43:12
drmeister
Now - this is the bclasp compiler - but the cclasp compiler isn't much different. The cclasp code is a bit better and the cclasp compiler does more work - it kind of balances out.
16:44:24
drmeister
I'm going to print the total compile time and llvm time in the compile-file output - so we can see it everytime we build.
16:55:13
Bike
we do have it compiling individual forms, or something, right? because it seems to have several compilation units per file
17:07:55
Shinmera
Yay, Staple2 is now on Quicklisp so you could test its docbrowser (and Definitions by extension) with clasp now.
17:09:56
Bike
since you asked me that stuff i've cleaned up the source locations a lot, so i'll take a look
18:32:36
Bike
"Could not find single dispatch target symbol[FPMATHTAG]"... don't think i've made hugechanges though
19:28:05
drmeister
The lines that start "Seconds real ..." Those are the real/run/llvm time for the immediately preceding load or compile-file
19:29:08
drmeister
Subtract the llvm time from the real time and you have the time bclasp is spending doing compilation and generating the module.
19:30:03
drmeister
When you run down the entries - it's clear that 50-80% of the time is spent in the llvm library.
19:44:25
Bike
also if i try to look at the result of common-lisp-backtrace-frames in the repl it hangs, so that's something
19:46:00
Bike
swank expects the "print-name"s in the backtrace to be symbols in simple cases, but it's getting strings
19:50:47
Bike
plus sourceinfo from that function is messed up even though other functions in the file are fine
20:00:38
Bike
and my make-array optimizer was slightly wrong. loading other people's code occasionally is helpful...
20:06:41
attila_lendvai
i'm staring at this cryptic build issue... maybe i should have stick to the bootstrap refactor, because clasp build is very hard to debug. so hard, that the focus should be on making that easier...
20:06:51
Shinmera
Bike: If there's anything I can help with to debug build issues with Staple, let me know
20:08:44
attila_lendvai
drmeister: how does that relate to my idea of separating the bootstrap stages?
20:09:03
attila_lendvai
drmeister: if you straight out write a wiki page, then i can read it and ask questions
20:10:20
drmeister
(2) So to avoid long startup times we need to completely eliminate compilation from startup.
20:11:09
drmeister
(3) The current way we startup prevents us from doing some important optimizations
20:11:58
drmeister
fasls eliminate lots of compilation - but GF discriminating functions aren't compiled into fasls.
20:12:15
attila_lendvai
AFAIU, this is moslty orthogonal to my staged bootstrap proposal (modulo extensive changes to the codebase)
20:12:53
drmeister
Also classes are created at startup as the defclass top level form is encountered - at that point there is compilation of accessors
20:13:34
drmeister
Here's how I'm envisioning building will happen. compile-file won't be involved at all.
20:14:05
drmeister
The interpreter will LOAD the aclasp source files and that will change the repl so that loaded forms are compiled by the aclasp compiler.
20:14:45
drmeister
Then the aclasp+bclasp source files will be loaded and JIT compiled, replacing all of the interpreted functions.
20:15:58
drmeister
Then the MPS Arena will be written out to a file and everything will be shut down.
20:16:34
drmeister
To start the system back up again the interpreter will be started, load the MPS Arena file, MPS will rebuild the memory and clasp will proceed from there.
20:17:46
drmeister
Sorry to hit you with that if you've put a lot of work into redoing the build system.
20:18:01
drmeister
I've been really, really, really anxious about everything and frustrated by the slow building.
20:18:24
drmeister
So in the past two weeks I knuckled down and dug into where the time was being spent during compilation.
20:19:02
drmeister
And just thinking about everything - how to solve all of our problems. We are currently in a development cul-de-sac.
20:19:49
drmeister
Bike is developing optimizations that we can't deploy because of the complicated startup/build process.
20:20:29
drmeister
The jupyter lab startup is really slow because of the compilation that is happening at startup.
20:20:44
drmeister
I have to keep developing bandaids to fix these problems and then debug the bandaids.