freenode/#clasp - IRC Chatlog
Search
14:18:23
Bike
turns out we've been masking undefined function style warnings if there are no other warnings
16:03:25
Bike
printing undefined function warnings results in a lot of undefined function warnings, but they're almost entirely spurious
16:14:38
beach
If so, I don't think it takes into account compilation units. It even complains about directly recursive functions.
16:17:58
Bike
there are handlers on the cleavir conditions anyway, because a call to an unknown function being a compile-time error wouldn't really work.
16:25:19
drmeister
I've reached a conclusion - Clasp's compiler is slow because LLVM is slow. Lowering llvm-IR to native code without optimization is expensive - it takes seconds.
16:25:52
drmeister
beach: The rat appears to have escaped by chewing his way out of the wall to the outside world.
16:27:32
drmeister
It's a little difficult to describe - but my dad, when he was in his 80's cut a hole in the side of the house and put a piece of molding on it to cover the hole and allow an electrical wire to pass from inside the house to the outside (not to code!). The cavity in the wall had a hole going into the pantry and this large hole with a piece of molding over it.
16:28:04
drmeister
I had trapped the rat in the cavity in the wall for two days while I waited for it to get hungry and I would trap it when it tried to reenter the pantry.
16:30:33
drmeister
But there was no evidence that it was still in the wall. So I proceeded closing things up.
16:33:39
drmeister
Looking from the outside of the house in to the cavity in the wall through to the hole in the pantry that the rat was using to enter the pantry https://usercontent.irccloud-cdn.com/file/xnjfDumc/1531326730.JPG
16:35:19
beach
Is it that they have some quadratic algorithms and you are compiling bigger chunks that usual.
16:37:21
drmeister
All I know is I turn off all optimizations and I now properly (I think) track all time spent in llvm - and LLVM comes out to 50% to 80% of the time it takes to take a form to a function pointer.
16:38:17
drmeister
I'm going to add these times to the output when clasp compiles files - so that we can see this every time.
16:38:35
frgo
We are calling llvm as a library - we don't start a process always, or do we? (Sorry - might be a very dumb question but ...)
16:39:06
drmeister
We are calling llvm as a library - there are no processes or interprocedural communication going on.
16:39:31
drmeister
This is time spent selecting instructions, assigning registers - all that good stuff.
16:41:51
drmeister
I added a cmp:with-track-llvm-time macro and I wrapped it around every llvm call that takes significant time.
16:43:12
drmeister
Now - this is the bclasp compiler - but the cclasp compiler isn't much different. The cclasp code is a bit better and the cclasp compiler does more work - it kind of balances out.
16:44:24
drmeister
I'm going to print the total compile time and llvm time in the compile-file output - so we can see it everytime we build.
16:55:13
Bike
we do have it compiling individual forms, or something, right? because it seems to have several compilation units per file
17:07:55
Shinmera
Yay, Staple2 is now on Quicklisp so you could test its docbrowser (and Definitions by extension) with clasp now.
17:09:56
Bike
since you asked me that stuff i've cleaned up the source locations a lot, so i'll take a look
18:32:36
Bike
"Could not find single dispatch target symbol[FPMATHTAG]"... don't think i've made hugechanges though
19:28:05
drmeister
The lines that start "Seconds real ..." Those are the real/run/llvm time for the immediately preceding load or compile-file
19:29:08
drmeister
Subtract the llvm time from the real time and you have the time bclasp is spending doing compilation and generating the module.
19:30:03
drmeister
When you run down the entries - it's clear that 50-80% of the time is spent in the llvm library.
19:44:25
Bike
also if i try to look at the result of common-lisp-backtrace-frames in the repl it hangs, so that's something
19:46:00
Bike
swank expects the "print-name"s in the backtrace to be symbols in simple cases, but it's getting strings
19:50:47
Bike
plus sourceinfo from that function is messed up even though other functions in the file are fine
20:00:38
Bike
and my make-array optimizer was slightly wrong. loading other people's code occasionally is helpful...
20:06:41
attila_lendvai
i'm staring at this cryptic build issue... maybe i should have stick to the bootstrap refactor, because clasp build is very hard to debug. so hard, that the focus should be on making that easier...
20:06:51
Shinmera
Bike: If there's anything I can help with to debug build issues with Staple, let me know
20:08:44
attila_lendvai
drmeister: how does that relate to my idea of separating the bootstrap stages?
20:09:03
attila_lendvai
drmeister: if you straight out write a wiki page, then i can read it and ask questions
20:10:20
drmeister
(2) So to avoid long startup times we need to completely eliminate compilation from startup.
20:11:09
drmeister
(3) The current way we startup prevents us from doing some important optimizations
20:11:58
drmeister
fasls eliminate lots of compilation - but GF discriminating functions aren't compiled into fasls.
20:12:15
attila_lendvai
AFAIU, this is moslty orthogonal to my staged bootstrap proposal (modulo extensive changes to the codebase)
20:12:53
drmeister
Also classes are created at startup as the defclass top level form is encountered - at that point there is compilation of accessors
20:13:34
drmeister
Here's how I'm envisioning building will happen. compile-file won't be involved at all.
20:14:05
drmeister
The interpreter will LOAD the aclasp source files and that will change the repl so that loaded forms are compiled by the aclasp compiler.
20:14:45
drmeister
Then the aclasp+bclasp source files will be loaded and JIT compiled, replacing all of the interpreted functions.
20:15:58
drmeister
Then the MPS Arena will be written out to a file and everything will be shut down.
20:16:34
drmeister
To start the system back up again the interpreter will be started, load the MPS Arena file, MPS will rebuild the memory and clasp will proceed from there.
20:17:46
drmeister
Sorry to hit you with that if you've put a lot of work into redoing the build system.
20:18:01
drmeister
I've been really, really, really anxious about everything and frustrated by the slow building.
20:18:24
drmeister
So in the past two weeks I knuckled down and dug into where the time was being spent during compilation.
20:19:02
drmeister
And just thinking about everything - how to solve all of our problems. We are currently in a development cul-de-sac.
20:19:49
drmeister
Bike is developing optimizations that we can't deploy because of the complicated startup/build process.
20:20:29
drmeister
The jupyter lab startup is really slow because of the compilation that is happening at startup.
20:20:44
drmeister
I have to keep developing bandaids to fix these problems and then debug the bandaids.
20:50:26
attila_lendvai
drmeister: luckily most of my efforst for the bootstrap refactor went into thinking. i need to think about this a bit more now, but i think your proposal will not change much fundamentally in my plans.
20:51:21
attila_lendvai
the problem will remain forever: if you add a new language feature, and you want to use it in the code that implements your language, then you need to go through a bootstrap stage
21:01:17
Bike
https://github.com/trivial-gray-streams/trivial-gray-streams/blob/master/package.lisp#L51 why is there an ellipsis
21:08:59
Bike
are cl:stream-element-type and stuff supposed to be generic with gray streams, or is it a separate package
21:16:35
attila_lendvai
ACTION has added a link to this irc log to the build wiki page: https://github.com/clasp-developers/clasp/wiki/The-Build-Process
21:29:14
scymtym
Bike: does clasp already use eclector? eclector reads "..." as a symbol but it should signal an error
21:30:12
Bike
of course, the "..." is only hit in the first place if trivial-gray-streams lacks implementation support
21:35:26
Shinmera
Bike: Definitions handles definition discovery, Staple handles documentation generation, and Staple-server provides on-the-fly docs in your browser.
21:37:55
Shinmera
Bike: Also see, http://shinmera.github.io/staple/#system http://shinmera.github.io/staple/staple-server/#system
21:38:07
Shinmera
::notify Bike https://irclog.tymoon.eu/freenode/%23clasp?around=1531344883#1531344883
22:07:34
scymtym
making ".." but not ".||." signal an error will probably require a protocol change in eclector
22:07:59
Colleen
Bike: Shinmera said 29 minutes, 52 seconds ago: https://irclog.tymoon.eu/freenode/%23clasp?around=1531344883#1531344883
22:33:07
scymtym
on the bright side, this forces me to tackle some inefficiencies around token escape handling
23:43:26
drmeister
The link time is added on top of real time to get the total time. I will add that to real and run time.
0:32:20
drmeister
It's in a ThirdLaw LLC area that you aren't connected to - so tell me if you are able to see it and edit it.
0:42:23
drmeister
Now - this is a proposal for one way to move forward. The rebuilding of C++ objects is a bit of an unknown for me - but if that's not too bad a problem then I think we can make this work.
0:43:13
Bike
from what i understand, martin wants to compact the arena and then pretty much just write it out verbatim
0:44:08
drmeister
It occurred to me as I was writing it. The bignum example was my best first example. bignums with lots of digits must be stored on the heap - it's probably got a resizable vector of parts that are on the C++ heap. We need to save that somehow and then regenerate it at load time.
0:45:34
drmeister
I describe what I think is a straightforward mechanism obj_save/obj_load that builds off of ideas that are in the MPS documentation - so the Ravenbrook folks will hopefully understand it right away.
0:46:33
drmeister
What it needs is details about objects that I should be able to recover - and maybe limiting ourselves to a subset of classes that can be written out this way.
0:46:55
Bike
i mean like, bignums for example - internally they could have a vector, or maybe a deep linked list or something
0:46:58
drmeister
The static analyzer should be able to help us identify classes that have pointers into the C++ heap.
0:47:45
drmeister
Yeah - but I wouldn't use that at all. I would just generate a text representation base10 of the bignum and write that out. Then recreate the bignum on loading from that string.
0:59:02
drmeister
Using ROOM we know what objects are in memory - and using the static analyzer I can figure out what classes have pointers to C++ memory.
1:03:59
drmeister
A question that is still buzzing around in my head is - "Is there another way" like changing the order in which objects are created at startup.
1:05:01
Bike
well, what we basically want to do is eliminate side effects at startup. that reduces complexity, bootstrap problems, and startup time.
1:34:31
drmeister
What are the side effects that we need to eliminate - (1) classes should all be defined at once at the start (2) functions should be bound to symbols at start.