freenode/#clasp - IRC Chatlog
Search
19:28:05
drmeister
The lines that start "Seconds real ..." Those are the real/run/llvm time for the immediately preceding load or compile-file
19:29:08
drmeister
Subtract the llvm time from the real time and you have the time bclasp is spending doing compilation and generating the module.
19:30:03
drmeister
When you run down the entries - it's clear that 50-80% of the time is spent in the llvm library.
19:44:25
Bike
also if i try to look at the result of common-lisp-backtrace-frames in the repl it hangs, so that's something
19:46:00
Bike
swank expects the "print-name"s in the backtrace to be symbols in simple cases, but it's getting strings
19:50:47
Bike
plus sourceinfo from that function is messed up even though other functions in the file are fine
20:00:38
Bike
and my make-array optimizer was slightly wrong. loading other people's code occasionally is helpful...
20:06:41
attila_lendvai
i'm staring at this cryptic build issue... maybe i should have stick to the bootstrap refactor, because clasp build is very hard to debug. so hard, that the focus should be on making that easier...
20:06:51
Shinmera
Bike: If there's anything I can help with to debug build issues with Staple, let me know
20:08:44
attila_lendvai
drmeister: how does that relate to my idea of separating the bootstrap stages?
20:09:03
attila_lendvai
drmeister: if you straight out write a wiki page, then i can read it and ask questions
20:10:20
drmeister
(2) So to avoid long startup times we need to completely eliminate compilation from startup.
20:11:09
drmeister
(3) The current way we startup prevents us from doing some important optimizations
20:11:58
drmeister
fasls eliminate lots of compilation - but GF discriminating functions aren't compiled into fasls.
20:12:15
attila_lendvai
AFAIU, this is moslty orthogonal to my staged bootstrap proposal (modulo extensive changes to the codebase)
20:12:53
drmeister
Also classes are created at startup as the defclass top level form is encountered - at that point there is compilation of accessors
20:13:34
drmeister
Here's how I'm envisioning building will happen. compile-file won't be involved at all.
20:14:05
drmeister
The interpreter will LOAD the aclasp source files and that will change the repl so that loaded forms are compiled by the aclasp compiler.
20:14:45
drmeister
Then the aclasp+bclasp source files will be loaded and JIT compiled, replacing all of the interpreted functions.
20:15:58
drmeister
Then the MPS Arena will be written out to a file and everything will be shut down.
20:16:34
drmeister
To start the system back up again the interpreter will be started, load the MPS Arena file, MPS will rebuild the memory and clasp will proceed from there.
20:17:46
drmeister
Sorry to hit you with that if you've put a lot of work into redoing the build system.
20:18:01
drmeister
I've been really, really, really anxious about everything and frustrated by the slow building.
20:18:24
drmeister
So in the past two weeks I knuckled down and dug into where the time was being spent during compilation.
20:19:02
drmeister
And just thinking about everything - how to solve all of our problems. We are currently in a development cul-de-sac.
20:19:49
drmeister
Bike is developing optimizations that we can't deploy because of the complicated startup/build process.
20:20:29
drmeister
The jupyter lab startup is really slow because of the compilation that is happening at startup.
20:20:44
drmeister
I have to keep developing bandaids to fix these problems and then debug the bandaids.
20:50:26
attila_lendvai
drmeister: luckily most of my efforst for the bootstrap refactor went into thinking. i need to think about this a bit more now, but i think your proposal will not change much fundamentally in my plans.
20:51:21
attila_lendvai
the problem will remain forever: if you add a new language feature, and you want to use it in the code that implements your language, then you need to go through a bootstrap stage
21:01:17
Bike
https://github.com/trivial-gray-streams/trivial-gray-streams/blob/master/package.lisp#L51 why is there an ellipsis
21:08:59
Bike
are cl:stream-element-type and stuff supposed to be generic with gray streams, or is it a separate package
21:16:35
attila_lendvai
ACTION has added a link to this irc log to the build wiki page: https://github.com/clasp-developers/clasp/wiki/The-Build-Process
21:29:14
scymtym
Bike: does clasp already use eclector? eclector reads "..." as a symbol but it should signal an error
21:30:12
Bike
of course, the "..." is only hit in the first place if trivial-gray-streams lacks implementation support
21:35:26
Shinmera
Bike: Definitions handles definition discovery, Staple handles documentation generation, and Staple-server provides on-the-fly docs in your browser.
21:37:55
Shinmera
Bike: Also see, http://shinmera.github.io/staple/#system http://shinmera.github.io/staple/staple-server/#system
21:38:07
Shinmera
::notify Bike https://irclog.tymoon.eu/freenode/%23clasp?around=1531344883#1531344883
22:07:34
scymtym
making ".." but not ".||." signal an error will probably require a protocol change in eclector
22:07:59
Colleen
Bike: Shinmera said 29 minutes, 52 seconds ago: https://irclog.tymoon.eu/freenode/%23clasp?around=1531344883#1531344883
22:33:07
scymtym
on the bright side, this forces me to tackle some inefficiencies around token escape handling
23:43:26
drmeister
The link time is added on top of real time to get the total time. I will add that to real and run time.
0:32:20
drmeister
It's in a ThirdLaw LLC area that you aren't connected to - so tell me if you are able to see it and edit it.
0:42:23
drmeister
Now - this is a proposal for one way to move forward. The rebuilding of C++ objects is a bit of an unknown for me - but if that's not too bad a problem then I think we can make this work.
0:43:13
Bike
from what i understand, martin wants to compact the arena and then pretty much just write it out verbatim
0:44:08
drmeister
It occurred to me as I was writing it. The bignum example was my best first example. bignums with lots of digits must be stored on the heap - it's probably got a resizable vector of parts that are on the C++ heap. We need to save that somehow and then regenerate it at load time.
0:45:34
drmeister
I describe what I think is a straightforward mechanism obj_save/obj_load that builds off of ideas that are in the MPS documentation - so the Ravenbrook folks will hopefully understand it right away.
0:46:33
drmeister
What it needs is details about objects that I should be able to recover - and maybe limiting ourselves to a subset of classes that can be written out this way.
0:46:55
Bike
i mean like, bignums for example - internally they could have a vector, or maybe a deep linked list or something
0:46:58
drmeister
The static analyzer should be able to help us identify classes that have pointers into the C++ heap.
0:47:45
drmeister
Yeah - but I wouldn't use that at all. I would just generate a text representation base10 of the bignum and write that out. Then recreate the bignum on loading from that string.
0:59:02
drmeister
Using ROOM we know what objects are in memory - and using the static analyzer I can figure out what classes have pointers to C++ memory.
1:03:59
drmeister
A question that is still buzzing around in my head is - "Is there another way" like changing the order in which objects are created at startup.
1:05:01
Bike
well, what we basically want to do is eliminate side effects at startup. that reduces complexity, bootstrap problems, and startup time.
1:34:31
drmeister
What are the side effects that we need to eliminate - (1) classes should all be defined at once at the start (2) functions should be bound to symbols at start.