freenode/#clasp - IRC Chatlog
Search
20:05:47
drmeister
I got snapshot save/load working on macOS - it requires a list of ~12,100 symbols that I currently don't have a streamlined way to generate.
20:19:55
drmeister
Yes - but I have a painful, hand-crafted symbol list that needs to be assembled to make it work. I don't have a good way to automate the symbol table construction.
20:20:42
drmeister
But as a proof-of-concept - yes - the snapshot save/load works on macOS for cando with :cando-jupyter loaded and the startup time is 6.7 seconds when the old time startup time is 46.4 seconds.
20:21:02
drmeister
I'm building everything on linux again - I don't think I have a symbol table problem there.
20:23:24
drmeister
I have to do this because of a decision made in the macOS ld64 linker. It only exports vtable symbol tables for vtables that are "anchored".
20:24:43
drmeister
It's like this - the ld64 linker makes all kinds of decisions about what symbols to export - it's all good for us except for that non-anchored vtable decision.
20:25:16
drmeister
Because of this one bad (for us) decision - I have to construct a complete list of symbols to export myself.
20:28:30
drmeister
Hey stassats - I added the ability for clasp to save and load snapshots of memory.
20:29:37
drmeister
Yeah - from 46.4 seconds on the mac (for our most complicated image) down to 6.7 seconds.
20:30:22
drmeister
I hope to do better in the future - I'm talking with an engineer at Apple who wants to optimize things. Most of the 6.7 seconds is linking thousands of object files.
20:30:51
drmeister
It doesn't run any code on startup. It's all llvm dynamic linking of object files.
20:32:26
drmeister
They are generated by our parallel compiler - it generates one object file per top-level form.
20:34:30
drmeister
I did some profiling of the startup - it's mostly in llvm where the time is being spent - I'll do some more now that I have it working on macOS again.
20:35:04
drmeister
I think it's probably more the number of symbols that need to be dealt with rather than the number of object files. But using the serial compiler would test that.
20:35:52
drmeister
Yes - that was the challenge. I'm also doing this with the boehm GC - with no cooperation other than the ability to walk objects in memory.
20:36:42
drmeister
I can also link a new executable that contains the snapshot embedded within it. So a single executable that contains everything needed to run the lisp session.
20:37:44
drmeister
I needed to do this because 46 seconds startup for our jupyterlab kernel was unacceptable. 6.7 seconds is still painful - but not as bad as 46 seconds.
21:29:15
drmeister
If I use compile-file-serial to build the quicklisp code the startup time drops to 5.1 seconds.
2:37:40
Bike
https://llvm.org/doxygen/classllvm_1_1PointerUnion.html til llvm uses tagged pointers internally
2:40:32
drmeister
I'm trying to come up with a way of generating a list of symbols to export automatically from just the object files.
2:58:50
drmeister
I'm going to have a weird symbol file that I'll concat into the symbol file I construct by scraping object files. We can look through it and remove anything that we don't need.
2:59:58
drmeister
I've got code that generates an 85,747 symbol file that is almost a superset of the 12,000 symbol file except for a few weird symbols like _cc_throw.
3:02:10
drmeister
Ah - they might be there but not exported. But my handcrafted symbol file might have picked them up - not because we need them exported - but they just got picked up somehow.