freenode/#clasp - IRC Chatlog
Search
14:47:32
drmeister
I concatenate them into a file 'insanity.txt' (ha ha) and add `__mh_execute_header`
14:48:00
drmeister
Ugh - I was explaining this to Lang on discord and started pasting into this wrong window. (sigh)
15:14:55
drmeister
From everything I read executables only need to export symbols if they load code that calls 'dlsym' on those symbols.
15:16:45
drmeister
So if I want to use 'dlsym' and I don't want to be groveling ELF files and macho files and trying to figure out absolute addresses of symbols - then the problem reduces to exporting the symbols that snapshot save/load needs to call 'dlsym' on.
15:17:23
drmeister
'dladdr' at snapshot save time is not a problem - it faithfully returns the mangled name of every address whether or not it is exported. Thank you whoever the heck implemented 'dladdr'.
18:12:57
drmeister
I'm naively sorting a huge list of addresses that happen to be in order already and that is blowing up the sort.
20:05:47
drmeister
I got snapshot save/load working on macOS - it requires a list of ~12,100 symbols that I currently don't have a streamlined way to generate.
20:19:55
drmeister
Yes - but I have a painful, hand-crafted symbol list that needs to be assembled to make it work. I don't have a good way to automate the symbol table construction.
20:20:42
drmeister
But as a proof-of-concept - yes - the snapshot save/load works on macOS for cando with :cando-jupyter loaded and the startup time is 6.7 seconds when the old time startup time is 46.4 seconds.
20:21:02
drmeister
I'm building everything on linux again - I don't think I have a symbol table problem there.
20:23:24
drmeister
I have to do this because of a decision made in the macOS ld64 linker. It only exports vtable symbol tables for vtables that are "anchored".
20:24:43
drmeister
It's like this - the ld64 linker makes all kinds of decisions about what symbols to export - it's all good for us except for that non-anchored vtable decision.
20:25:16
drmeister
Because of this one bad (for us) decision - I have to construct a complete list of symbols to export myself.
20:28:30
drmeister
Hey stassats - I added the ability for clasp to save and load snapshots of memory.
20:29:37
drmeister
Yeah - from 46.4 seconds on the mac (for our most complicated image) down to 6.7 seconds.
20:30:22
drmeister
I hope to do better in the future - I'm talking with an engineer at Apple who wants to optimize things. Most of the 6.7 seconds is linking thousands of object files.
20:30:51
drmeister
It doesn't run any code on startup. It's all llvm dynamic linking of object files.
20:32:26
drmeister
They are generated by our parallel compiler - it generates one object file per top-level form.
20:34:30
drmeister
I did some profiling of the startup - it's mostly in llvm where the time is being spent - I'll do some more now that I have it working on macOS again.
20:35:04
drmeister
I think it's probably more the number of symbols that need to be dealt with rather than the number of object files. But using the serial compiler would test that.
20:35:52
drmeister
Yes - that was the challenge. I'm also doing this with the boehm GC - with no cooperation other than the ability to walk objects in memory.
20:36:42
drmeister
I can also link a new executable that contains the snapshot embedded within it. So a single executable that contains everything needed to run the lisp session.
20:37:44
drmeister
I needed to do this because 46 seconds startup for our jupyterlab kernel was unacceptable. 6.7 seconds is still painful - but not as bad as 46 seconds.
21:29:15
drmeister
If I use compile-file-serial to build the quicklisp code the startup time drops to 5.1 seconds.