freenode/#clasp - IRC Chatlog
Search
20:05:41
drmeister
Nah - I've already got this. The problem looks like a small one now - just figure out how to get these vtable pointers with external linkage.
20:13:10
frgo
So, there are vtable pointers that are exported? If so, I seem to see that you can ignore the ones that are not exported.
20:14:32
drmeister
I can't save an snapshot and then load it back on macOS until I figure out how to make these external.
20:15:23
drmeister
I'm going to collect them in a file and try adding '-exported_symbols_list <symbol-list>' to the link command line.
20:22:21
drmeister
That appears to define the ENTIRE symbol list to export. All sorts of new problems arise if I just list the symbols I want to add to the list of exported symbols.
20:24:06
frgo
Add in the fact that macOS is based on BSD - which, strictly speaking - is not a Unix and even has a Mach-O kernel, then yeah - Oh for crying out loud.
20:30:48
drmeister
If I can use dladdr/dlsym - I'd rather do that. Figuring out the absolute value of symbols that I read from symbol tables from ELF and macho files is a massive headache.
20:31:43
drmeister
For the last couple of weeks I've been trying one thing after another and every approach has had challenges and nightmares.
20:32:40
drmeister
Right now the problem appears to be reduced to "macos vtable pointers are sometimes external and sometimes internal linkage - how do I control it?"
21:01:54
frgo
drmeister: Does clasp use clang with -fvisibility=hidden? (I can't recall if it is even the default now). If so, explicitly making a symbol external is done using something like #define EXTERNAL __attribute__((visibility("default"))) and use that to mark the failing symbols with that. Just a possibility - and not a good one as it needs code changes.
21:04:22
drmeister
I am looking at classes and I can't see any obvious difference between classes that have external vtable symbols and those that have local vtable symbols.
21:11:41
drmeister
Templated ones are a bit part of the problem - but I see regular classes as part of the problem as well.
21:27:37
drmeister
I'm writing small test cases now and compiling them. vtable symbols are all external
22:03:56
drmeister
The vtable symbol for my classes starts out in an object file with external visibility but in the executable some of them become local - so it's the linker (macOS) that makes the call. Googling how linkers decide symbols have external or local visibility.
2:17:05
Bike
i have thought about it but not super deeply. the basic issue is i don't have a sense of how much we depend on vtables
2:17:19
Bike
for actual lisp functions i don't think removing vtables would be very hard, but i don't know what the C++ is doing as well
2:27:21
Bike
i think it's probably a good idea overall, we can save a word in all objects, and we already have to do dispatch through generic functions anyway, might as well use that
2:36:37
drmeister
Uh - or the compiler - actually - never mind that. The problem is that vtable pointer symbols are hidden in the final executable - so dlsym doesn't work with them - but dladdr does work with their addresses.
2:37:17
drmeister
In snapshot save/load during save I take function/method/vtable pointers and lookup their symbols with dladdr - and build symbol tables - that's not a problem.
2:37:42
drmeister
During snapshot load I use dlsym to lookup the address of the symbols - that's where the problem shows up for some vtable pointer symbols.
2:38:06
drmeister
If the vtables are "anchored" - then the vtable symbol is exported and there is no problem.
2:38:36
drmeister
To "anchor" a vtable you need the class to have at least one out-of-line virtual method.
2:39:06
drmeister
I can do that for our classes - most have this - but I'll have to anchor a few of them like Lisp_O.
2:40:06
drmeister
However, the template classes for builtin function wrappers - those I can't anchor because they are template classes - they are created at compile time based on function pointer types - I can't anchor them. There are about 3,000 of them in cando.
2:41:33
drmeister
Now - there is a facility where I can force symbols to be exported (ld option -exported_symbols_list <filename>) - but this is a PITA - it means we have to maintain a list of about 9,000 C++ mangled symbols Bleh Bleh Bleh.
2:42:46
drmeister
The problem is that we do depend on vtables for function objects. 'templatedSizeof' is one of the reasons we need vtables.
2:46:10
drmeister
This is for function objects in general - if the function object wraps a function pointer - that adds 8 bytes. If it wraps a method pointer - that adds 16 bytes.
2:48:02
drmeister
It's not clear to me that we absolutely need vtables though - we might be able to get around the need.
2:48:37
drmeister
It would take some work and some thinking though - the simpler thing would be if we could get these vtable symbols exported.
2:50:50
drmeister
I'm going to change the names of the template classes that derive from Function_O so I can grep them out of nm output.
2:53:56
drmeister
Hmm, I'm talking with Lang again - we've been chatting about this for a couple of hours - that's how I learned all this.