freenode/#clasp - IRC Chatlog
Search
23:39:12
drmeister
I'm well on the way to getting the atomic/fasl-as-directory thing to work - but it turns out it's not the problem for the fasl debug-info problem.
23:40:46
drmeister
Each defun is a top-level form and the ASTs are compiled serially and then the HIR and llvm parts are done in separate threads.
23:41:48
drmeister
Then compile-file-parallel generates two object files. foo_12.o and foo_13.o (the numbers are arbitrary).
23:42:10
drmeister
When I link these two object files into a fasl file the fasl file contains a thing called a 'debug-map'.
23:42:35
drmeister
The debug-map lists the object files that it contains and for each object file there is a list of symbols that it provides.
23:43:06
drmeister
When dsymutil is run it reads the .fasl file and the debug-map and then goes back to the object files to get the debug info.
23:45:32
drmeister
Then only foo_12.o shows up in the foo.fasl debug-map - even though the foo.fasl link looks like it went fine.
23:48:34
drmeister
Now, in the link command I'm using some switches that I don't quite understand...
23:49:17
drmeister
-bundle <<<< This one makes the fasl file a macOS "bundle" library - I think that means you can load it into a running executable and it will link in properly.
1:51:41
drmeister
I think we may need to add a bogus exported function with a linkage that allows collisions.
1:52:52
drmeister
I set up a test case in C++ where I have two files containing private symbols and I think I can reproduce what we need but don't currently generate.
1:54:19
drmeister
When I construct C++ files that have static functions that have their function pointers exported the struct that is static initialized at startup to register the functions has a symbol name that should collide but doesn't.
1:58:43
drmeister
It's a bit difficult to describe. When you link a bunch of .o files into a fasl file inside the fasl file there is a 'debug-map'.
1:59:22
drmeister
Right - I'm guessing that something about our object file symbol tables is confusing the linker.
1:59:38
drmeister
One weird thing about them is they have NO exported symbols - they are all internal.
2:00:59
drmeister
The problem I've had with external symbols is that we get symbol collisions when we reload things.
2:02:19
drmeister
In Clasp we set things up a bit different. We have all internal symbols and there is one static constructor function that is also an internal symbol.
2:03:29
Bike
i'm not really clear on how we can be doing anything if there are no external symbols at all, but it works somehow, i guess
2:04:13
Bike
well so yeah, when you link this into something the linker has to know how to construct static_foo
2:05:32
drmeister
Note that __ZN6ExposeC1Ev and __ZN6ExposeC2Ev are external symbols. They are both defined by foo.o and bar.o - but they don't collide.
2:06:07
drmeister
But if I remove the 'static' prefix to 'int zzz()...' then I get complaints about 'zzz' being a collision.
2:08:49
Bike
well, let's think here... in the case everything's static, the symbol has to be known to the loader, but not to the linker in the same way. when the linker sees two files with the same symbol marked internal it can do some renaming for the final product, i guess.
2:16:33
drmeister
You don't need constructors to be external linkage - because they are in a special table in the .ll file.
2:17:08
Bike
according to the manual on my linux machine, lowercase means local "usually", and uppercase means global
2:18:44
drmeister
Is there a way to get more information on the linkage type of the symbol. These 't' and 'T' are like 1970's unix
2:19:38
drmeister
I think there must be - because in this example there isn't a collision between __ZN6ExposeC1Ev even though they are external from both foo.o and bar.o.
2:20:40
drmeister
If I remove static from static int zzz() I get a collision and the symbols go from 't' to 'T' in the nm output - just like __ZN6ExposeC1Ev and __ZN6ExposeC2Ev are at the beginning. That makes me thing there are different kinds of 'T' symbols.
2:33:06
drmeister
No - they have to have the same calling signature - they are all static constructors.
2:33:07
drmeister
@llvm.global_ctors = appending global [1 x { i32, void ()*, i8* }] [{ i32, void ()*, i8* } { i32 9, void ()* @CLASP-CTOR_36, i8* null }]
2:33:37
Bike
i mean each file has a different constructor, and all of those constructors need to be called.
2:34:31
Bike
odr is like when you have the same function in multiple .cc since it's a template instantiation or something.
2:34:37
drmeister
The linking needs to put them into the list of static constructors in the right order - here we are hoping that the linker does the right thing.
2:35:57
drmeister
I hope (HOPE!) the linker sets up the constructors to be called in the order they appear in the linker command line.
2:42:00
drmeister
Another idea is to export all of these functions as linkonce_odr and not put them in the llvm.global_ctors variable in each object file.
2:42:49
Bike
yeah that sounds safer. though i don't know about the linkonce_odr thing. we could just give them different names, couldn't we?
2:45:46
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cmp/cmpintrinsics.lsp#L800
3:22:31
drmeister
That's converted from the llvm symbol LinkOnceODRLinkage. There is a dash between every transition from lowercase to uppercase.
3:23:08
drmeister
I wish I'd made it a bit more intelligent so that a string of uppercase followed by lowercase like FOOBar got translated to foo-bar.
3:24:28
drmeister
Bike: I think I'll ask you if you can refactor the make-boot-function-global-variable function and add-global-ctor-function so that.
3:25:28
drmeister
and compile-file-parallel probably calls add-global-ctor-function directly and accumulates the names of ctor functions...
3:26:11
drmeister
so that compile-file-parallel can compile a function that calls all the ctors in sorted order.
3:26:46
drmeister
They won't be ctors anymore - they will be regular functions that look like ctors for simplicity. The function that calls them all will become the only global ctor.
3:27:58
drmeister
All I'm going to try to determine tonight is if by making these ctors link-once-odrlinkage - if that creates entries in the debug-map of the fasl.
3:28:22
drmeister
We may get luck and the linker may also order the ctors properly, but I agree with you, I don't want to depend on that.
3:42:35
drmeister
I'm linking everything together here to create the bclasp fasl file. The external linkage is probably screwing that up.
4:20:25
drmeister
We accumulate the names of the ctor function in each module and then compile a module that just contains a bunch of declarations of those ctor functions and a llvm.global_ctors global variable that lists them all in order.