libera/#clasp - IRC Chatlog
Search
14:25:46
drmeister
I'm now hacking the clang ASTMatcher code, putting print statements into it to print addresses of internal data structures to try and figure out what is going on.
14:26:20
drmeister
We set up clang ASTMatchers from Clasp Lisp and they look like they get set up fine but then all of a sudden they are corrupt.
14:27:04
drmeister
With udb I should be able to identify what is doing the stomping as soon as I figure out exactly what is being stomped.
14:27:52
drmeister
I am hacking my own copy of llvm14 and it is installing itself in /opt/llvm14 - we used to use that before llvm14 was available.
14:28:14
drmeister
If you start seeing weirdness, check that you aren't building with /opt/llvm14/bin/llvm-config
14:29:04
drmeister
This is slow and painful debugging. I'm just so glad that we have udb and that I've learned how to work around it's slow debug registration.
14:30:17
drmeister
Bike: I really want compile-file semantics working in the bytecode compiler. The less judiciously chosen compiled hot code we have - the faster udb will be able to work with it.
14:31:44
drmeister
If we can get that down to a few hundred or thousand and the rest is bytecode (with compile-file'd bytecode) then udb will be able to work with it better.
14:35:07
Bike
shipping bytecode should be fine, but i'm more worried about making the bytecode compiler normally accessible
14:41:47
Bike
unrelated: i'm seeing if i can't figure out bytecode wrappers for methods now but this code is kind of confusing
14:42:00
Bike
it kind of looks like we have the exact same class definition repeated twice? in wrappers.h?
14:45:12
drmeister
We can't fully get rid of LambdaListHandler_O until we fix the cases where its use has leaked out into the Common Lisp code.
14:46:30
drmeister
https://github.com/clasp-developers/clasp/blob/vm-boehm/src/lisp/kernel/lsp/defstruct.lisp#L361
14:46:51
drmeister
https://github.com/clasp-developers/clasp/blob/vm-boehm/src/lisp/kernel/lsp/defvirtual.lisp#L23
14:48:25
drmeister
Bike: I am removing LambdaListHandler_O carefully. I'm setting everything up with alternate code that works with bytecode wrappers - I'll do the methods.
14:49:06
drmeister
I want to set it up so we can switch from LambdaListHandler to BytecodeWrapper and then test everything to make sure it works with BytecodeWrapper before throwing the switch and removing the LambdaListHandler code.
14:49:34
Bike
i missed that we use lambda list handlers in defstruct and defvirtual, let me see about removing that
14:49:45
drmeister
yitzi: We need to add a static analyzer run to the github Actions. It's too important to leave for occasional running and then hitting problems like I'm trying to debug.
14:50:23
Bike
in defstruct we just use it to check for correctness. that's relatively easy to replace, at least
14:51:14
drmeister
Bike: Yes - that would be a good idea - that's more in your wheelhouse. It does interact with core:ensure-single-dispatch-method - so we will probably need to discuss it.
14:53:28
drmeister
The `:selection-pattern "commandLineOptions"` - that lets us limit the static analyzer to files that contain that substring.
14:54:05
drmeister
It still generates huge amounts of output and takes a while. The action will need to deal with that.
14:57:45
drmeister
And it's segfaulting deep inside the clang ASTMatcher code inside of private classes.
14:58:24
drmeister
It's awful. They don't have a common root class for everything (C++) so on the outside it looks simple - but inside they have all sorts of `if` statements and dispatching going on. Awful, awful, awful.
15:01:27
drmeister
So - I give it a `DynTypedMatcher &NodeMatch` and then it adds it to different std::vector objects based on what that `NodeMatch` can be type cast to.
15:01:53
drmeister
One (or more) of the std::vector objects that hold these appears to be getting stomped on.
15:35:22
Bike
it looks like the lambda list handler for ensure-single-dispatch-method is only used for a sanity check as to whether the method and gf have the same required arguments
15:37:23
Bike
that said, the SD generic functions also have a lambda list handler, though i'm not clear on where they get it from
17:30:35
Bike
looks like i can delete it ok. though it's another thing we'll need to run the analyzer on.
17:34:02
drmeister
There is a MatchFinder object that is being garbage collected when I don't think it should be.
17:35:03
drmeister
By adding print statements to the clang code I found what address in memory was corrupt.
17:35:49
drmeister
I can see that address being modified by our finalizer code and then before that when we are allocating a MatchFinder object.
17:36:18
drmeister
The MatchFinders are all put into a linked list and the head of that list should be on the stack.
17:36:42
drmeister
Maaaybe - then that would put us all the way to "How the heck does anything work".
17:37:28
drmeister
But I've been there so many times that I've set up a chair and umbrella to make myself comfortable.
17:38:31
drmeister
It will take more time to sort this out. I may have to write more Python debugging tools - like a conservative memory marker - bleh.
17:39:40
Bike
i know sbcl has some fancy tools to track memory leaks, but that's kind of the opposite of this problem
18:25:03
Bike
it still uses lambda list handlers... in a way that can be replaced, but it'll be a little messy
18:41:02
yitzi
Bike: The new list is here https://github.com/clasp-developers/clasp/blob/vm-boehm/src/lisp/cscript.lisp
18:46:55
drmeister
Bike: What direct-calls.lisp did will be replaced with the bytecode wrappers. The BytecodeModule_O will carry the form that generated the bytecode wrapper with them and we will be able to compile them to native code at any time simply using COMPILE.
18:47:28
drmeister
bytecode wrappers will be considered (compiled-function-p #'bytecode-wrapper) -> NIL