freenode/#clasp - IRC Chatlog
Search
8:44:44
attila_lendvai
so, the memory usage of the build is reasonable up until the point of linking. did anyone consider making the linking tasks serial, while everything else remains parallel?
9:11:33
attila_lendvai
ACTION has found this re llvm latest: https://github.com/NixOS/nixpkgs/issues/114828
9:25:36
attila_lendvai
drmeister, re latest llvm: is it just a single fix that is needed? i.e. i could just cherry-pick on top of llvm 12 and make it work? or does the future branch need an entangled web of various llvm developments?
12:13:02
Bike
"missing file: '/Users/bike/src/clasp/build/boehm/generated/initializers_inc.h'" ok then.
13:33:31
drmeister
Bike: I made some changes to improve the C++ interoperation demo. You install the extensions in the 'extensions' directory now. All of the path and building problems for the extensions are then solved because it now integrates with our build system.
13:38:31
drmeister
I'm building in like four different clones - I fixed it in one of them - hang on.
13:51:21
drmeister
I'm squeezing some useless classes out of Cando and changing some class fields to GC managed objects to eliminate hand editing of the static analyzer generated file.
14:22:54
attila_lendvai
drmeister, re llvm and the future branch: is it 1) just a simple fix that you need from LLVM's git HEAD or 2) new developments in LLVM
14:23:57
drmeister
The LLJIT and JITLink facility has been rapidly advancing motivated by things we needed in clasp.
14:25:16
drmeister
The JITLink facility is a dynamic linker for jitted code - it's only in the last month or so that support for linux advanced to the point where JITLink works on linux.
14:25:20
attila_lendvai
ok, that's a little more complicated nix-wise, but there's work on packaging the latest llvm from git
15:32:27
Bike
drmeister: https://github.com/clasp-developers/clasp/wiki/Manual is the manual - obviously it's sketchy and not much to look at
15:57:36
Bike
drmeister: i remember a few years ago (geez) you mentioned talking with people about how to use the lldb stuff to parse dwarf, and were told you kind of need an external debugger process?
18:03:08
Bike
the backtrace code is fundamentally kind of confusing in that it fills up our custom structure and then we sort of build things off that
18:03:20
Bike
it might be easier to have just a bare "list of return addresses" that we then process
18:03:38
Bike
if we want to support backtraces early this might involve having duplicate code in both C++ and lisp, though
18:45:16
Bike
"../../src/core/debugger.cc:1006:search_symbol_table I disabled symbol list searching for now - use DWARF instead I think" i see.
18:52:18
Bike
so just using the existing functions and a more basic operating_system_backtrace, it seems to get the function names and line numbers for most frames
18:52:40
Bike
though there are some mysterious elements, like having a "Line" of 999905 which is obviously a placeholder, but a "StartLine" of 891 which is correct
18:54:00
drmeister
Regarding finding object files for addresses - that's straightforward. I'll walk you through it in a moment.
18:55:06
drmeister
To be clear - there are "object files" these are the blobs of bytes that contain an Elf or Macho data structure.
18:55:45
Bike
what I mean is that M-. from lisp to a lisp function defined in C++ puts to the wrapper code
18:56:59
drmeister
Then there are the new classes "Code_O" and "Library_O". A Code_O is a stretchy vector blob of bytes that CONTAINS ALL OF THE RELOCATED CODE
18:57:46
drmeister
Furthermore, a Code_O instance points to an ObjectFile_O instance that references an "object file".
18:59:37
drmeister
Here's Library_O: https://github.com/clasp-developers/clasp/blob/future/include/clasp/llvmo/code.h#L218
19:00:38
drmeister
https://github.com/clasp-developers/clasp/blob/future/include/clasp/llvmo/code.h#L157
19:01:02
drmeister
ObjectFile_O: https://github.com/clasp-developers/clasp/blob/future/include/clasp/llvmo/code.h#L41
19:01:28
Bike
okay, so just using the existing core:object-file-address-information function seems to return something coherent for jitted functions already
19:01:58
drmeister
The "object file" is referenced using this: https://github.com/clasp-developers/clasp/blob/future/include/clasp/llvmo/code.h#L45
19:03:12
drmeister
Now - FYI and for future referece, an llvm::MemoryBuffer is interesting - it can point to memory on disk.
19:06:02
drmeister
https://github.com/clasp-developers/clasp/blob/future/include/clasp/core/lisp.h#L324
19:07:56
drmeister
I am not removing anything from them - they are thread-safe - you can walk them and you can add things to them using CAS.
19:09:00
drmeister
The only time I remove anything from them is when I save a snapshot - I wipe out the _AllObjectFiles list and collect garbage a couple of times.
19:09:27
drmeister
Any Code_O, ObjectFile_O instance at that point that isn't rooted gets collected and does not get saved into the snapshot.
19:10:27
Bike
i guess first i should just figure out what's screwing up the backtrace code, snce it looks like the pieces work perfectly well already
19:11:18
drmeister
Oh - also, ObjectFile_O instance point to a Code_O instance after their "object file" is passed to the JIT.
19:12:16
drmeister
I suspect we are using the DWARF functions somewhere. I hollowed out the symbol table stuff in debugging.cc/debug_unix.cc/debug_macos.cc
19:13:03
drmeister
All this Code_O/ObjectFile_O/Library_O stuff - it applies all the way from the interpreter->aclasp->bclasp->cclasp
19:13:48
Bike
i guess there's some kind of two layer thing where when this doesn't work it searches object files, and that's failing. i see
19:15:51
Bike
the problematic search_symbol_table function gets cllaed from elf_loaded_object_callback
19:16:33
drmeister
Compile-file generates faso files, they are just concatenated "object file"s with an index The index bins the "object file"s into JITDylibs - I can tell you about how those fit in later. They aren't that big a deal.
19:18:18
drmeister
The search_symbol_table needs a fresh approach. I think it needs to figure out if the address is in a library and then build a DwarfContext for that library.