libera/#clasp - IRC Chatlog
Search
18:10:00
drmeister
Still no change. I'm changing the name of sharp-!-reader to sharp-!-reader-subshell so I can interogate things easier
18:11:41
yitzi
Try :readtable (let ((r (copy-readtable))) (set-dispatch-macro-character #\# #\! 'sharp-!-reader r) r)
18:40:30
Bike
i did make a change, yes, but i didn't try to actually change what it does, juts how the code is arranged (still possible i messed up of course)
18:52:10
drmeister
https://github.com/cando-developers/cando/blob/main/src/lisp/leap/grammar.lisp#L65
18:52:40
yitzi
No, its this https://github.com/cando-developers/cando/blob/1a8469fe37be68423908f1f8e8bdb512ce78c2c1/src/lisp/cando-jupyter/kernel.lisp#L20-L28
18:54:20
yitzi
I mean, yes the leap parser is treating it as a comment, but it is getting sent to the leap parser b.c. lisp-code-p returns NIL.
19:06:13
yitzi
I think some of the function signatures have changed in llvm15. I had to add an extra argument to DIBuilder::createFunction
19:09:29
drmeister
I don't think I've had a llvm transition that didn't involve a change to a DIBuilder method.
19:10:32
drmeister
Dammit - I got the `Duplicate definition of symbol 'clasp_startup_388'` again when building apptainer.
19:10:36
yitzi
Yeah....the segmentation faults are happening right after auto-compile is loaded...so it is hard to find.
19:12:46
drmeister
Also, put a `(gctools:wait-for-user-signal "Waiting")` just before the form that causes the crash.
19:13:15
drmeister
Then connect to the process with udb and use 'signal SIGUSR1' within udb to start it up again.
19:29:49
Bike
highest c++ stamp seems to be core:class-holder, at 2894. whereas c++-stamp-max is 1643, or shifted over one, 3286
19:34:19
Bike
i don't know what could have caused the clasp_startup_ thing. those numbers are from what variable, image-startup-position?
19:35:11
Bike
hm, is STAMPWTAG_max output by the static analyzer? maybe the static analysis is just stale.
19:36:56
drmeister
the clasp_startup_ thing - I forget the details - I'll look into it in about an hour.
19:37:37
drmeister
There's a unique symbol in each object file that the LLJIT uses to initiate linking of the code.
19:38:48
Bike
i did rearrange how compile-file's arguments work, but i didn't mess with that one, at least not intentionally
19:40:09
Bike
https://github.com/clasp-developers/clasp/blob/main/src/lisp/kernel/cmp/compile-file-parallel.lisp#L328 this would be where the index comes from in cf-parallel, maybe?
19:59:23
yitzi
In irc-store it is pulling out the first contained type in the destination, but I am getting a segmentation fault b.c. there are zero contained types. (in llvm15) ... anybody understand the logic there?
20:02:20
yitzi
So that would just call (llvm-sys:create-store *irbuilder* val destination is-volatile) with no checks?
20:52:11
Bike
rerunning the analyzer did not affect STAMPWTAG_max. womp womp. i have no idea where it's being defined then.
20:55:47
drmeister
It's defined in build/boehmprecise/generated/clasp_gc.cc and build/boehmprecise/generated/init_classes_inc.h
20:56:29
yitzi
It just takes the max of all the stamps in generate-gc-enum and then does (logior stamp-max +max-wtag+)
20:56:49
yitzi
https://github.com/clasp-developers/clasp/blob/1d987a609307e28bd3f6731372ed116f2b0f99ea/src/scraper/code-generator.lisp#L651
21:01:25
Bike
i have STAMPWTAG_comp__Module_O = ADJUST_STAMP(1431), and in the image (core:class-stamp-for-instances (find-class 'cmp:module)) => 2862, shifted one
21:05:24
Bike
i'm seeing that cmp:+c++-stamp-max+ is 1643, but there are c++ classese with higher stamps like 2862
21:11:35
Bike
c++-stamp-max iss defined in llvm-sys:cxx-data-structures-info as make_fixnum(gctools::STAMPWTAG_max)
21:12:43
Bike
right, but i mean, i don't know why class-stamp-for-instances is apparently shifted one more than c++-stamp-max
21:12:48
drmeister
I think this is wrong: `c++-stamp-max iss defined in llvm-sys:cxx-data-structures-info as make_fixnum(gctools::STAMPWTAG_max)`
21:16:50
drmeister
https://github.com/clasp-developers/clasp/blob/main/src/llvmo/llvmoPackage.cc#L228
21:17:11
drmeister
`gctools::STAMPWTAG_FIXNUM<<(gctools::BaseHeader_s::general_mtag_shift-gctools::fixnum_shift)`
21:19:54
drmeister
Like this: ` ENTRY(list, "INSTANCE-STAMP", make_fixnum(static_cast<Fixnum>(gctools::STAMPWTAG_INSTANCE)));`
21:22:11
drmeister
We bind these to dynamic variables (that look like constants +instance-stamp+) in runtime-info.lisp
21:25:15
drmeister
Yeah, any of these dynamic variables that we don't use, remove them - tear them out at the root.
21:26:39
Bike
the dtree compiler and etc use it to figure out where stamps are in objects. if it wants to specialize on a class with a stamp less than +c++-stamp-max+, it looks for it in the header stamp of objects.
21:29:09
Bike
oh-ho, and with this change all the dispatch misses i was seeing are gonzo. very nice.
21:31:46
drmeister
Ok, yes - that was a bug - that was supposed to be shifted one bit - this: `<<(gctools::BaseHeader_s::general_mtag_shift-gctools::fixnum_shift)`
21:32:49
drmeister
So how did anything work? We had lots of dispatch misses - that was the only consequence?
21:34:30
Bike
yeah, i think it just meant the fast path never worked for those classes. but the slow path doesn't bother about +c++-stamp-max+ so it worked ok.
21:35:30
drmeister
So generic functions that dispatch on classes in the upper half of the stamp range would go slow path.
21:37:17
Bike
i'm also going to put in some automatic regression tests like i mentioned, to hopefullly catch this kind of thing without digging through perf output
21:59:44
drmeister
This is set here: https://github.com/clasp-developers/clasp/blob/main/src/core/compiler.cc#L549
22:00:18
drmeister
And here... https://github.com/clasp-developers/clasp/blob/main/src/core/compiler.cc#L659
22:11:02
Bike
it's the serial compile-file function, except it takes an input stream instead of a file
22:11:19
Bike
i moved as much of the file stuff as i could into a compile-file function that both serial and parallel use
22:13:04
drmeister
Those numbers in clasp_startup_xxx is the keyword argument ... (image-startup-position (core:next-startup-position))
23:37:31
drmeister
I'm going to build it right up to the cando-user-install and then go in and see if I can figure out what is going on.
23:43:51
Bike
https://github.com/clasp-developers/clasp/pull/1423 PR for the CLOS stuff and also other stuff.
0:13:51
drmeister
yitzi: it occurred to me that it's failing in apptainer when it's running cando-user-install . So I'm running `./extensions/cando/cando-user-install`
0:50:59
drmeister
I just got `apptainer shell cando.sif` to work - I'll install gdb in the apptainer and try running cando under gdb.
1:32:57
yitzi
I suppose if it is failing during snapshot that could be do to clasp_gc.cc missing the stuff from my set_column addition.
1:46:26
yitzi
extension.fasp is built from the faso from base.fasp and the new fasos in cando. There could be an overlap between them?
1:49:10
yitzi
Basically, the fasos from base are compiled on iclasp with the base stuff loaded first. extensions reuses those and then add the additional fasos of cando which are compiled on top of the base image. \
1:50:50
yitzi
On a different subject...looks like there were a few hickups for llvm15 for cando (specifically bordeaux atomic stuff) ... I guessed that it was more of the llvm-sys:get-contained-type issues...and it looks like cando works now!
1:55:29
drmeister
Every jitDylib object gets object files added to it and each object file has a single exported symbol `clasp_startup_<ObjectID>`
1:57:35
yitzi
Well, it might...cause koga and clasp-builder need to count them for the offset counter into the extension image
2:00:06
drmeister
I think I need to add a command line option to print the ObjectID and jitDylib index every time it adds an object file.
2:29:42
drmeister
The "position = 391" hasn't changed in the new build. Perhaps something else is going on - I'll find out once it's finished building.
2:31:13
drmeister
It's gotta be something different between the host build and the apptainer build and it makes sense that it's something stale in the build-apptainer directory. Now that I've removed it - I'll find out if it fixed it soon.
2:43:01
yitzi
I think there was an old faso left over in the extension image that had a position that was invalidated by the addition of files to the extension image.
2:44:39
drmeister
I don't have the final form. I generated an apptainer that just builds cando and does not run cando-user-install so I can shell in and play with cando.
5:20:14
drmeister
`[LabBuildApp] PermissionError: [Errno 13] Permission denied: '/home/cando/miniconda3/share/jupyter/lab/extensions'`