libera/#clasp - IRC Chatlog
Search
14:06:27
drmeister
Gah! We already have a `#!` reader macro in clasp . https://github.com/clasp-developers/clasp/blob/main/src/lisp/kernel/lsp/mislib.lisp#L267
14:07:33
drmeister
yitzi: If there is a `[` reader macro - doesn't that mean I can't edit source files that contain symbols like COMPILER:%T*[DUMMY]% ?
14:11:37
drmeister
That would work within a jupyter notebook and if I exported the jupyter input cells as a script it would work within Cando.
14:12:46
drmeister
We could still use: `foo = 1234` in jupyter input cells because that's what users will want - but that can't be exported to a script unless you go back and wrap each leap command in `#[` and `]` - right?
14:14:10
drmeister
yitzi: The current `#!` macro could be there to support the --script option of clasp - does that sound right?
14:20:38
yitzi
It was probably an early attempt to ignore shebangs. It added the --script option so shebangs will never be seen by the reader
14:23:20
yitzi
There are tools to do stuff like that, we could either make sure the tool wraps with #[] when needed or add a feature to Cando's REPL that emulates the Leap/Lisp pattern of Jupyter kernel
14:24:26
yitzi
There is also an ASDF system in common-lisp-jupyter that will convert a lisp file into a Jupyter notebook. Just fyi
14:35:06
yitzi
Well, to be fair Jupyter is it's own repl. It does all the usual repl stuff like binding /, //, etc. ;)
14:36:11
yitzi
The difference is that I compile the code and maintain source code references so you can actually debug step through a cell.
15:10:10
drmeister
So that doesn't sound too bad. I worry though about breaking the slime repl by adding leap script.
15:16:00
drmeister
I invoke a subshell to evaluate commands - that makes `cd` not work because the effect is lost after the cd.
15:17:34
yitzi
I think it would be weird for cd to propagate back to cando. That is not how shell scripts generally work
15:18:54
drmeister
That makes sense - so I should just live with this behavior - it's the best behavior.
15:51:17
drmeister
So `#! <bash-command>` launches a subshell that starts in the current working directory and you remain in that directory after each invocation.
16:01:07
drmeister
I have this thing where we can start a swank server from within cando `(cando-user:start-swank)`. It depended on you defining the SWANK_HOME environment variable for where to find slime.
16:02:00
drmeister
I'm adding another option to set `cando-user:*start-swank*` because I find it super annoying to have to shut down clasp, set the variable, and then start clasp again when I don't have SLIME_HOME set.
16:16:22
drmeister
I've had SLIME_HOME at... `~/.emacs.d/slime`, `~/Development/slime`, and now `~/.emacs.d/elpa/slime-20230131.1950/`
16:19:41
drmeister
yitzi: I made this change to the apptainer build - is it ok for me to push it. It adds some more libraries.
16:23:47
yitzi
On the llvm15 branch you said that it fails when it ClaspJIT_O fails to set the target. How did you determine that? I am currently getting a segmentation fault that I can't seem to trap in gdb
16:30:11
drmeister
This crash is very early so you don't have to deal with the slow gdb/jit registration problem.
16:31:11
drmeister
I added the sharp-bang-subshell.lisp file a few minutes ago and I've done a `koga --deep-clean` and emptied the `~/.cache/common-lisp/`
16:36:00
drmeister
With this `#!` reader macro I should be able to do all of the HPC job preparation and submission workflow from inside the Cando jupyter notebook.
16:37:44
yitzi
0-begin is the names I used to delineate the loading stages, btw. Basically the "tag" files we used to use in bootstrapping.
16:39:25
drmeister
Yeah - I could turn this into a dynamics trajectory. I could limit it to one spiroligomer structure and search conformational space.
16:39:49
drmeister
Imagine a protein surface and a spiroligomer hanging over it like a ship from Space1999
16:41:01
drmeister
The spiroligomer will come down onto the protein surface and it has a bunch of parts sticking out of it like legs and they are going to wiggle around to fit the surface of the protein as the land.
16:42:12
drmeister
https://www.google.com/search?q=space1999+eagle+landing+on+planet&rlz=1C5CHFA_enUS929US929&oq=space1999+eagle+landing+on+planet&aqs=chrome..69i57j33i10i160l2j33i299.6478j0j7&sourceid=chrome&ie=UTF-8#fpstate=ive&vld=cid:905b2edb,vid:XIW3c8iJsm8
16:42:46
yitzi
Oh. I did add screenshot button. I forgot. https://github.com/cando-developers/cando/blob/48c0d666928050a7615ba57bd9d775afa40f67ce/src/lisp/cando-widgets/show.lisp#L275
16:47:24
Bike
cmp:module is a builtin class defined in bytecode_compiler.h. (core:class-stamp-for-instances (find-class 'cmp:module)) => 2862. however, cmp:+c++-stamp-max+ => 1643
16:47:54
Bike
i think this is causing a lot of slow dispatch misses in the new fasl compiler, and possibly other code
16:48:53
Bike
for example, if i (defmethod foo ((x string)) x), every time i do (foo "hello") it dispatch misses, because core:simple-character-string also has a stamp over 1643
16:50:10
Bike
+c++-stamp-max+ i defined from STAMPWTAG_max. maybe it's just another shift confusion?
17:01:52
drmeister
Stamps can be a bit confusing because of wtags and mtags (sorry) - are you sure those are the same kind of stamp value?
17:02:13
drmeister
No - it's ok. If we find deep old bugs that have been slowing down - that's super valuable.
17:02:43
Bike
i am not sure, but dtree.lisp directly compares a class-stamp-for-instances and +c++-stamp-max+, so at least it thinks they're the same kind of stamp value.
17:03:14
Bike
i ran perf on the bytecode slime load and it's taking 24.41% of its time in dispatch-miss, thus leading me here
17:04:43
Bike
i'm going to go off and get lunch and groceries now. this probably isn't as high a priority as all the other stuff you're doing, but i thought i'd mention it, ince it will probably take ma a while to figure out how STAMPWTAG_max i defined
17:05:32
drmeister
The values in the header word of an object have the form stamp | wtag (2bits) | mtag (2bits)
17:06:05
drmeister
A valid object should have mtag as #b00 so it looks like a FIXNUM with the value (stamp | wtag (2bits)) in Common Lisp.
17:09:42
Bike
something else i should do (note to self) is add a programmatic record mode to this clos perf thing, and then i can add regression tests to make sure these misses don't happen in the future
17:37:46
drmeister
yitzi: Do you have any idea why the new `#!` would not produce output on jupyter-lab?
17:38:02
drmeister
I have the reader macro returning `(values)` - does it suppress output in that case?
17:39:50
drmeister
https://github.com/cando-developers/cando/blob/main/src/lisp/cando-user/sharp-bang-subshell.lisp#L23
17:40:03
drmeister
https://github.com/cando-developers/cando/main/src/lisp/cando-user/sharp-bang-subshell.lisp#L23
17:41:14
yitzi
https://github.com/cando-developers/cando/blob/main/src/lisp/cando-user/sharp-bank-subshell.lisp
17:42:43
yitzi
I'm not sure Jupyter will know about the modifications to the readtable.....let me look at the code
17:43:55
drmeister
https://github.com/cando-developers/cando/blob/main/src/lisp/cando-user/sharp-bang-subshell.lisp#L23
17:44:42
yitzi
Sort of https://github.com/yitzchak/common-lisp-jupyter/blob/139ed3edb7f448abe0cf69e3bcf0755afabc0c47/src/cl-jupyter/kernel.lisp#L691
17:45:21
drmeister
Ok, so what does that mean? It's installed in the cl:*readtable* but not the eclector one?
17:46:08
yitzi
No, clj uses eclector even when it's not using clasp....that is just some clasp specific adaptations.
17:46:51
yitzi
Are you sure you are not running a jupyter kernel that uses a clasp build with the old #! macro?
18:00:09
drmeister
Bike: I think there is an issue with compiling and linking - I've been seeing problems like this surface the last couple of days.
18:01:30
drmeister
These startup symbols can get nasty because they have to be unique, or they have to be internal.
18:02:02
drmeister
This came up in the apptainer build - which is worrisome. I'll try it again and see if it goes away.
18:04:19
yitzi
Try just adding :readtable (copy-readtable) here https://github.com/cando-developers/cando/blob/main/src/lisp/cando-jupyter/kernel.lisp#L16
18:07:08
yitzi
clj saves the readtable between cells so that code executed outside the cells doesn't have side effects in the cells.
18:10:00
drmeister
Still no change. I'm changing the name of sharp-!-reader to sharp-!-reader-subshell so I can interogate things easier
18:11:41
yitzi
Try :readtable (let ((r (copy-readtable))) (set-dispatch-macro-character #\# #\! 'sharp-!-reader r) r)
18:40:30
Bike
i did make a change, yes, but i didn't try to actually change what it does, juts how the code is arranged (still possible i messed up of course)
18:52:10
drmeister
https://github.com/cando-developers/cando/blob/main/src/lisp/leap/grammar.lisp#L65
18:52:40
yitzi
No, its this https://github.com/cando-developers/cando/blob/1a8469fe37be68423908f1f8e8bdb512ce78c2c1/src/lisp/cando-jupyter/kernel.lisp#L20-L28
18:54:20
yitzi
I mean, yes the leap parser is treating it as a comment, but it is getting sent to the leap parser b.c. lisp-code-p returns NIL.
19:06:13
yitzi
I think some of the function signatures have changed in llvm15. I had to add an extra argument to DIBuilder::createFunction
19:09:29
drmeister
I don't think I've had a llvm transition that didn't involve a change to a DIBuilder method.
19:10:32
drmeister
Dammit - I got the `Duplicate definition of symbol 'clasp_startup_388'` again when building apptainer.
19:10:36
yitzi
Yeah....the segmentation faults are happening right after auto-compile is loaded...so it is hard to find.
19:12:46
drmeister
Also, put a `(gctools:wait-for-user-signal "Waiting")` just before the form that causes the crash.
19:13:15
drmeister
Then connect to the process with udb and use 'signal SIGUSR1' within udb to start it up again.
19:29:49
Bike
highest c++ stamp seems to be core:class-holder, at 2894. whereas c++-stamp-max is 1643, or shifted over one, 3286
19:34:19
Bike
i don't know what could have caused the clasp_startup_ thing. those numbers are from what variable, image-startup-position?
19:35:11
Bike
hm, is STAMPWTAG_max output by the static analyzer? maybe the static analysis is just stale.
19:36:56
drmeister
the clasp_startup_ thing - I forget the details - I'll look into it in about an hour.
19:37:37
drmeister
There's a unique symbol in each object file that the LLJIT uses to initiate linking of the code.
19:38:48
Bike
i did rearrange how compile-file's arguments work, but i didn't mess with that one, at least not intentionally
19:40:09
Bike
https://github.com/clasp-developers/clasp/blob/main/src/lisp/kernel/cmp/compile-file-parallel.lisp#L328 this would be where the index comes from in cf-parallel, maybe?
19:59:23
yitzi
In irc-store it is pulling out the first contained type in the destination, but I am getting a segmentation fault b.c. there are zero contained types. (in llvm15) ... anybody understand the logic there?
20:02:20
yitzi
So that would just call (llvm-sys:create-store *irbuilder* val destination is-volatile) with no checks?
20:52:11
Bike
rerunning the analyzer did not affect STAMPWTAG_max. womp womp. i have no idea where it's being defined then.
20:55:47
drmeister
It's defined in build/boehmprecise/generated/clasp_gc.cc and build/boehmprecise/generated/init_classes_inc.h
20:56:29
yitzi
It just takes the max of all the stamps in generate-gc-enum and then does (logior stamp-max +max-wtag+)
20:56:49
yitzi
https://github.com/clasp-developers/clasp/blob/1d987a609307e28bd3f6731372ed116f2b0f99ea/src/scraper/code-generator.lisp#L651
21:01:25
Bike
i have STAMPWTAG_comp__Module_O = ADJUST_STAMP(1431), and in the image (core:class-stamp-for-instances (find-class 'cmp:module)) => 2862, shifted one
21:05:24
Bike
i'm seeing that cmp:+c++-stamp-max+ is 1643, but there are c++ classese with higher stamps like 2862
21:11:35
Bike
c++-stamp-max iss defined in llvm-sys:cxx-data-structures-info as make_fixnum(gctools::STAMPWTAG_max)
21:12:43
Bike
right, but i mean, i don't know why class-stamp-for-instances is apparently shifted one more than c++-stamp-max
21:12:48
drmeister
I think this is wrong: `c++-stamp-max iss defined in llvm-sys:cxx-data-structures-info as make_fixnum(gctools::STAMPWTAG_max)`
21:16:50
drmeister
https://github.com/clasp-developers/clasp/blob/main/src/llvmo/llvmoPackage.cc#L228
21:17:11
drmeister
`gctools::STAMPWTAG_FIXNUM<<(gctools::BaseHeader_s::general_mtag_shift-gctools::fixnum_shift)`
21:19:54
drmeister
Like this: ` ENTRY(list, "INSTANCE-STAMP", make_fixnum(static_cast<Fixnum>(gctools::STAMPWTAG_INSTANCE)));`
21:22:11
drmeister
We bind these to dynamic variables (that look like constants +instance-stamp+) in runtime-info.lisp
21:25:15
drmeister
Yeah, any of these dynamic variables that we don't use, remove them - tear them out at the root.
21:26:39
Bike
the dtree compiler and etc use it to figure out where stamps are in objects. if it wants to specialize on a class with a stamp less than +c++-stamp-max+, it looks for it in the header stamp of objects.
21:29:09
Bike
oh-ho, and with this change all the dispatch misses i was seeing are gonzo. very nice.
21:31:46
drmeister
Ok, yes - that was a bug - that was supposed to be shifted one bit - this: `<<(gctools::BaseHeader_s::general_mtag_shift-gctools::fixnum_shift)`
21:32:49
drmeister
So how did anything work? We had lots of dispatch misses - that was the only consequence?
21:34:30
Bike
yeah, i think it just meant the fast path never worked for those classes. but the slow path doesn't bother about +c++-stamp-max+ so it worked ok.
21:35:30
drmeister
So generic functions that dispatch on classes in the upper half of the stamp range would go slow path.
21:37:17
Bike
i'm also going to put in some automatic regression tests like i mentioned, to hopefullly catch this kind of thing without digging through perf output
21:59:44
drmeister
This is set here: https://github.com/clasp-developers/clasp/blob/main/src/core/compiler.cc#L549
22:00:18
drmeister
And here... https://github.com/clasp-developers/clasp/blob/main/src/core/compiler.cc#L659
22:11:02
Bike
it's the serial compile-file function, except it takes an input stream instead of a file
22:11:19
Bike
i moved as much of the file stuff as i could into a compile-file function that both serial and parallel use
22:13:04
drmeister
Those numbers in clasp_startup_xxx is the keyword argument ... (image-startup-position (core:next-startup-position))
23:37:31
drmeister
I'm going to build it right up to the cando-user-install and then go in and see if I can figure out what is going on.
23:43:51
Bike
https://github.com/clasp-developers/clasp/pull/1423 PR for the CLOS stuff and also other stuff.
0:13:51
drmeister
yitzi: it occurred to me that it's failing in apptainer when it's running cando-user-install . So I'm running `./extensions/cando/cando-user-install`
0:50:59
drmeister
I just got `apptainer shell cando.sif` to work - I'll install gdb in the apptainer and try running cando under gdb.
1:32:57
yitzi
I suppose if it is failing during snapshot that could be do to clasp_gc.cc missing the stuff from my set_column addition.
1:46:26
yitzi
extension.fasp is built from the faso from base.fasp and the new fasos in cando. There could be an overlap between them?
1:49:10
yitzi
Basically, the fasos from base are compiled on iclasp with the base stuff loaded first. extensions reuses those and then add the additional fasos of cando which are compiled on top of the base image. \
1:50:50
yitzi
On a different subject...looks like there were a few hickups for llvm15 for cando (specifically bordeaux atomic stuff) ... I guessed that it was more of the llvm-sys:get-contained-type issues...and it looks like cando works now!
1:55:29
drmeister
Every jitDylib object gets object files added to it and each object file has a single exported symbol `clasp_startup_<ObjectID>`
1:57:35
yitzi
Well, it might...cause koga and clasp-builder need to count them for the offset counter into the extension image
2:00:06
drmeister
I think I need to add a command line option to print the ObjectID and jitDylib index every time it adds an object file.