libera/#clasp - IRC Chatlog
Search
13:42:41
drmeister
Would there be a problem defining `!` (exclamation mark) as a reader macro in cando that would take everything to the next carriage return and invoke (ext:system "xxx")?
13:43:29
drmeister
So - within jupyterlab or slime: `! mkdir foo<cr>` would evaluate `(ext:system "mkdir foo")`
13:47:42
drmeister
I've got the ability to build spiroligomers now and I'm streamlining the workflow on the HPC to build the training molecules before I move on and forget all the complicated back and forth I did in the shell and submitting jobs to the HPC queue.
13:47:56
yitzi
I don't think it is a problem. Its basically like magic commands in the Python shell.
13:48:25
drmeister
If I could do everything from within a jupyter notebook then a jupyter notebook that submits jobs to the HPC queue becomes documentation for the next time I do it.
13:56:46
drmeister
Leap script allows: `foo = 1234`. I'd like to use `#@ foo = 1234` and `#@ foo = { 1 2\<cr> 3 4}`
13:59:31
drmeister
I assume I can't use a reader macro if there are symbols that contain the reader macro sequence.
14:00:17
yitzi
You make [ a dispatch reader macro that reads everything up to the next ]. It then feeds the string to leap-eval.
14:06:27
drmeister
Gah! We already have a `#!` reader macro in clasp . https://github.com/clasp-developers/clasp/blob/main/src/lisp/kernel/lsp/mislib.lisp#L267
14:07:33
drmeister
yitzi: If there is a `[` reader macro - doesn't that mean I can't edit source files that contain symbols like COMPILER:%T*[DUMMY]% ?
14:11:37
drmeister
That would work within a jupyter notebook and if I exported the jupyter input cells as a script it would work within Cando.
14:12:46
drmeister
We could still use: `foo = 1234` in jupyter input cells because that's what users will want - but that can't be exported to a script unless you go back and wrap each leap command in `#[` and `]` - right?
14:14:10
drmeister
yitzi: The current `#!` macro could be there to support the --script option of clasp - does that sound right?
14:20:38
yitzi
It was probably an early attempt to ignore shebangs. It added the --script option so shebangs will never be seen by the reader
14:23:20
yitzi
There are tools to do stuff like that, we could either make sure the tool wraps with #[] when needed or add a feature to Cando's REPL that emulates the Leap/Lisp pattern of Jupyter kernel
14:24:26
yitzi
There is also an ASDF system in common-lisp-jupyter that will convert a lisp file into a Jupyter notebook. Just fyi
14:35:06
yitzi
Well, to be fair Jupyter is it's own repl. It does all the usual repl stuff like binding /, //, etc. ;)
14:36:11
yitzi
The difference is that I compile the code and maintain source code references so you can actually debug step through a cell.
15:10:10
drmeister
So that doesn't sound too bad. I worry though about breaking the slime repl by adding leap script.
15:16:00
drmeister
I invoke a subshell to evaluate commands - that makes `cd` not work because the effect is lost after the cd.
15:17:34
yitzi
I think it would be weird for cd to propagate back to cando. That is not how shell scripts generally work
15:18:54
drmeister
That makes sense - so I should just live with this behavior - it's the best behavior.
15:51:17
drmeister
So `#! <bash-command>` launches a subshell that starts in the current working directory and you remain in that directory after each invocation.
16:01:07
drmeister
I have this thing where we can start a swank server from within cando `(cando-user:start-swank)`. It depended on you defining the SWANK_HOME environment variable for where to find slime.
16:02:00
drmeister
I'm adding another option to set `cando-user:*start-swank*` because I find it super annoying to have to shut down clasp, set the variable, and then start clasp again when I don't have SLIME_HOME set.
16:16:22
drmeister
I've had SLIME_HOME at... `~/.emacs.d/slime`, `~/Development/slime`, and now `~/.emacs.d/elpa/slime-20230131.1950/`
16:19:41
drmeister
yitzi: I made this change to the apptainer build - is it ok for me to push it. It adds some more libraries.
16:23:47
yitzi
On the llvm15 branch you said that it fails when it ClaspJIT_O fails to set the target. How did you determine that? I am currently getting a segmentation fault that I can't seem to trap in gdb
16:30:11
drmeister
This crash is very early so you don't have to deal with the slow gdb/jit registration problem.
16:31:11
drmeister
I added the sharp-bang-subshell.lisp file a few minutes ago and I've done a `koga --deep-clean` and emptied the `~/.cache/common-lisp/`
16:36:00
drmeister
With this `#!` reader macro I should be able to do all of the HPC job preparation and submission workflow from inside the Cando jupyter notebook.
16:37:44
yitzi
0-begin is the names I used to delineate the loading stages, btw. Basically the "tag" files we used to use in bootstrapping.
16:39:25
drmeister
Yeah - I could turn this into a dynamics trajectory. I could limit it to one spiroligomer structure and search conformational space.
16:39:49
drmeister
Imagine a protein surface and a spiroligomer hanging over it like a ship from Space1999
16:41:01
drmeister
The spiroligomer will come down onto the protein surface and it has a bunch of parts sticking out of it like legs and they are going to wiggle around to fit the surface of the protein as the land.
16:42:12
drmeister
https://www.google.com/search?q=space1999+eagle+landing+on+planet&rlz=1C5CHFA_enUS929US929&oq=space1999+eagle+landing+on+planet&aqs=chrome..69i57j33i10i160l2j33i299.6478j0j7&sourceid=chrome&ie=UTF-8#fpstate=ive&vld=cid:905b2edb,vid:XIW3c8iJsm8
16:42:46
yitzi
Oh. I did add screenshot button. I forgot. https://github.com/cando-developers/cando/blob/48c0d666928050a7615ba57bd9d775afa40f67ce/src/lisp/cando-widgets/show.lisp#L275
16:47:24
Bike
cmp:module is a builtin class defined in bytecode_compiler.h. (core:class-stamp-for-instances (find-class 'cmp:module)) => 2862. however, cmp:+c++-stamp-max+ => 1643
16:47:54
Bike
i think this is causing a lot of slow dispatch misses in the new fasl compiler, and possibly other code
16:48:53
Bike
for example, if i (defmethod foo ((x string)) x), every time i do (foo "hello") it dispatch misses, because core:simple-character-string also has a stamp over 1643
16:50:10
Bike
+c++-stamp-max+ i defined from STAMPWTAG_max. maybe it's just another shift confusion?
17:01:52
drmeister
Stamps can be a bit confusing because of wtags and mtags (sorry) - are you sure those are the same kind of stamp value?
17:02:13
drmeister
No - it's ok. If we find deep old bugs that have been slowing down - that's super valuable.
17:02:43
Bike
i am not sure, but dtree.lisp directly compares a class-stamp-for-instances and +c++-stamp-max+, so at least it thinks they're the same kind of stamp value.
17:03:14
Bike
i ran perf on the bytecode slime load and it's taking 24.41% of its time in dispatch-miss, thus leading me here
17:04:43
Bike
i'm going to go off and get lunch and groceries now. this probably isn't as high a priority as all the other stuff you're doing, but i thought i'd mention it, ince it will probably take ma a while to figure out how STAMPWTAG_max i defined
17:05:32
drmeister
The values in the header word of an object have the form stamp | wtag (2bits) | mtag (2bits)
17:06:05
drmeister
A valid object should have mtag as #b00 so it looks like a FIXNUM with the value (stamp | wtag (2bits)) in Common Lisp.
17:09:42
Bike
something else i should do (note to self) is add a programmatic record mode to this clos perf thing, and then i can add regression tests to make sure these misses don't happen in the future
17:37:46
drmeister
yitzi: Do you have any idea why the new `#!` would not produce output on jupyter-lab?
17:38:02
drmeister
I have the reader macro returning `(values)` - does it suppress output in that case?
17:39:50
drmeister
https://github.com/cando-developers/cando/blob/main/src/lisp/cando-user/sharp-bang-subshell.lisp#L23
17:40:03
drmeister
https://github.com/cando-developers/cando/main/src/lisp/cando-user/sharp-bang-subshell.lisp#L23
17:41:14
yitzi
https://github.com/cando-developers/cando/blob/main/src/lisp/cando-user/sharp-bank-subshell.lisp
17:42:43
yitzi
I'm not sure Jupyter will know about the modifications to the readtable.....let me look at the code
17:43:55
drmeister
https://github.com/cando-developers/cando/blob/main/src/lisp/cando-user/sharp-bang-subshell.lisp#L23
17:44:42
yitzi
Sort of https://github.com/yitzchak/common-lisp-jupyter/blob/139ed3edb7f448abe0cf69e3bcf0755afabc0c47/src/cl-jupyter/kernel.lisp#L691
17:45:21
drmeister
Ok, so what does that mean? It's installed in the cl:*readtable* but not the eclector one?
17:46:08
yitzi
No, clj uses eclector even when it's not using clasp....that is just some clasp specific adaptations.
17:46:51
yitzi
Are you sure you are not running a jupyter kernel that uses a clasp build with the old #! macro?
18:00:09
drmeister
Bike: I think there is an issue with compiling and linking - I've been seeing problems like this surface the last couple of days.
18:01:30
drmeister
These startup symbols can get nasty because they have to be unique, or they have to be internal.
18:02:02
drmeister
This came up in the apptainer build - which is worrisome. I'll try it again and see if it goes away.
18:04:19
yitzi
Try just adding :readtable (copy-readtable) here https://github.com/cando-developers/cando/blob/main/src/lisp/cando-jupyter/kernel.lisp#L16
18:07:08
yitzi
clj saves the readtable between cells so that code executed outside the cells doesn't have side effects in the cells.
18:10:00
drmeister
Still no change. I'm changing the name of sharp-!-reader to sharp-!-reader-subshell so I can interogate things easier
18:11:41
yitzi
Try :readtable (let ((r (copy-readtable))) (set-dispatch-macro-character #\# #\! 'sharp-!-reader r) r)
18:40:30
Bike
i did make a change, yes, but i didn't try to actually change what it does, juts how the code is arranged (still possible i messed up of course)
18:52:10
drmeister
https://github.com/cando-developers/cando/blob/main/src/lisp/leap/grammar.lisp#L65
18:52:40
yitzi
No, its this https://github.com/cando-developers/cando/blob/1a8469fe37be68423908f1f8e8bdb512ce78c2c1/src/lisp/cando-jupyter/kernel.lisp#L20-L28
18:54:20
yitzi
I mean, yes the leap parser is treating it as a comment, but it is getting sent to the leap parser b.c. lisp-code-p returns NIL.
19:06:13
yitzi
I think some of the function signatures have changed in llvm15. I had to add an extra argument to DIBuilder::createFunction
19:09:29
drmeister
I don't think I've had a llvm transition that didn't involve a change to a DIBuilder method.
19:10:32
drmeister
Dammit - I got the `Duplicate definition of symbol 'clasp_startup_388'` again when building apptainer.
19:10:36
yitzi
Yeah....the segmentation faults are happening right after auto-compile is loaded...so it is hard to find.
19:12:46
drmeister
Also, put a `(gctools:wait-for-user-signal "Waiting")` just before the form that causes the crash.
19:13:15
drmeister
Then connect to the process with udb and use 'signal SIGUSR1' within udb to start it up again.
19:29:49
Bike
highest c++ stamp seems to be core:class-holder, at 2894. whereas c++-stamp-max is 1643, or shifted over one, 3286
19:34:19
Bike
i don't know what could have caused the clasp_startup_ thing. those numbers are from what variable, image-startup-position?
19:35:11
Bike
hm, is STAMPWTAG_max output by the static analyzer? maybe the static analysis is just stale.
19:36:56
drmeister
the clasp_startup_ thing - I forget the details - I'll look into it in about an hour.
19:37:37
drmeister
There's a unique symbol in each object file that the LLJIT uses to initiate linking of the code.
19:38:48
Bike
i did rearrange how compile-file's arguments work, but i didn't mess with that one, at least not intentionally
19:40:09
Bike
https://github.com/clasp-developers/clasp/blob/main/src/lisp/kernel/cmp/compile-file-parallel.lisp#L328 this would be where the index comes from in cf-parallel, maybe?