libera/#clasp - IRC Chatlog

15:35:28 drmeister Hello again.

15:36:31 jackdaniel hey

15:37:10 drmeister jackdaniel: Due to my unfortunate name choices in the deep past we have a package called "core" and so I called a hunk of compiled fasls that clasp loads into memory an "image" and then I call the thing that we generate when I dump all memory into a file a "snapshot".

15:38:05 jackdaniel OK, so dump(core+image) = snapshot ;?

15:38:06 drmeister People tell me that SBCL made the choice to call their hunks of compiled fasls a "core"s and a dump of all memory that can be reloaded later an "image".

15:38:26 drmeister Clasp: core --- image --- snapshot

15:38:38 drmeister SBCL: something -- core -- image

15:38:57 jackdaniel OK

15:40:00 drmeister Hookay - so I wrote code to generate snapshots to implement SAVE-LISP-AND-DIE and these snapshots can be reloaded later. They can also be linked into the executable so that clasp can start up from a single executable.

15:41:54 jackdaniel so the snapshot is something you load from running clasp?

15:43:22 drmeister It lets you save the state of a running clasp and then reload it later.

15:43:50 drmeister It speeds up startup of our most complex configuration in Cando with jupyter-lab code in around 2 seconds.

15:44:20 jackdaniel OK, but say that you've loaded a bunch of fasl files, how do you dump them into a snapshot so you can reload them?

15:45:06 drmeister I took the code that I wrote to support MPS that fixes pointers and generalized it so that I could compact all of memory into a linear block of bytes and then reverse the process at load time.

15:45:43 jackdaniel is this something like https://dancol.org/pdumperpres.pdf ?

15:46:15 drmeister All fasl code is in memory as object file blocks of bytes. At save time I write those object files into the snapshot and at load time I pass those object files to the llvm JIT and it generates relocated, runnable code.

15:47:24 drmeister Ok, that's going to be a problem for ecl.

15:47:27 jackdaniel why do you need to pass it to llvm jit?

15:47:54 drmeister The llvm jit does the runtime relocation and linking.

15:48:47 drmeister You could use llvm. Lang Hames is developing a runtime linker that works very much like `ld`.

15:48:52 jackdaniel so llvm fixes all already existing pointers to new locations?

15:48:58 drmeister Yep.

15:49:40 drmeister We have two copies of all fasl code in memory at any time. One as an object file and the other as a relocated, runnable block of code/data.

15:51:01 drmeister At snapshot save time I walk all of reachable memory, basically doing one mark/copy cycle and copy all reachable memory into a buffer.

15:51:47 drmeister Then I append all of the object files into another linear buffer in memory.

15:52:35 drmeister I use forwarding pointers for the copied objects and I fix up all the pointers in the buffer so that they point back to their appropriate destination objects in the buffer.

15:53:05 dostoevsky8 ** NICK dostoevsky

15:54:03 drmeister Loading the snapshot is the reverse process. When loading mmap the snapshot buffer into memory and I I allocate objects using the GC and copy the data from the snapshot buffer back into the GC allocated objects.

15:54:32 drmeister I maintain some temporary vectors of GC allocated objects so the GC doesn't cleanup anything while I'm getting everything into place.

15:55:05 drmeister I don't need anything extra from Boehm to achieve this. It's snapshot save/load using a naive garbage collector.

15:55:35 drmeister You know the layout of all your objects, you have far fewer classes than we have. We have more than 500 in Cando.

15:56:23 jackdaniel I need to process this; I will have more questions when I wrap my head around it

15:56:26 jackdaniel thanks

15:56:28 drmeister I use the layout information that we derive from our static analyzer. The plain-old-data in the objects just gets written to the snapshot and read back unmodified.

15:57:02 drmeister Tricky things are vtable pointers and function pointers back into the executable. I had to write my own linker to handle those.

15:57:20 drmeister No worries!

16:00:50 jackdaniel I'm not sure whether I've bragged about that here yet, but I've compiled ecl with emscripten and can run it in a browser ,)

16:02:43 drmeister Never say never but that's something we will probably never be able to do.

16:03:14 jackdaniel heh

16:10:32 yitzi drmeister: How would you feel about adding a bindings to netcdf directly to Cando vs cl-netcdf? Not asking you to do it, just the principle of it.

16:13:58 drmeister I hacked cl-netcdf from some demo I found on the internet - it's far from ideal. You suggest we create C++ bindings for the library in Cando?

16:15:05 drmeister That would be a fine idea. Is there a problem with cl-netcdf other than it being awful?

16:15:19 yitzi Yes. We are not using much from netcdf. Just reading files. It would free us from cffi and static vectors in eclasp.

16:15:59 drmeister We still need static vectors don't we?

16:16:40 yitzi I don't know yet. I put them in clj, but I regret it cause garbage collection is a mess with them.

16:16:44 drmeister The key to using netcdf efficiently is to create a non-moving buffer and shuttle info in and out of it through netcdf.

16:17:55 yitzi I understand. Obviously, we don't want to make it inefficient or slow.

16:18:26 drmeister My recollection is that with static vectors you reduce copying. I think it's because you can give the pointer to the buffer to netcdf and it can do what it wants with it.

16:19:17 yitzi That is correct, although CFFI does some copying also, so once has to be careful.

16:19:26 drmeister I don't remember the details but I distinctly remember the "aha" moment that static vectors make working with netcdf much more efficient.

16:20:44 drmeister Creating bindings from cando is straightforward and orthogonal to that.

16:21:33 yitzi Well, if we can't resolve it with our own bindings then we will have to move the static-vector/netcdf out of leap and into a separate loadable system. Static vectors is doing some CFFI groveling when isn't really compatible with create a lisp only eclasp image.

16:21:50 yitzi s/when/which/

16:22:53 drmeister Oh - got it. We can have static vectors in Cando without the cl static-vector support built in.

16:23:35 yitzi Correct. If we really need it we could put it in the snapshot.

16:23:45 drmeister So you are saying that this: https://github.com/sionescu/static-vectors does groveling? Does anything else use this other than cl-netcdf?

16:24:36 yitzi Yes. Here https://github.com/sionescu/static-vectors/blob/25fac6815ede870bfa91d7de9d8639170df4c6b1/static-vectors.asd#L11-L17

16:25:01 yitzi Plus, I think it is a REALLY bad idea to include cffi in the eclasp image.

16:25:20 drmeister Ah - that's a compelling argument to create netcdf bindings in cando.

16:26:07 drmeister That is an organizing principle for eclasp then isn't it? Avoid CFFI and everything that depends on it?

16:26:24 yitzi We want simple lisp only systems that we can spoof SBCL into thinking it is Clasp during introspection of the ASDF systems so we can build list of the lisp files to compile into the image.

16:26:31 yitzi Yep.

16:27:27 drmeister The jupyter stuff needs CFFI to talk to zmq - what about that?

16:27:35 yitzi Systems that have feature expressions captured by the lisp reader are also and issue, although I can hack in a few kludges for specific systems.

16:28:15 drmeister jupyter stuff goes in the snapshot - got it.

16:28:29 drmeister I'm still wrapping my head around elisp vs slisp.

16:28:47 yitzi I'm moving cando-widgets out of cando-user. Therefore the jupyter stuff will show up in the snapshot, where we have quicklisp.

16:28:48 yitzi Bingo.

16:29:39 drmeister Ok - yes - you make a compelling argument for netcdf bindings in cando.

16:30:02 yitzi Good, hopefully it won't be too bad.

16:30:24 drmeister Pffffft - I knock those out in my sleep.

16:30:44 yitzi This would also make quickclasp not as important once I add more of my own projects to quicklisp.

16:31:10 drmeister After seqan we can do anything.

16:31:14 yitzi sleep + fugue state = x10 code powerup

16:32:08 drmeister I'm finishing up a better PDB loader and then launch-program and then cando-netcdf bindings.

16:33:00 yitzi Ok. If I get there I can make an attempt on cando-netcdf, although you will be light years faster then me.

16:33:05 drmeister I have to check - I kinda remember that netcdf being eclipsed by some other high performance file format.

16:33:22 drmeister In Amber I mean.

16:33:30 yitzi Ahh

16:34:20 drmeister Maybe not - there's no mention about it here: https://ambermd.org/FileFormats.php#netcdf

16:34:28 drmeister I'll talk to my Amber peeps.

16:37:00 yitzi ok.

16:39:54 yitzi Seems like we are only using netcdf:get-dimension, get-vara-double, netcdf:nc-open, netcdf:len, netcdf:get-vara-float, and netcdf:nc-close. Its all in leap/topology and cando-widgets/ngl-util.

16:44:02 yitzi Wow, so static-vectors grovels for size_t. Thats it. I think that is no longer needed in CFFI....

17:49:09 yitzi Systems currently in eclasp produced by my new ASDF groveler. https://plaster.tymoon.eu/view/3229#3229

18:18:07 SAL9000_ ** NICK SAL9000

18:37:05 yitzi drmeiser: On zeus `cando --non-interactive` takes 20.5 seconds for a cando that loads cando-user from quicklisp. This after quicklisp has already compiled cando-user on a previous call to `cando --non-interactive`

18:38:17 yitzi Using the eclasp image this takes 10.9 seconds. If you skip the loading of the rc file it takes 5.2 seconds.

19:24:26 samebchase3 ** NICK samebchase

19:37:04 yitzi drmeister: Also making scando on zeus with a completely empty cache takes 9 minutes. With the eclasp image it takes 2.5 minutes!

19:45:31 yitzi At which point we are back to 2.5 startup time.

20:05:16 yitzi drmeister: My PR for static-vectors got merged. Still uses cffi, but doesn't grovel anymore.

0:18:39 drmeister Ok - I went in a hike with folks from the meeting. It is achingly beautiful here.

0:18:51 drmeister Here

0:27:31 yitzi Look back at the logs for some stats on the eclasp image if you want.

0:27:45 yitzi Just like the last 10 lines.

0:54:44 drmeister This? https://plaster.tymoon.eu/view/3229#3229

1:01:35 yitzi No....i'll summarize, sorry

1:02:33 drmeister Bike: You wrote a function at some point that makes a shallow copy of an instance of a clos class. Do you know where that is?

1:02:49 yitzi It takes 20 seconds for cando to startup without eclasp and while loading quicklisp. WIth eclasp it takes 10 seconds. Without an RC file cando eclasp starts up in 5 seconds.

1:03:35 yitzi Making a snapshot from non-eclasp cando takes 9 minutes. Making a snapshot from eclasp cando takes 2.5 minutes.

1:04:12 drmeister Can you try the serial compiler for the eclasp compilation rather than the parallel compiler? It may speed up startup.

1:04:38 yitzi Yes. The final snapshot still takes two seconds to start.

1:05:02 drmeister Oh - you tried it?

1:05:17 yitzi Not the serial compilation yet.

1:05:54 yitzi I am still going through each ASDF system and making sure every gets groveled correctly. And preparing PR where needed.

1:07:13 drmeister (let ((COMPILER:*COMPILE-FILE-PARALLEL* nil)) <compile-everything>)

1:07:28 yitzi Ok

1:07:45 yitzi Overall, it looks really promising.

1:07:57 drmeister I have some early data that says if we have fewer, larger object files in the snapshot that they link faster.

1:08:18 drmeister Parallel compilation breaks up each source file into one object file per top level form.

1:08:43 drmeister Yes, I agree - it's very promising. Once we have llvm14 it will be even faster.

1:09:49 drmeister I'm in a workshop - only part of my brain is here.

1:10:39 yitzi No worries. Can you put giving me commit on cl-netcdf on your todo list, please?

1:11:47 drmeister Done.

1:11:58 yitzi thamks

1:44:55 Bike drmeister: no, but i can write one again pretty quickly. also we might have one in clasp already, let me check

1:45:56 Bike https://github.com/clasp-developers/clasp/blob/main/src/lisp/kernel/lsp/sharpmacros.lisp#L66-L72 something along these lines, i guess. use allocate-instance to get the new instance and then set all its slots like this

3:18:11 Bike (defun copy (object) (let* ((class (class-of object)) (copy (allocate-instance class))) (dolist (slotd (clos:class-slots class) copy) (when (clos:slot-boundp-using-class class object slotd) (setf (clos:slot-value-using-class class copy slotd) (clos:slot-value-using-class class object slotd))))))