libera/#commonlisp - IRC Chatlog
Search
18:51:47
younder
Don't remember the exact details of the discussion, but I remember that. Could someone explain the issues?
19:01:52
gilberth
I mean READ-CHAR must do some dispatch anyways, doesn't matter much whether it's a Gray Stream or some standard stream.
19:07:45
yitzi
younder: There were some objections raised when the proposal was submitted than it contained an excessive amount of generic functions. Most implementations have fast generic dispatch, so this isn't as much of an issue now.
19:11:27
yitzi
There is also "simple streams" which the Allegro people came up with (as I recall). They claimed that Gray streams is too "high level" to be efficient and that the proposal doesn't make it clear what method to implement. Personally, I think their arguments are silly and it really just boils down to "we don't like them." I have used Gray streams to implement streams at a lot of different abstraction levels and I think they are pretty awesome. As usual,
19:14:17
gilberth
What I would like to see is two things: a READ-SEQUENCE that blocks only for the first element read like read(2) and then some means to get all buffered data and then only consume some of it.
19:15:12
gilberth
Both are crucial to avoid a function call per char or byte read. The first is important for network applications while the latter is important for modularity.
19:16:54
yitzi
Well, the spec also implies that LISTEN is only good for character streams, so that is problem.
19:18:16
nij-
Is there a syscall in linux or mac that freezes, saves, and kills a process immediately? Next time the saved file is run, it starts as if it's never killed?
19:18:58
gilberth
Anyhow, ANSI-CL never considered network connections and means to have fast i/o. The number one thing that kills all your performance is doing a function call per char/byte read.
19:20:10
yitzi
nij-: s-l-a-d is implementation specific. Most implementations walk through the object references and serialize them along with closing dynamic libraries, etc. It is highly nontrivial.
19:22:04
gilberth
First of you may want to run cleanup forms. I mean all the threads that are currently running won't run anymore when the dump is started again.
19:23:21
gilberth
Programs aren't created anymore by dumping the contents of the heap and turning that into an executable.
19:24:38
nij-
The diff between the current heap and the init heap (i.e. the heap when a fresh lisp is launched)?
19:24:51
gilberth
There however is some magic. Dynamic libraries were mentioned. And CCL e.g. would invalidate all your foreign pointers. They would point to non-sense in the new image.
19:28:13
gilberth
CCL just kills all threads, runs a final GC, invalidates said pointers, and then writes out the whole Lisp heap. On startup that file is mmap(2)ed and ready for use. It's even immobile. It cannot be mapped to some different address.
19:29:47
gilberth
nij-: No, that would be tricky. C libraries are not prepared for that. A dynamic library on starting the dumped image would still run its init routines. Think of e.g. malloc(3).
19:31:15
gilberth
And when started again the exact dynamic lib might even be a different one. With a different layout of its internal data structures. So the C heap is not dumped.
19:35:28
gilberth
I have an image/core file of about 1GB. When you mmap(2) it this happens instantly. What would happen when you compress it? Also to RAM usage? And to RAM usage when you run it more than once?
19:36:40
gilberth
So all you save is some disk space and trade cycles and RAM for it. How much is 1MB of disk space?
19:40:09
gilberth
See, when the Lisp starts it doesn't need to read all the image. All it has to do is make the heap be present. When you compress it, you must read all of it and decompress it. And the OS doesn't have a clue that what you get is the very same data as before.
19:42:07
gilberth
Not so with mmap(2). The data is just mapped into the address space. There is no need to bring it in from disk at all. Only when objects therein are actually looked at by your program. And for scripts: You may want to run it a second time. Chances are that the pages you looked at are still in the file system cache. And when you run the script twice at the same time, those pages can be shared even.
19:42:52
gilberth
By compressing it you rob the OS of the opportunity to recognize that the heap this time or with this process is mostly the same as last time or with the other process.
19:44:39
gilberth
This is why the text segment usually is r/o. Even very ancient UNIX arranged it so that when you run two copies of /bin/sh the text segment is in RAM only once.
19:51:44
nij-
There's a reddit thread where people asked about how to get smaller CL executables, and a solution.
20:05:27
gilberth
nij-: I'm tired of that decade old discussion. It's more about UNIX and Lisp having a different culture. And besides when people really care about size they shouldn't use the largest CL implementation to begin with. Besides: Again, how much 1MB of disk space? How long does it take to transfer 1MB? This all was somewhat relevant in the 90s at which time a 20MB CMUCL core file was huge. Today it's tiny.
20:10:33
bike
https://github.com/sbcl/sbcl/commit/7976926f8112b708d5927a69923cf7a3dd003c55 pkhuong, apparently
20:13:49
random-nick
I remember reading about a CL for unix-likes which was built around shared objects/dynamic linking, but I can't remember what it was called
20:16:11
reb
The best use case for core compression that I can think of is a situation where you want to distribute an application to many places and would like to reduce the amount of data transferred.
20:23:23
gilberth
I don't recall that well either. But mentioning shared libs made me remembering once having read a paper. There was a "W" in the name. But I might as well mix up things.
20:46:04
dbotton
I am trying to bind *query-io* to my fundamental-character-* based streams using (let (*query-io* (make-two-way-stream in-str out-str)) - the streams are never being called invoke-restart-interactively when in theory should (if I remove the binding shows on console)
21:46:50
aeth
In theory yes, but in practice, unrun code has undiscovered bugs. That's why iteration time between making a change and seeing the change needs to be as short as possible.
22:02:31
nij-
In mezanno, is the underlying CL implementation running as well, or does it get stripped away?
23:11:24
random-nick
it would probably be possible to port another implementation with a native code compiler like SBCL to run on bare metal, but that would be a substantial task on its own and might require adapting/rewriting a bunch of its parts
23:17:17
random-nick
mezzano essentially consists of a CL implementation, a simple kernel (called the supervisor) and some additional stuff like the desktop compositor, simple desktop applications like a REPL application, etc.
23:20:34
random-nick
it's all written in common lisp expect for the bootloader, but the supervisor is written in "supervisor safe" code which is essentially a restricted subset of CL (has restrictions like avoiding allocations, mostly operating with wired memory, I don't really know all of them) and also there are many things written as LAP functions, essentially assembler code written in CL source files
23:31:38
random-nick
oh, and to answer your question, it's all a single CL system and there are no processes (as in separate address spaces/environments/systems), but there are threads (and a scheduler of course)
23:44:45
random-nick
mezzano isn't compatible with any other OS, so any non-CL application would require porting using Iota (which is an LLVM backend which compiles things to some CL code that can run on mezzano), and any CL application would require porting in how it interfaces with the OS
23:45:19
random-nick
unless it uses CL libraries which are already ported to mezzano, e.g. usockets and mcclim
23:51:40
aeth
so, that's, what? one out of dozens? hundreds? not a good sign if someone wants to try it.
23:53:12
aeth
and porting a major web browser by compiling C++ (etc.) to CL would be about as hard of a project