libera/#sicl - IRC Chatlog
Search
11:24:14
nij-
beach I see. Sounds like a dangerous way.. I wonder how they keep their modifications.
11:24:32
nij-
Perhaps.. they have their first lisp image stored, and the rest are written in a file.
11:25:34
nij-
Hmmm but then I wonder, take LW for example (which is developed with the image-based method), how do they support more external features (e.g. OS threads) simply by modifying the image?
11:36:37
beach
And, as I recall from the paper by Krystof, only SBCL (among the implementations written essentially in Common Lisp) can build from source. And they don't evaluate code in the host. So SICL is the only one going one step further.
11:39:37
beach
Another interesting thing is the, apparently widespread, idea that a Common Lisp system must be written in some (presumably lower level) language. I know ECL and Clasp were made that way for reasons of interoperability, but I hear this argument from relative newbies who want to write their on Lisp implementation.
11:44:12
nij-
Their initial image can be written in whatever low-level language (e.g. C), so that if they want to support other things, they can change it.
11:44:21
beach
I doubt it has those kinds of advantages. I mean, nothing prevents SBCL from developing that way, and then just do a final build from source.
11:44:31
nij-
They save the, say C, source and version control. The rest, they can all do with lisp.
11:47:25
nij-
Building from a lower-level lang certainly feels easier cause that's closer to the end result.
11:47:55
nij-
And while CL is very powerful, any naive way would produce an image of things you want and all the extra CL that you don't want, which are hard to tree-shake.
11:48:15
beach
But as I have explained, it is much harder because you then need to write additional modules in "pidgin" Lisp.
11:48:18
nij-
The actual difficulty of your approach seems to be figuring out a way to "remove" the unwanted part.
11:49:05
nij-
Right. I'm still trying to understand why that's "harder". I guess I need to have hands-on experience to feel that.
11:49:17
beach
I think you are still imagining that SBCL and SICL are built as additions to a host Common Lisp system. They are not.
11:51:00
nij-
So you still need to produce the end result, but you need to write it in a lang that is relatively unnative comparing to the target (e.g. x86 or asm).
11:51:53
nij-
But it gains something back (e.g. we can have independent cl libs written and maintained in CL.)
11:51:56
beach
You still have to write the compiler to produce machine code, unless go generate C the way ECL does.
11:57:13
nij-
bike - If having an implementation that gets to LLVM and C++ is the main purpose of clasp, why don't just write an LLVM backend for sbcl to start with?
11:58:13
nij-
I asked the question for ECL the other day, and the answer is that ECL is aiming to be small and embeddable, while sbcl focuses on speed. These goals are fundamental tradeoffs and cannot really be compromised.
11:59:10
beach
nij-: As I recall, the problem is that LLVM has only a C++ interface that can be used reliably. So the Common Lisp system must already be able to execute C++ code.
12:04:52
nij-
There's a C interface. And also, I guess generating LLVM code directly is easier than generating x86 code...
12:05:37
nij-
Seems like you can GC in llvm. How flexible is it? I dunno.. https://llvm.org/docs/GarbageCollection.html
12:05:54
beach
Yes, that's why I want a backend library, so that code generation can be centralized in one place.
12:07:33
beach
I mean a library that takes a control graph representing a program, and produces machine code.
12:08:12
beach
On the same level as LLVM, but without all the stuff needed by C++ and the like, and with the stuff needed by Common Lisp.
12:09:50
beach
Input would probably be closer to MIR than to HIR or BIR, because MIR does not require knowledge about object representation.
12:12:13
beach
No, memory management must be decided by the client. But there might need to be some configuration for things to help the memory manager in some situations.
12:13:11
beach
But the library would choose its main representation, like for instance "sea of nodes" and it would implement all the relevant optimizations.
12:13:57
nij-
I'm still curious what bike would say.. and it's getting long so forgive my posting it again..
12:14:06
nij-
bike - If having an implementation that gets to LLVM and C++ is the main purpose of clasp, why don't just write an LLVM backend for sbcl to start with?
12:15:43
bike
just using llvm for a compiler doesn't help with C++ integration. clasp can actually call C++ functions and use C++ objects in a way that SBCL can't. A lot of that isn't actually closely tied to LLVM.
12:18:37
bike
the garbage collection page you already linked explains LLVM's GC handling better than i could
12:20:49
bike
"backend" isn't really the right term, it's more a bridging system. generating C++ code would also not in itself help you use C++ code.
12:22:03
bike
generating C code doesn't in itself help with FFI either. SBCL and so on have FFI just fine without a compiler that generates C code. what you need are lisp objects representing things like pointers, and then some Lisp functions and compiler hooks that let lisp code call C code and vice versa.
12:22:32
bike
clasp's C++ integration is like that but more complicated, like it has things to make C++ objects with all their fields and virtual functions available in lisp.
12:24:15
nij-
My main question is.. why couldn't we start from a developed lisp. What was the main bottleneck? - But I haven't thought it through.. so I need to think a bit more.
12:25:01
nij-
().. In a way that can be an extension of ECL, which can be turned on and off easily.)
12:39:18
bike
ECL does not and probably never will have C++ integration as a design goal, and similarly clasp doesn't and won't need to output C
12:39:50
bike
but i can't tell you exactly what was going through drmeister's head when he forked it. i wasn't there, except that i told him on IRC that it would take years to get a new lisp implementation working, and that prediction has definitely panned out
12:41:32
yitzi
I'm not sure I would think of it as a fork now. It is so different in build system, snapshots, scrapper, etc. They merely share some C and lisp code.
12:43:40
yitzi
MKCL is a fork of ECL, but it really hasn't diverged significantly. It still has the same build system, overall design, etc.
12:48:39
bike
part of the forking has been us making stuff more portable, though. cleavir 2 and maclina both came out of my clasp work, and yitzi's gray streams extensions and such also
12:48:52
bike
hopefully we'll be able to use khazern and inravina and such in clasp someday soon ish
12:49:46
yitzi
Inravina works, it is just slow. I've managed to speed it up by about 40% in the last couple of months.
12:50:51
yitzi
I suspect Khazern will be more work? I mean it depends on CLOS and right now we load LOOP before CLOS.
12:52:43
bike
yeah, i figure i'll have to swing the new build procedure i've been slowly preparing for
12:57:59
bike
related to the earlier talk, while clasp will probably always need a "base image" written in C++, that doesn't necessarily mean clasp has to be built by bootstrapping up from that primitive start
12:58:18
bike
i certainly hope it doesn't, since that's what we do now and it's caused me no end of grief
13:02:32
yitzi
beach: That would probably work. Its currently ECL's loop which I believe is MIT LOOP with a few tweaks. It has the usual flaws we fixed in Khazern.
13:06:21
yitzi
Although, we might be able to load MIT LOOP during the initial staging step (which is just LOAD to create functioning memory kernel for parallel compilation) and then skip it in the compilation stage of the bootstrap image.
13:07:46
beach
Those ECL tweaks might be unnecessary for the build procedure, so that you can use any old MIT LOOP.
13:11:59
beach
nij-: But, I have often suggested intermediate solutions. Take READ for instance. A simple reader can be written in C or C++ that is capable of reading only very common-looking code. Then that reader can be used to read Eclector.
13:12:01
yitzi
Including some of them is trivial since when they are normally loaded in the bootstrap CLOS has already been loaded, i.e. the pretty printer and format aren't hard to integrate.
13:13:25
beach
nij-: But unless it has changed since, the Clasp reader is a C++ program that can call reader macros written in Common Lisp. So it must be a very complicated thing indeed.
13:14:50
nij-
Right. In order to make lib loads and works right intrinsically, there must be some operators that have been working before loading.
13:16:07
beach
nij-: Unless you do it like SICL does it, yes. That is, rely on the host version of those operators.
13:26:52
beach
I thought you might be interested in working on BOCL, but that's probably out of the question.
13:27:00
nij-
I don't know pragma, for example. I don't fully undersatnd all the scoping keywords either.
13:28:07
beach
It's partly a joke, but partly a serious thing. Some operating system distributions allow only code that can be compiled from source using just a C compiler.
13:28:32
beach
So to build something like SICL, you would then first build BOCL using C and then SICL using BOCL.
13:31:06
beach
But creating BOCL would teach a lot about bootstrapping Common Lisp, and why it is generally not a great idea to write a Common Lisp implementation in a lower-level language.
13:32:03
beach
And, it could be fun to just ignore all the hints about how to make the code run faster, like how immediate objects are often represented for reasons of performance.
13:39:28
nij-
Do you have a plan how BOCL can be as minimal as it needs to become a full CL implementation by "loading" the libraries?
13:41:02
beach
Sure, as much as possible. But it can't be built like SICL is, so it would be more the idea of starting with a primitive C version of a library and then loading a Common Lisp version later.
13:44:52
nij-
"As an exception to the goal of simplicity, BOCL will contain a full implementation of the CLOS meta-object protocol (MOP)"
13:46:33
bike
what i want to do is: 1) build the C++ core with clang 2) fire up some lisp, load maclina+extrinsicl+etc, and use it to compile fasls for clasp's lisp code 3) load those fasls into the C++ core
13:46:35
beach
But all of CLOS can be loaded as a library, and it is in most Common Lisp implementations.
13:47:21
beach
nij-: PCL (Portable Common Loops) was designed with this in mind. It loads the MOP in order to create CLOS.
13:48:24
nij-
beach So could we remove this from BOCL? "As an exception to the goal of simplicity, BOCL will contain a full implementation of the CLOS meta-object protocol (MOP)"
13:48:43
bike
it would be pretty ridiculous to load clos some other way, and then load mop as another way of doing it
13:49:34
beach
nij-: If BOCL is to be used to build SICL, and SICL needs the MOP to build, it would be silly to create BOCL without the MOP.
13:50:51
beach
nij-: Like I said, you could build BOCL as a pre-ANSI Common Lisp implementation and then load PCL to create both the MOP and CLOS.
13:51:16
nij-
Right, so can we remove this line? "As an exception to the goal of simplicity, BOCL will contain a full implementation of the CLOS meta-object protocol (MOP)"
13:51:17
beach
nij-: Like I said, you could build BOCL as a pre-ANSI Common Lisp implementation and then load PCL to create both the MOP and CLOS.
13:52:17
beach
nij-: BOCL is a complete Common Lisp implementation including the MOP. How it is built is a different story.
13:53:02
nij-
Then I read that sentence wrongly. I thought that sentence implied that MOP is so fundamental that needs to be built in C.
13:53:41
beach
But that's exactly the challenge. How do you determine that minimal subset of Common Lisp.
13:55:31
nij-
yitzi Yes but no thanks. If I want to challenge myself I'd rather implement MOP in brainf*** :D (Half joke.)
13:56:26
nij-
beach So apart from the data types and a basic compiler.. are there more to be written in C?
13:56:35
beach
nij-: I would represent every BOCL object as a standard object, and define the representation of standard objects in C.
14:00:28
beach
nij-: Oh, and for arithmetic, BOCL can use the GNU bignum library. I forget the name.
14:01:00
nij-
(Could you update readme when you have time..? To mention that lib and mentioned that GC is not necessary.)
14:02:53
beach
random-nick: Because those implementations were written for good performance, so they have a much higher maintenance burden that BOCL would have.
14:04:15
beach
random-nick: My (not so) secret (and of course impossible) plan is to eliminate all C code from all Common Lisp implementations, except for BOCL. :)
14:04:36
yitzi
ECL is currently filling this need for platforms like Homebrew. I am sure some still use CLISP in this way, but is not really maintained.
14:06:08
beach
Indeed, but it is more like a feasibility study. I am a researcher, so I want to know what is possible.
14:06:51
beach
random-nick: Not even that. I don't actually care about those silly operating systems and their limitations. I just want to see what is doable.
14:07:17
nij-
will we automatically have a fully compliant CL implementation *sans* compiler related parts?
14:07:28
beach
I think it is fun to think about a Common Lisp implementation that was not written for good performance, and to see the implications on maintainability of the choices made.
14:08:36
beach
nij-: Oh, it would definitely not compiler to native code. An interpreter would most likely be the good choice. And it can be a minimal interpreter, just good enough to evaluate a Common Lisp version of the evaluator.
14:10:56
beach
In fact, I might have rewritten it completely at some point. I'll update the site at some point.
14:11:43
nij-
Certainly not the whole Cl spec E.g. It won't do anything useful when evaluating (compile-file ..)
14:12:01
beach
There are an infinite number of ways you can strip down Common Lisp to something minimal.
14:12:57
random-nick
the interpreter itself only needs to interpret a handful of special forms, other things can be installed as builtin functions into the environment
14:15:50
beach
yitzi: I don't know. Like I said, I don't even care much about those distributions. So at this point, it is more like a mental challenge.
14:15:57
bike
C can also target pretty much any machine designed in the last fifty years, for when you want to run common lisp on your dishwasher.
14:52:52
nij-
beach (1) Is there a full implementation of PCL already? (2) In order to know what's the minimal requirement of BOCL, shouldn't we wait for sicl be done first, as whatever requirement would depend on what SICL would need?
14:55:17
beach
No, BOCL is not meant just for SICL. It is supposed to be a complete Common Lisp implementation plus the MOP.
14:55:53
beach
I am sure there is a complete implementation of PCL because that's what is used by most Common Lisp implementations.
15:03:54
bike
it does a little more than minimal compilation, but clasp has the maclina compiler written in C++, that can handle full lisp but doesn't do anything native
15:05:58
bike
a really minimal compile file would be, what, macroexpand-all plus load time value handling? so you'd need some kind of compile-file format
15:06:37
nij-
Hmm I remember reading it in the spec that a conforming lisp must eval through compilation.
15:07:46
beach
nij-: And even if it did, it says nothing about the "machine code" format, so it could be some very limited transformation of Common Lisp code.
15:07:48
bike
Sure, I mean what I'm thinking of is more the definition of the loader. It... might be possible to have "FASL"s that are just straightforward sources?
15:09:31
bike
Yeah, I mean it's the load time values I'm thinking of. I think you might have to, say, replace (load-time-value form) with *some-variable*, and then have (defvar *some-variable* form) earlier in the file
15:10:06
bike
since if you just LOAD a source file the load-time-value forms will have the eval semantics, so they'd be evaluated every time you evaluate the form, which is inappropriate for compile-file
15:12:51
beach
There would also have to be code to reconstruct objects that can't be created by READ.
15:16:50
nij-
"interpreted implementation n. an implementation that uses an execution strategy for interpreted functions ..."
15:17:06
nij-
Does it mean instead "an implementation that uses interpreted functions as an execution strategy"?
15:17:41
bike
"an implementation that uses an execution strategy for interpreted functions that does not involve a one-time semantic analysis pre-pass"
15:19:06
bike
a lisp implementation certainly has to have a compiler, but just a minimal compiler, so it doesn't have to mess around with native code or even bytecode
15:22:11
nij-
Oh, minimal compilation doesn't have to transform code unless there are compiler macros or macros.
15:22:59
nij-
beach Interesting, so BOCL + all the libs would result in an interpret-based conforming CL, and in particular it must pass the ansi test suite by then.
15:23:14
bike
it would be pretty weird for a language spec to specify what to compile into with any specificity
15:24:05
beach
nij-: As it turns out, I started a new BOCL from scratch a long time ago. I'll make the new one available after cleanup.
15:33:45
beach
And, since we don't know what BOCL_min contains, it is hard to know whether the total system will be conforming.
15:35:27
beach
nij-: Khazern requires CLOS, and CLOS requires LOOP. Eclector requires CLOS, and to read the CLOS code, you need a reader.
15:36:07
beach
We have Common Boot that contains an evaluator, but you need an evaluator to evaluate the code in Common Boot.
15:39:42
nij-
So at least.. we need a minimal reader that can read CLOS code, and support that can load LOOP correectly.
15:39:46
beach
Like I often repeat, my small brain couldn't cope with a "pidgin" subset of Common Lisp for the libraries, so they are often written to use the full language.
15:43:51
nij-
Seems like the first milestone for BOCL is to have a working LOOP. What do you think about this goal?
15:44:24
nij-
To have a working LOOP it must already support enough data types, an IO, a minimal reader.
15:51:05
beach
About the new BOCL, it could be crap since I haven't written serious C for several decades.
18:47:09
nij-
Is there any implementation that's "simpler" than CLISP? I say it's simpler because it only compiles to its own bytecode machine.
18:57:28
bike
interpreters are pretty dumb. if you're going to go through all the effort to get a lisp implementation you probably don't want to leave it dumb