freenode/#lisp - IRC Chatlog
Search
21:28:35
fiddlerwoaroof
Yeah, it's a jane street project and so it's probably related to an ocaml codebase
21:43:47
no-defun-allowed
Well, at the moment beach is finalising a paper for ELS and I think there's some discussion about improving the compiler.
21:44:29
no-defun-allowed
You can test it, it only takes about two minutes to compile with low debug settings.
21:48:31
no-defun-allowed
Then you'd need to wire in the compiler and make a code generator too. I don't know how SICL manages that, or if the test environment/REPL goes through Cleavir even.
21:53:55
no-defun-allowed
However, that just goes to a backend which generates more Common Lisp to run on the host from memory.
21:54:30
francogrex
I was surprised to learn that closure cl always seeded itself from a previous image. few are lisp implementations that can be build from the ground up with just an assembler?
21:55:54
verisimilitude
That's how I want to do it. For one, you avoid the malicious compiler attack.
21:56:57
francogrex
computers work with assembly instructions and it is more transparent and clear to know what is going on
21:57:21
verisimilitude
With how I'd do it, there'd be no building at all; you'd have the machine code and that loads the rest of the Common Lisp package and there you have it.
21:57:47
verisimilitude
I wouldn't be harmed by a malicious assembler, because I'm writing my own machine code development tools.
21:58:09
no-defun-allowed
It's less transparent, there's much more to read and assembler requires a lot of state to remember.
21:58:42
verisimilitude
All you need to do is implement the very base everything else builds from, no-defun-allowed; you wouldn't even need the base to include the GC.
22:01:29
francogrex
but yes i expect that wiould be only the very base, everything else in lisp, sure
22:01:40
verisimilitude
Yes; I like programming without any complex building stages; it's disgusting to see software that wants over an hour to compile on a machine with well over a gigahertz of speed and gigabytes of working memory.
22:02:30
verisimilitude
If you want me to go in a tad more detail, I'll tell you about the general scheme I've thought up for this implementation I want to eventually find myself writing, francogrex.
22:02:49
francogrex
yes that being said, writing and loading machine code is not the greatest fun either
22:04:56
verisimilitude
Well, I have an article concerning the machine code development tool, I've been working on, if you'd want to see it.
22:06:26
verisimilitude
Now, how I'd like to do it is have a simple machine code program that implements READ and a few other necessary CL functions. You could have an EXT package that exposes everything that needs to be there, but isn't in CL.
22:07:20
verisimilitude
As an example, you could expose the data structures used for GC in this EXT package and after loading the functions that will manipulate it, UNINTERN this symbol. Then, it can't be accessed further and such a reference won't ever be created again.
22:07:58
pjb
francogrex: Toaster from assembler (from scratch): https://www.youtube.com/watch?v=R3Qn98bE880 Toaster from sophisticated tools: https://www.youtube.com/watch?v=crc8n7D4kYg
22:08:16
verisimilitude
You'd need a loading program that's what's actually called. This could also contain the version and help information. This would be written in, say, sh.
22:09:14
verisimilitude
Closing, all the machine code would do is READ and one of the first things that the loading program would do is feed in the rest of the CL package before locking it. Then you have a working implementation. One thing this would permit is customizing how most of the CL package is compiled, if that's desired.
22:10:03
pjb
verisimilitude: if you're building your assembler using stuff you didn't build yourself, you're fucked. See Reflection on Trusting Trust. https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf
22:10:47
verisimilitude
Clearly, I'm eventually going to need to use an iteration of the tool to write a version of itself in machine code and then manually inspect that.
22:14:03
verisimilitude
I eventually want a 6502 version of this tool I can run on a C64, which is trustworthy enough. But that's then tangential.
22:14:18
verisimilitude
I'm glad you found it interesting, francogrex; did you refer to what I wrote here or the article I linked to?
22:14:47
francogrex
ok guys. I will save this to read it again later. verisimilitude yes i saved for reading it. thanks
22:15:34
verisimilitude
It's no issue. So, you haven't read the article yet; in that case, I anticipate your thoughts concerning that.
22:17:18
francogrex
not yet. I will and I come back here hoping to discuss it with you. I will try to understand :)
22:30:13
fiddlerwoaroof
as it turns out, most lisps do use assembler, they just don't use an external assembler...
22:31:02
fiddlerwoaroof
all the vops in sbcl, for example, bottom out in an sexp-syntax assembly language
22:42:25
pjb
verisimilitude: That said, while as an individual you might not want to invest the resources to make your own toaster, as a country we should consider making our own processors and other components. The case Huawei shows that's important. (and who would trust Intel apart from the US government and companies?)
4:09:49
beach
Writing a Common Lisp system in assembler? It would have been funny if it were a joke, but I fear it wasn't.
4:11:16
no-defun-allowed
Some people wonder why there are no (large) Common Lisp systems written in C, or if they are educated enough, why ECL is the only Common Lisp in C.
4:12:08
no-defun-allowed
I think people expect other languages to be bootstrappable from a C compiler, but most languages are implemented in themselves once they're developed enough.
4:14:52
diip
I am quite the opposite, I wonder why most CL implementations must have C in them and don't seem to be able to interact with the hardware directly, I know of clozure and SBCL, but even clozure has some C in it.
4:15:21
beach
The other is a conviction that higher level languages must somehow be written in lower-level languages.
4:15:31
no-defun-allowed
So, I suppose I will tell them to compile ECL or CLISP and use that to bootstrap SBCL.
4:16:33
no-defun-allowed
I imagine in some places of SBCL and other implementations, it would be terrible for the programmer to accidentally cons or do any mutations to the Lisp image's state for some reason.
4:17:41
no-defun-allowed
I could also imagine an internal special form or declaration that refuses to compile any consing code. It'd be like assembler with nested parens.
4:19:37
rme
been hacking CL for RavenPack since july and working on Opusmodus and CCL in my copious spare time
4:20:09
diip
beach: I meant to do anything hardware related, you seem to have to go through the CFFI and generally end up using C rather than straight to the hardware itself (or assembly)
4:21:19
djeis[m]
I mean, strictly speaking the CFFI is no more CL than the hardware access something like Genera or Mezzano provides.
4:23:00
diip
I am still very much learning though but there is little to no documentation on hardware interaction with CL, it seems implementation specific and generally not advertised by the implementations themselves
4:24:01
beach
diip: It not only SEEMS implementation specific. It IS, because there is nothing like that defined in the standard, just like with most other languages.
4:24:10
djeis[m]
diip: It is entirely implementation specific, CFFI just abstracts over the differences in many implementations API's for C.
4:25:08
no-defun-allowed
When it's provided, I think it'd be hidden quite well or regulated so the Lisp image doesn't destroy itself.
4:26:14
diip
beach: nothing like "hardware interaction" defined in other languages. some languages (C inline ) do
4:26:23
rme
RavenPack would like it if I moved to Spain, but I don't know Spanish and the kids are too old now to just go to Spanish public schools and pick up the language by immersion.
4:27:09
diip
no-defun-allowed: I think this image part is where I get lost, how would using assembly affect the lisp image?
4:27:56
djeis[m]
It's more that direct hardware access gives you the tools to royally shoot yourself in the foot.
4:27:59
no-defun-allowed
Assembly has very little protection, and can be used to harm a Lisp system.
4:28:33
no-defun-allowed
diip: One simple experiment is to write a function with (declaim (optimize (speed 3) (safety 0))) and some type declarations, and then funcall it with different types.
4:29:02
djeis[m]
I think it's just that there aren't a lot of people that want to do that sort of thing, so the interface isn't well documented for the vast majority of implementations.
4:29:28
no-defun-allowed
You will quickly encounter strange values, and objects which the Lisp system doesn't understand. Often, the garbage objects will cause the garbage collector and printer to error.
4:29:32
djeis[m]
Telling SBCL to generate arbitrary ASM is actually relatively (relatively) straightforward when you've done it a couple times.
4:30:56
rme
in CCL, we implement certain functions in asm, e.g., https://github.com/Clozure/ccl/blob/master/level-0/X86/x86-pred.lisp#L33 but as you say, it is extremely easy to mess up.
4:31:02
djeis[m]
And the interface for it is only meant to be used by people with knowledge of the compiler internals, so the docs for it aren't written to be user-facing and are also relatively sparse.
4:34:00
diip
djeis[m]: I agree, but that assumes that the lisp compiler make use of all the optimisations of the chip (which for x86_64 I find hard to believe)
4:35:27
djeis[m]
Are you allowing for SSE? because SBCL can make use of those, although admittedly I haven't seen it generate them on its own.
4:35:32
fiddlerwoaroof
Part of the problem is that oftentimes the "below-C" level is an implementation detail
4:35:52
fiddlerwoaroof
e.g. on OSX, a lot of the stable interfaces are defined in terms of C or Objective-C
4:36:25
fiddlerwoaroof
If you work below that level, you have a lot more maintenance work to do every time Apple releases an update.
4:36:42
no-defun-allowed
diip: You'll need to write your own function which requires specific types. One good, simple function just adds two fixnums.
4:36:49
beach
I personally think that if your goal is to turn Common Lisp into C or assembler, you might as well use C or assembler in the first place.
4:37:17
fiddlerwoaroof
Well, I would generally like to write CL applications that "fit in" with the rest of the system.
4:38:47
djeis[m]
If you just want to hand optimize a particular function for use cases involving very specific types and you know how SBCL has laid those types out in memory, go for it and write an SBCL VOP and teach the compiler the hand written assembly for your problem. But that sort of thing is well beyond diminishing returns for a lot of use cases- the compiler is very clever, especially given the type annotations you'd need for that
4:39:47
fiddlerwoaroof
Also, from my fiddling with the objective-c runtime and the various APIs exposed on a Mac, my general impression is that the technologies underlying OSX are much nicer than the equivalent layer on Linux
4:39:55
diip
beach: don't like C and I do sometimes use assembler, it is just a bit tedious. but I am not trying to turn CL into anything else, I just wondered why.
4:40:14
fiddlerwoaroof
They're quite inspired by Smalltalk and so they feel more "lispy" when you integrate with them.
4:41:14
fiddlerwoaroof
diip: if you're on sbcl, you might also look at pvk.ca and/or figure out how to use sbcl's built-in assembler
4:41:49
beach
diip: I used properly, Common Lisp is a language that is much safer than assembler, C or C++. That is one of its major strengths to me. By "turning Common Lisp into C", I mean making Common Lisp as unsafe as C, which is taking away one of its main advantages.
4:43:40
diip
fiddlerwoaroof: I am thinking of that but like I mention the documentation is a bit evasive, I have only heard of rumours online which point to read the source code eventually
4:44:21
djeis[m]
There are a couple blogposts on that site which walk through the process of writing and interacting with VOPs.
4:45:26
djeis[m]
Past that yea, you basically have to read the source. There are some manuals for the CMUCL compiler and the architecture/terminology is pretty similar to SBCL, so those are helpful.
4:47:13
diip
beach: I would like to do some unsafe stuff from time to time even if just to find out why it is unsafe, I like assembler for that I just which I could experiment within the CL environment.
4:48:20
djeis[m]
Did you look at this one in particular: https://pvk.ca/Blog/2014/03/15/sbcl-the-ultimate-assembly-code-breadboard/ ?
4:50:23
djeis[m]
I'd suggest going back to it every now and then as you learn more of the higher level terminology and such, it actually covers a lot.
4:51:22
djeis[m]
There are SBCL-specific calls unrelated to ASM that you're kinda just expected to know about in order to follow along.
4:52:27
djeis[m]
I'm not honestly sure how similar they are at the line-by-line code level at this point.
4:56:38
djeis[m]
I'm happy to answer particular questions to the best of my own knowledge, and that's also a good way for us both to arrive at right answers as someone more knowledgeable jumps in to correct me.
5:06:05
no-defun-allowed
If all my threads are waiting on mutexes (with bt:make-lock mutexes) for some reason, is it likely that would appear as high CPU usage?
5:07:56
no-defun-allowed
My parallel program sometimes stops and I can't tell why. I think there are mutexes halting everything.
5:11:17
no-defun-allowed
Well, with these pauses there's little benefit to parallelism. It just appears to go very, very slowly after a while.
5:12:11
no-defun-allowed
Maybe it is for a while. The one lock I can think of that's important is usually free though.
5:13:14
no-defun-allowed
It could also be garbage collecting, actually, but the pauses are very, very long for a GC.
5:14:26
Jachy
For deadlock to occur you need to have mutual exclusion, lock & wait (once a thread has a resource, it won't let it go until it's acquired all the others and done its work), no preemption, and circular wait (T1 needs R1 and R2, has R1, waiting on R2. T2 also needs R1 and R2, has R2, waiting on R1.)
5:14:40
Jachy
If you can rule out any of those then you're ok as far as deadlock is concerned, though might have other problems.
5:15:33
no-defun-allowed
Yeah, I think there wouldn't be any circular waits since the threads will only ever wait on two independent resources, the statistics output and the database.
7:15:04
no-defun-allowed
So....I think the issue with my parallel program was that the cache was being overwritten by many differenct processes working on different parts of the computation, so I would need to make the cache larger to accomodate.