freenode/#clim - IRC Chatlog
Search
3:29:01
nyef
Theory: Common Lisp and CLIM are mid-to-late-1990s era "export versions" of artifacts from a software design philosophy that dates back to the 1960s... But that philosophy may have been killed off in the late 1990s.
3:35:19
beach
The answer to my question might determine whether it is worth our while to revive that philosophy, or drop it for some philosophy that killed it off.
3:36:14
nyef
That's an interesting implicit dichotomy, and it ignores the possibility of synthesizing a new philosophy that partakes of both the old and the new.
3:37:37
beach
True. But, given our available manpower, often the best we can hope for is to copy the best existing designs we can get our hands on.
3:38:43
beach
I do try to come up with better things, but it is hard work, and there is no guarantee that it will actually BE better.
3:38:50
nyef
But at the same time, a philosophy shows *which* designs to steal, and which attempt to improve upon.
3:40:19
nyef
I believe that we have some fundamental difference in underlying philosophy, somewhere, but that the artifacts that we produce may tend to be compatible in practice.
3:43:31
nyef
Neither have I, really. The only way in which I see them as showstoppers is that I don't expect that our *overall systems* to be considered usable by the other.
3:43:38
beach
But I can't do research by accepting that what we have is already the best. I have to explore extreme alternatives. That's what research is about.
3:44:28
beach
Luckily, there are many philosophy-independent byproducts that can be independently useful.
3:46:47
nyef
I sortof came to the conclusion this evening that if I were to sink about three months of full-time work into producing a LispOS, I'd end up with a very good start.
3:47:38
nyef
At the same time, I don't think that it'd be something that I'd want to use on a day-to-day basis.
3:48:08
beach
I totally agree about the three months. You are very knowledgeable, and it is not as hard as some people might think.
3:48:36
beach
And, I don't know for sure that it would be something I myself would want to use. But I don't know that yet.
3:50:58
beach
Anyway, people tend to be awed by "bare metal" Common Lisp implementations, but, seriously, that aspect is neither very hard nor very important.
3:51:27
nyef
Mmm. We have at least two existence-proofs of x86oid "bare metal" CL implementations now.
3:52:10
nyef
The philosophy that I see behind Common Lisp and CLIM is not yet well-defined as far as I'm concerned, but includes some aspect of trust in the user-as-programmer, or possibly programmer-as-user.
3:54:00
beach
However, these days, there is not only the user/programmer, but an OS has to have some protection against malware.
3:55:37
beach
That's a different aspect, yes. I am trying to push shared-memory, multi-core as much as possible, because then I can keep the Lisp philosophy without going into a territory that I have no idea how to work with.
3:56:41
nyef
Yeah, that's the thing. I don't believe in shared-memory multi-core. It *breaks* once you get past a certain point, and that point isn't very far off single-core.
3:57:19
nyef
The further you can push it, the better, absolutely. But hitting double digits is *hard*.
4:00:35
beach
What I am saying is that very few people need massive parallelism, and even those who do don't need it most of the time. So there is still room for shared-memory multi-core software.
4:05:39
nyef
I don't disagree about there being *room* for shared-memory multi-core software. But I'm being absolutely hammered at work with something that is, fundamentally, a stream-processing system that only needs parallelism in order to increase throughput. None of the processes needs to talk to another directly, but they get hammered on the GC, or on other "process limits".
4:07:21
nyef
One comment I saw was that bouncing a "border router" (the gateway from your lan to the internet at large) can cause a noticable lag on the order of seconds on a modern machine, even for applications that don't talk to the internet in the first place.
4:09:20
beach
I recently read a very interesting article in CACM about non-volatile memory technology that is significantly faster than disk. Such technology exists, but the performance is completely wasted because of the slowness of the drivers (created for slow disks) used to access that memory.
4:10:37
beach
And I firmly believe that there are significant performance problems with context switches, buffer copying, etc.
4:12:08
nyef
beach: Context switches amount to dumping your L0 cache, at the very least. Buffer copies are exactly that, burning memory accesses.
4:16:58
beach
It depends on the kind of OS you want to write. For the one I am planning, there is no need for a file system.
4:18:55
nyef
If you're implementing Common Lisp, there's need for a file system, even if it's "just" a backing store against the whole streams-and-filenames thing.
4:21:13
nyef
At the same time, I Do Not consider Filenames, CLOS, the compiler, and whatnot to necessarily be part of Lisp.
4:24:21
nyef
I'm not sure where to draw the line when it comes to some Common Lisp concepts, though.
4:25:05
nyef
FUNCALL, FUNCTION, VALUES, MULTIPLE-VALUE-BIND, MULTIPLE-VALUE-CALL, and so on are clearly language.
4:26:13
nyef
In fact, ED is an egregious example of why I consider CL to be an "export version" of an artifact of a design philosophy.
4:26:59
nyef
Because there's really no justification for it other than as part of an integrated environment, which isn't otherwise called for in the CL spec.
4:31:00
beach
One thing I would like to see is a driver architecture in the form of CLOS protocols, inspired by I/O Kit.
4:31:22
beach
Except that we can do better, because I/O Kit is restricted by the fact that it is written in C++.
4:32:57
nyef
Mmm. Actually, we can probably do a decent job in terms of "driver architecture in the form of CLOS protocols" alone, nevermind I/O Kit.
4:34:17
nyef
The main thing for a "driver architecture" is finding the devices in the first place and matching them to drivers.
4:34:57
beach
We need a collection of classes (protocol or instantiable) and a collection of generic functions.
4:35:23
nyef
From there, we find the resources associated with the devices and make them accessible to the drivers. And that amounts to the "bus" concept in Linux.
4:36:42
nyef
Everything above that is based on families of devices ("this is block storage", "this is an event input stream", "this is data packet I/O", "this is memory with a particular organization representing a graphical bitmap", and so on).
4:38:09
beach
I think the idea behind I/O Kit is that object-oriented technology makes it easier to factor the code and to identify commonalities between different devices.
4:45:48
nyef
Hypothesis: The two things that kill any real interest in a "bare metal" LispOS are lack of useful applications and lack of supported hardware.
4:46:52
nyef
Nobody will use a "bare metal" LispOS on bare metal (except as a curiousity) unless it supports applications that cover all of their main use cases AND it works on easily obtainable hardware.
4:49:02
beach
... just like people use a virtual machine for a few Windows applications while still using GNU/Linux for most stuff.
4:49:50
nyef
Heh. I actually have a physical Windows box... for a few device drivers, and use Linux for most stuff.
4:50:24
nyef
(Okay, *two* physical windows boxes. I have multiple device drivers that need figuring out.)
4:51:49
beach
Because in the beginning, you would not have the tools for debugging. At least that's what I imagine.
4:53:20
nyef
The worst-case scenario is that you have the ability to get a single bit of information from a build cycle: "did it get here?"
4:54:06
nyef
But once you're beyond that point, you typically have at least FORMAT debugging, if not an interactive prompt.
4:55:17
nyef
And I say this having bootstrapped more than one x86 protected-mode kernel, and a full Linux port to an unsupported MIPS system.
5:03:44
nyef
Another thing is that a "device driver" that conforms to a specific set of interfaces (say, an USB bus structure plus an "input device" surface) essentially doesn't *care* what sort of address space structure it resides within.
5:05:27
beach
That's what I meant by "Furthermore, it would be useful no matter what LispOS design it is used in".
5:06:12
beach
So, if you were to write such a thing for an OS with multiple address spaces, I could still use it in my OS.
5:07:00
nyef
Were I to write such a thing, I'd probably put address space structure independence as an explicit design goal. (-:
5:09:13
nyef
I intended it somewhat orthogonally: "object-oriented technology" as represented by C++ doesn't particularly help with device driver design over older techniques such as "abstract data types".
5:10:54
beach
They do report success (of course). But maybe they didn't compare it to the best possible non-OO design.
5:11:05
nyef
(To be fair, the II, the SE/30, the IIfx, and the Quadra 800 are awesome, and are post-Jobs machines, but they were from back when Apple had an engineering culture.)
5:13:14
nyef
Yes. It's an object pointer where there's a certain presumption of "stream-ness", and then an escape hatch (IOCTL) for everything else.
5:14:59
beach
When I imagine my shared-address-space LispOS, I also imagine that some Lisp functions might actually run in a separate address space (to accommodate code written in other languages), and then, there would be a concept similar to that of a file descriptor to access OS-wide objects in a safe way and in a different address space.
5:17:28
beach
IOCTL is an interesting example of the kind of things you need to do when all you have is a static programming language, but you need to accomplish more dynamic things. IOCTL is also a fantastic source of inexplicable program behavior.
5:21:01
nyef
You still end up with "families" of IOCTLs. There's a set that applies to all serial ports, for example. Or the set that applies to all framebuffer devices.
5:21:18
beach
And, of course, the lack of a decent condition system means that, in order for your code to be good, a large portion of it is dedicated to handling error return values.
5:23:41
nyef
There's also the Catch-22, as it were, where someone is unlikely to write a web browser for a LispOS that doesn't exist, and someone is unlikely to write a LispOS unless there's going to be a web browser for it.
5:26:37
beach
I'll say this again, then: I do not take considerations like that into account, simply because my primary goal is not for it to be used by as many people as possible; maybe not even by me. I am mainly interested in either proving that such a thing is possible and would be great if used, or finding problems with my imagined design.
5:27:54
nyef
Part of what I see as the "Common Lisp philosophy" is a certain hard-headed pragmaticism.
5:29:44
nyef
But it's pragmatics focused on a *system*, only part of which was externalized into the Common Lisp standard.
5:30:40
nyef
My criteria for adopting software is largely orthogonal to "good", as is basically everyone else's.
5:31:21
nyef
Alternately, "good" can only be measured in context and there is no "absolute or abstract good".
5:32:56
nyef
So I try to write software that either I expect to use myself, I expect "my people" to use, or that will teach me something that will serve either myself or "my people".
5:34:10
beach
For my LispOS, I am in fact not particularly interested in the end user. Unix has shown that, with enough effort, you can create useful applications running on a crappy OS. I am instead interested in the amount of work for the application programmer to produce high-quality and safe applications.
5:34:45
nyef
With my original "SBCL-os", I actually proved my initial hypothesis correct, and then ran into trouble turning it into a useful system.
5:36:50
nyef
And I do not account myself purely a "researcher". I demand a certain degree of relevance.
5:39:45
beach
I find I hate having words put in my mouth, like "if you really want people to use this, then...". Yes, but I have no such desire.
5:40:55
nyef
Agreed. And, while I *do* tend to want people to use my things, I find that "internet randos" have a poor sense of what's actually required in order for that to happen.
5:45:02
beach
It looks like I need to reboot my computer in the next month or so. It is getting slow, and there are some Firefox complaints that I need to override in some situations.
5:46:49
nyef
I'm currently attributing a good deal of my current system quirkiness to a long-running Firefox process.
5:48:23
nyef
But as far as general system stability goes, the only thing that's really-really messy is the sound card driver, which doesn't always handle a suspend/resume or hibernate/resume cycle all that well, and can take a couple of cycles to sort itself out when it goes wrong.
5:55:28
nyef
Here's another thing about Common Lisp: It was mostly developed before wide-scale software distribution was a thing, thus there's no mechanisms or policies defined for long-term maintenance of software across multiple versions, such as dealing with the evolution of interfaces and forwards and backwards compatibility.
5:56:52
nyef
The Linux kernel basically commits to maintaining any user-space ABI with a full decade of a deprecation period.
5:57:30
nyef
When was the last time you remember using a Common Lisp implementation with FASL compatibility from one minor version to the next?
5:59:14
nyef
It's an acknowledgement that we have no real tools to enforce compatibility for FASLs, and it's a thorough pain in the ass to maintain without such tools.
6:00:31
nyef
At one point I started working out what "backwards compatibility" meant for Common Lisp code. What changes did and did not require rebuilding "dependent" FASLs.
6:01:33
nyef
On the one side, there's a lot of stuff that you can change without breaking anything already built. On the other side, there's a lot of stuff that basically requires a rebuild if you change it.
6:02:56
nyef
Change the definition of a NOTINLINE function without changing its type signature? Fine. An INLINE function? No. A macro? Well, it depends on the changes, and the client code.
6:10:08
nyef
Except that you might be able to change the definition of an INLINE function anyway, depending on the change that you're looking to make.
6:13:00
nyef
I think that much of the "low hanging fruit" (to use a rather tired metaphor) when it comes to Lisp systems is in terms of development tools, rather than the Lisp implementation itself.
6:13:38
nyef
But that the implementation MUST support these tools, rather than being treated as something to abstract over. Build the tools to the implementation, and the implementation to the tools.
6:17:33
nyef
And the Common Lisp language itself is somewhat antagonistic to certain kinds of (static or dynamic) analysis.
6:23:12
beach
Sorry, I am off and on. My favorite coauthor is coming for lunch today, so I am preparing food, etc.
6:24:18
beach
I totally agree about the development tools. And I totally agree that the implementation needs to support those tools.
6:25:33
jackdaniel
providing uniform interface for different implementations of such tools (which root in implementation-specific code) wouldn't hurt
6:29:04
jackdaniel
it may be, but the alternative is fragmenting ecosystem even more (plus - sacrifacing portability)
6:29:31
beach
I imagine Clordane would require profound changes to all implementations that want to support it.
6:29:45
nyef
The "binary file descriptor" library is a compatibility layer over various binary file formats for use by GNU binutils. The name is a backronym, as the original author, when presented with the high-level problem description, said "BFD".
6:32:38
jackdaniel
I get that portability doesn't have to be a priority for everyone, but fragmentation = smaller userbase = less contributors & more duplicated effort(s)
6:33:48
jackdaniel
while I get, that for some things it makes sense to do whatever is more convenient, providing superficially identical syntax for different mechanisms doing declaratively same job makes perfect sense to me
6:34:21
nyef
Less applies when there is a quantity of a single thing, fewer when there are multiple discrete things.
6:34:30
beach
jackdaniel: I think you must have a specific case in mind, but I can't figure out which one it is.
6:37:03
beach
So, suppose some implementation wants to support Clordane. One important aspect of Clordane is that it uses one thread to debug another thread, and things like traces, breakpoints, etc., are only visible in the thread running Clordane. So an implementation that implements tracing by modifying the function, or by wrapping the function in tracing code, would have to be completely altered.
6:38:46
beach
Obviously, when designing SICL, I have Clordane in mind, so the SICL design will support such tools. But I can't imagine trying to convince (say) the SBCL maintainers to make such profound changes.
6:40:00
nyef
I'm having a hard enough time, as an SBCL maintainer, figuring out how to make breakpoints work again in the first place. Once we have breakpoints that work at all, making a thread-affinity wrapper becomes a lot easier.
6:40:07
jackdaniel
specifing protocol which implementation must fulfill sound OK (so programmer may use Clodrane uniofmly on all implementations, which implement this protocol) - that's what I mean by superficially identical syntax
6:40:34
jackdaniel
so instead of having itch-driven-development, where we implement functionality on top of whatever we have, I prefere design-driven development
6:41:07
jackdaniel
where we carefully think about the interface, which covers all programmer needs and don't tie system vendor hands on how he needs to implement it