freenode/#lisp - IRC Chatlog
Search
10:25:00
lukego
Is anybody else using ql2nix (https://github.com/bradleyjensen/ql2nix) to snapshot quicklisp distributions with nix or is it just me? I'm noticing that some packages aren't happy living on read-only filesystems and trying to figure out a fix/workaround.
10:26:39
lukego
seems like it's mostly packages that want to do some code generation and store the results on disk. I'm hoping it will be enough to ASDF:LOAD-OP them on a read/write filesystem and then transport the files onto read-only later. Or will they want to always be able to read/write their source dirs?
10:27:36
Shinmera
For the packages I control they either do their caching on first load, or whenever the user requests it.
10:29:44
Shinmera
From the top of my head, those would be qt-libs, uax-9, uax-14. There might be more, I don't quite remember.
10:33:00
phoe
lukego: I remember Xach mentioning this issue some time ago for packages that require write access to their own directory. Most notably it was CL-UNICODE I think.
10:33:38
Shinmera
Question: there is the DECLARATION declaration, which allows you to declare custom declarations symbols. My question is this: how would I now process these declarations if they appear in a top-level context?
10:33:45
phoe
lukego: in general, you cannot do that, since Lisp code can write stuff into asdf:system-relative-pathname at any time.
10:34:48
phoe
lukego: in practice, most systems *should* be happy with read-only access after they are compiled and built. At this point, if RW access is required, I'd try to file a bug so the system uses some temporary directories or configuration paths instead.
10:35:09
phoe
Shinmera: process? As in, do you want to be able to (DECLAIM (FROBNICATE FOO BAR BAZ))?
10:35:55
phoe
Impossible in pure ANSI CL. You'd need some mechanism to hook into the declaration system of each implementation.
10:35:57
Shinmera
I can do it if I have a macro that controls a body wherein this declaration may occur
10:36:20
phoe
Yes, but you can't do it at toplevel. The only thing standard ANSI CL can do is ignore these declarations.
10:36:27
Shinmera
just weird that DECLARATION exists at all given it's pretty useless if you can't react to it.
10:37:08
phoe
Shinmera: every Lisp implementation is allowed ignore *ALL* declarations aside from notinline, optimize safety and two more I think.
10:37:45
phoe
So, in pure theory, e.g. DECLARE CONNECTED from qtools should be freely ignorable in standard CL.
10:48:12
lukego
phoe, Shinmera: Seems like in this case the package gets a (QL:QUICKLOAD ...) before being transported onto read-only. Seems like that should trigger an ASDF:LOAD-OP right? I'm surprised there is more generated stuff being done after this.
10:53:55
jackdaniel
in principle compile-op compiles everything, load-op may use pre-existing artifacts
10:57:23
phoe
OK, to be correct - LOAD-OP may invoke COMPILE-OP if something is not compiled, but after LOAD-OP the system is sure to have been compiled and loaded
11:00:41
lukego
OK maybe I need to start digging in on a package-by-package basis and see what it is trying to do when it errors on r/o fs
11:08:11
phoe
lukego: that's the way to go, sadly. There are no general rules that can be applied here to help you.
11:08:50
lukego
I suppose that my "nuke them from orbit" option would be to setup some kind of read-write overlay/temp file system but let's see if it comes to that.
11:09:28
lukego
I'm looking at CL-UNICODE .asd now and seeing :perform (load-op (o c) (symbol-call :cl-unicode '#:create-source-files)) and wondering if it generates *every* time loaded?
11:12:11
phoe
or at least that is what https://github.com/edicl/cl-unicode/blob/911767add6656990e523cf3840f9f59a36a05385/build/dump.lisp#L241 tells me
11:12:57
jackdaniel
phoe: if you specialize load-op to do something more, then it will be done at each load
11:13:00
phoe
there is no need to generate this every time the system is loaded, only every time it is compiled
11:14:08
phoe
my question, though, is - can't CL-UNICODE cache the results of its compilation, probably by only generating and dumping the data in COMPILE-OP instead of LOAD-OP?
11:15:23
lukego
lemme see how to coax ql2nix to override the standard quicklisp version of cl-unicode with a local one that moves this from LOAD-OP to COMPILE-OP..
11:19:44
edgar-rft
https://edicl.github.io/cl-unicode/ says "CL-UNICODE builds parts of its source code automatically the first time it is compiled. This is done by parsing several Unicode data files which are included with the distribution and might take some time. This happens only once." but don't ask me if and how it works.
11:21:07
jackdaniel
so it is generated only on the first load (unless the .lisp source files get wiped out for some reason before next load)
11:22:14
phoe
but then it means that cl-unicode only dumps the new .lisp files on the first call to load-op
11:26:44
phoe
ASDF manual: "compile-op depends on prepare-op which itself depends on a load-op of all of a component’s dependencies"
11:37:11
phoe
so I don't get what is the problem with ql2nix in that case - by the time cl-unicode is compiled, all cl-unicode/build FASLs are in their proper places
11:39:10
jackdaniel
afaict lukego was only asking if it generates them every time, not stated that it does
11:44:22
lukego
I'm a little confused but I should note that I don't see the files present on the read-only file system. So the problem is consistent with jackdaniel being correct about the files not being regenerated unnecessarily - root problem might be that they failed to be created earlier for some reason and that's why they're being created now.
11:45:43
lukego
thanks for pointing this out jackdaniel. I'm rusty at reading system definition files. I saw :cl-unicode/build system and just assumed that was one of the dependencies, but now as you say I see that it's only conditionally a dependency.
11:46:50
Shinmera
for define-declaration, is there a way to determine whether the declaration occurs as a top-level declaration or within a function body?
11:49:01
Shinmera
Or, better yet, how would I retrieve the current function the declaration is expanded in?
11:54:36
Shinmera
I was hoping that for top-level functions I could gain access to the function name.
12:01:55
Shinmera
just because that exists doesn't mean the implementation isn't going to have named function objects, or the name available during compilation of a defun
12:07:01
phoe
But I have no idea how to retrieve the function name from inside SB-IMPL::NAMED-LAMBDA.
12:33:02
lukego
Curently I suspect the issue is not with cl-unicode/quicklisp/asdf but rather that it's local to the nix packaging and the shell script that tries to copy the results of asdf builds: https://github.com/bradleyjensen/ql2nix/blob/master/nixlispBundle.nix#L53-L77
12:38:12
lukego
Oh hm... my ignorance of both quicklisp and ql2nix is staring me in the face here... maybe it only does the ASDF:LOAD-OP for discovery purposes and then uses a quicklisp "bundle" API to make the actual package, which might skip the load step
12:40:15
lukego
Yes that's my working hypothesis now: that this tool is using ASDF:LOAD-OP to discover dependencies, then creating a "bundle" with ql:bundle-systems, then putting that bundle onto a read-only file system. But the generated artifacts for e.g. CL-UNICODE are being lost when creating the bundle.
12:42:43
lukego
So maybe I just need to add another load-op after the bundle is created but before it is migrated onto read-only fs
12:55:41
lukego
So the question now is, given a quicklisp bundle directory, how do I force load all the systems in that bundle
13:27:06
shka__
i have an unusual question: i would want to access and manipulate numpy arrays from CL
13:54:42
lottaquestions
Hi all, I inserted a breakpoint in a function using (break), and it worked as expected. However, once I was done with the breakpoint, I deleted it from the code and recompiled the whole buffer, C-c C-k, and the breakpoint kept being hit. I tried other options like M-x slime-eval-region, and even C-c C-c to compile the toplevel form, but the
13:57:31
lukego
maybe the function object was captured with #'foo? in that case seems tricky, have to work out where the reference would be stashed
13:58:29
lukego
maybe recompile the cl-ppcre:regex-replace-all usage in case that has expanded into code that captured the old definition?
13:58:43
lukego
you need to rerun th at code that does #'foo and make it tak the new function object
13:59:13
lukego
(and consider changing it to just 'foo if you want to indirect through the symbol to always use the current function definition in the future)
14:00:30
lukego
if you're in the debugger maybe you can patch the function object in the stack frame but that's beyond my ken..
14:02:46
lukego
another fun solution would be to hit the BREAK, get a reference to the old function object in the inspector, and then rewrite the heap to replace all pointers to the old function object with the new one. but I don't know how to do that either :) likely sbcl has some internal API for it..
14:07:09
lottaquestions
yeah, that would be fun. I know how to do somewhat similar things in gdb and C, and woudn't mind getting to the same skill level with sbcl
14:32:51
trittweiler
Clearly slime needs an interactive hex-editor of the dissasembly of function objects :)
14:33:41
trittweiler
lukego: Are you aware of sbcl's 20th birthday celebration in vienna in roughly 2 weeks, and if so, are you going to join?
14:40:40
lukego
I'm totally tying myself in knots with this ql2nix problem now. I've made a new version that hopefully works, but it seems to be failing with the same error, and much earlier than makes sense, so I'm wondering if the new "working" one is trying to bootstrap from the old broken one for some reason
14:50:23
lukego
oh, I see, I'm an idiot :). I have been working in *shell* instead of a real shell and now I remember that I started this Emacs session in a shell environment that had the broken lisp setup. So I've been fighting that old version while thinking I'm fighting the new one. (isn't nix-shell great?)
15:07:36
lukego
Xach: Thanks! Just now what I want to do is force each system in a Quicklisp bundle to be loaded. That's because some packages need to do extra init on the first load and I want that to happen now (before I copy them onto a read-only file system.) Is groveling system-index.txt for names to pass to REQUIRE a reasonable solution?
15:09:27
phoe
lukego: remember that some systems may fail to load if their foreign dependencies are not loaded.
15:10:28
lukego
I'm able to declare foreign libs as dependencies here so long as I can identify them. (McCLIM is complaining it can't find truetype fonts but I'm putting that to one side for now.)
15:11:36
lukego
somehow feels dirty to (intern (string-upcase (pathname-name (pathname line-from-system-index.txt))) :keyword) but this is not the time to be squeamish..
15:28:12
pjb
lukego: you don't need to call pathname, it's called automatically by pathname-name when given a namestring.
15:28:49
pfdietz
In twitter I am told that writing seven lines of CL is "on the verge" of reimplementing the language. The things one learns.
15:30:54
lukego
it it too onomatopoeiac to write (loop for /some/system.asd = (read-line in nil) ...) ?
15:48:33
Xach
phoe: pfdietz's suggestion to use *macroexpand-hook* to work around a problem with code that can't be directly modified was met with "well then i may as well rewrite common lisp"
15:57:25
Xach
lukego: i don't know exactly how i feel about loop variables that look like pathnames but i am feeling something
16:12:01
Shinmera
So a brief test shows that a top-level invocation of a declaration defined with cltl2:define-declaration does not actually get invoked at all on SBCL. If I use the same declaration as part of a function definition it does get invoked. Am I missing something or is that expected behaviour?
16:16:03
Bike
cltl2 doesn't seem to mention this specifically, but it refers to it in the context of augment-environment, which wouldn't be used with a proclamation.
16:17:45
Bike
not based on my quick reading here, but if it did work for proclamations too that seems fine
16:18:26
Bike
i don't know if sbcl is prepared to do that, though. looking at the implementation, user declarations augment the lexenv, but that's not where proclamations go
16:20:46
Shinmera
To give context: I'd like to register functions to be automatically invoked when a change happens elsewhere in my system. Being able to use a declaration for that would have been really nice.
16:21:23
Shinmera
But given that I can't do proclamations, nor get the function name from a declaration being expanded, the only way to do it would be annoying for the user.
16:22:30
Bike
well, i think you could change implementations to have it work for proclamations, if only because cltl2 is used infrequently enough that they won't mind modifications
16:24:28
Shinmera
I don't think I'm patient enough to lobby for that to happen, nor to implement it myself.
16:56:52
pfdietz
I like *macroexpand-hook*. Using it to layer changes onto a code base, without changing that code, can be quite useful. Rejection made me sad. :(
20:43:00
Bike
i'm talking about uh, slide 7-1. i don't understand what it would mean for assert and MAXLIMIT to compose. i mean if nothing else is assert is taking a form that has a MAXLIMIT form in it, it's not a direct composition, but what would a direct composition mean then if you have compound forms
20:48:46
LdBeth
Until 7-1 it’s all about introduction to new comers. Jump ahead to 8 to see what is macro compose
21:10:50
copec
Some people aren't particularly fond of Graham's writing style: http://www.paulgraham.com/onlisp.html
21:17:50
copec
I've never actually found myself really needing continuations for anything, but they are awesome.
21:18:42
LdBeth
copec: nope, paul graham only talks CPS and macro individually. Combining CPS and macro has become interests only after 2000 in Daniel Friedman’s article
21:19:41
copec
I read it a decade ago, but iirc he builds up a macro system for continuations in CL, although I can't recall if it is actually CPS
21:22:35
LdBeth
copec: https://www.semanticscholar.org/paper/Writing-Macros-in-Continuation-Passing-Style-Hilsdale-Friedman/752575dc24b2bb6e74ee6146df71860e10e3aee9
21:24:20
LdBeth
Here’s the article. It makes a disclaimer that On Lisp implements call/cc with macro but not to be confused with the notation of cps style macro
22:10:23
jmercouris
I assume therefore that most of the functions and implementation is implemented in CL
22:11:06
jmercouris
so my question is, what is the minimum amount of functions you need to write to bootstrap a CL implementation?
22:11:42
phoe
you can generate the whole binary from Lisp, including what is currently generated from C as the "kernel"
22:12:08
phoe
which includes directly interfacing with the OS using Lisp, writing the required assembly using Lisp, actually assembling it using Lisp, doing GC using Lisp and so on
22:13:37
jmercouris
I don't see how that's possible, ultimately something has to go through the Potato compiler
22:13:43
Xach
jmercouris: in the 80s, there was a plan to separate the spec into "blue pages" and "yellow pages", where "blue" was the core, and "yellow" was built on that core, where blue was much smaller and yellow was all the convenience stuff.
22:13:47
sjl_
The number for Shen is 46. http://shenlanguage.org/OSmanual.htm I'd imagine CL would probably need a bit more.
22:14:58
Xach
jmercouris: beach has recently been working on things along those lines, in a researchy sense, and drmeister in a practical sense
22:15:13
phoe
but these are not 100% compatible with each other and all have been customized and maintained differently
22:15:22
sjl_
http://www.shenlanguage.org/shendoc.htm#The%20Primitive%20Functions%20of%20K%20Lambda is a better link for the Shen Kl core
22:17:05
jmercouris
how could one make a most portable language that can be bootstrapped really easily?
22:17:29
jmercouris
from what I understand, this is a topic of significant research, so I'm not expecting a hard answer here
22:18:35
phoe
in case of CL, it's complicated - CL is a sizeable language even when it comes to its basics upon which the rest can be built
22:18:42
Xach
Anyone making a "new" lisp these days has a lot of public domain source code they can reuse that is relatively portable, but it's not neatly divided into core+library.
22:20:39
phoe
CATCH/THROW and TAGBODY/GO are obvious control flow operators that are hard to construct otherwise
22:20:42
drmeister
jmercouris: It's been ready for distribution as a Common Lisp implementation for several months.
22:21:19
phoe
jmercouris: obviously it is, yes - the list of all symbols of CL package is public and we can align them into a directed graph that shows what can be implemented using what.
22:22:12
jmercouris
how could we align them into a directed graph without manually trying to implement all of CL in all of CL?