freenode/#lisp - IRC Chatlog
Search
10:25:00
lukego
Is anybody else using ql2nix (https://github.com/bradleyjensen/ql2nix) to snapshot quicklisp distributions with nix or is it just me? I'm noticing that some packages aren't happy living on read-only filesystems and trying to figure out a fix/workaround.
10:26:39
lukego
seems like it's mostly packages that want to do some code generation and store the results on disk. I'm hoping it will be enough to ASDF:LOAD-OP them on a read/write filesystem and then transport the files onto read-only later. Or will they want to always be able to read/write their source dirs?
10:27:36
Shinmera
For the packages I control they either do their caching on first load, or whenever the user requests it.
10:29:44
Shinmera
From the top of my head, those would be qt-libs, uax-9, uax-14. There might be more, I don't quite remember.
10:33:00
phoe
lukego: I remember Xach mentioning this issue some time ago for packages that require write access to their own directory. Most notably it was CL-UNICODE I think.
10:33:38
Shinmera
Question: there is the DECLARATION declaration, which allows you to declare custom declarations symbols. My question is this: how would I now process these declarations if they appear in a top-level context?
10:33:45
phoe
lukego: in general, you cannot do that, since Lisp code can write stuff into asdf:system-relative-pathname at any time.
10:34:48
phoe
lukego: in practice, most systems *should* be happy with read-only access after they are compiled and built. At this point, if RW access is required, I'd try to file a bug so the system uses some temporary directories or configuration paths instead.
10:35:09
phoe
Shinmera: process? As in, do you want to be able to (DECLAIM (FROBNICATE FOO BAR BAZ))?
10:35:55
phoe
Impossible in pure ANSI CL. You'd need some mechanism to hook into the declaration system of each implementation.
10:35:57
Shinmera
I can do it if I have a macro that controls a body wherein this declaration may occur
10:36:20
phoe
Yes, but you can't do it at toplevel. The only thing standard ANSI CL can do is ignore these declarations.
10:36:27
Shinmera
just weird that DECLARATION exists at all given it's pretty useless if you can't react to it.
10:37:08
phoe
Shinmera: every Lisp implementation is allowed ignore *ALL* declarations aside from notinline, optimize safety and two more I think.
10:37:45
phoe
So, in pure theory, e.g. DECLARE CONNECTED from qtools should be freely ignorable in standard CL.
10:48:12
lukego
phoe, Shinmera: Seems like in this case the package gets a (QL:QUICKLOAD ...) before being transported onto read-only. Seems like that should trigger an ASDF:LOAD-OP right? I'm surprised there is more generated stuff being done after this.
10:53:55
jackdaniel
in principle compile-op compiles everything, load-op may use pre-existing artifacts
10:57:23
phoe
OK, to be correct - LOAD-OP may invoke COMPILE-OP if something is not compiled, but after LOAD-OP the system is sure to have been compiled and loaded
11:00:41
lukego
OK maybe I need to start digging in on a package-by-package basis and see what it is trying to do when it errors on r/o fs
11:08:11
phoe
lukego: that's the way to go, sadly. There are no general rules that can be applied here to help you.
11:08:50
lukego
I suppose that my "nuke them from orbit" option would be to setup some kind of read-write overlay/temp file system but let's see if it comes to that.
11:09:28
lukego
I'm looking at CL-UNICODE .asd now and seeing :perform (load-op (o c) (symbol-call :cl-unicode '#:create-source-files)) and wondering if it generates *every* time loaded?
11:12:11
phoe
or at least that is what https://github.com/edicl/cl-unicode/blob/911767add6656990e523cf3840f9f59a36a05385/build/dump.lisp#L241 tells me
11:12:57
jackdaniel
phoe: if you specialize load-op to do something more, then it will be done at each load
11:13:00
phoe
there is no need to generate this every time the system is loaded, only every time it is compiled
11:14:08
phoe
my question, though, is - can't CL-UNICODE cache the results of its compilation, probably by only generating and dumping the data in COMPILE-OP instead of LOAD-OP?
11:15:23
lukego
lemme see how to coax ql2nix to override the standard quicklisp version of cl-unicode with a local one that moves this from LOAD-OP to COMPILE-OP..
11:19:44
edgar-rft
https://edicl.github.io/cl-unicode/ says "CL-UNICODE builds parts of its source code automatically the first time it is compiled. This is done by parsing several Unicode data files which are included with the distribution and might take some time. This happens only once." but don't ask me if and how it works.
11:21:07
jackdaniel
so it is generated only on the first load (unless the .lisp source files get wiped out for some reason before next load)
11:22:14
phoe
but then it means that cl-unicode only dumps the new .lisp files on the first call to load-op
11:26:44
phoe
ASDF manual: "compile-op depends on prepare-op which itself depends on a load-op of all of a component’s dependencies"
11:37:11
phoe
so I don't get what is the problem with ql2nix in that case - by the time cl-unicode is compiled, all cl-unicode/build FASLs are in their proper places
11:39:10
jackdaniel
afaict lukego was only asking if it generates them every time, not stated that it does
11:44:22
lukego
I'm a little confused but I should note that I don't see the files present on the read-only file system. So the problem is consistent with jackdaniel being correct about the files not being regenerated unnecessarily - root problem might be that they failed to be created earlier for some reason and that's why they're being created now.
11:45:43
lukego
thanks for pointing this out jackdaniel. I'm rusty at reading system definition files. I saw :cl-unicode/build system and just assumed that was one of the dependencies, but now as you say I see that it's only conditionally a dependency.
11:46:50
Shinmera
for define-declaration, is there a way to determine whether the declaration occurs as a top-level declaration or within a function body?
11:49:01
Shinmera
Or, better yet, how would I retrieve the current function the declaration is expanded in?
11:54:36
Shinmera
I was hoping that for top-level functions I could gain access to the function name.
12:01:55
Shinmera
just because that exists doesn't mean the implementation isn't going to have named function objects, or the name available during compilation of a defun
12:07:01
phoe
But I have no idea how to retrieve the function name from inside SB-IMPL::NAMED-LAMBDA.
12:33:02
lukego
Curently I suspect the issue is not with cl-unicode/quicklisp/asdf but rather that it's local to the nix packaging and the shell script that tries to copy the results of asdf builds: https://github.com/bradleyjensen/ql2nix/blob/master/nixlispBundle.nix#L53-L77
12:38:12
lukego
Oh hm... my ignorance of both quicklisp and ql2nix is staring me in the face here... maybe it only does the ASDF:LOAD-OP for discovery purposes and then uses a quicklisp "bundle" API to make the actual package, which might skip the load step
12:40:15
lukego
Yes that's my working hypothesis now: that this tool is using ASDF:LOAD-OP to discover dependencies, then creating a "bundle" with ql:bundle-systems, then putting that bundle onto a read-only file system. But the generated artifacts for e.g. CL-UNICODE are being lost when creating the bundle.
12:42:43
lukego
So maybe I just need to add another load-op after the bundle is created but before it is migrated onto read-only fs
12:55:41
lukego
So the question now is, given a quicklisp bundle directory, how do I force load all the systems in that bundle
13:27:06
shka__
i have an unusual question: i would want to access and manipulate numpy arrays from CL
13:54:42
lottaquestions
Hi all, I inserted a breakpoint in a function using (break), and it worked as expected. However, once I was done with the breakpoint, I deleted it from the code and recompiled the whole buffer, C-c C-k, and the breakpoint kept being hit. I tried other options like M-x slime-eval-region, and even C-c C-c to compile the toplevel form, but the
13:57:31
lukego
maybe the function object was captured with #'foo? in that case seems tricky, have to work out where the reference would be stashed
13:58:29
lukego
maybe recompile the cl-ppcre:regex-replace-all usage in case that has expanded into code that captured the old definition?
13:58:43
lukego
you need to rerun th at code that does #'foo and make it tak the new function object
13:59:13
lukego
(and consider changing it to just 'foo if you want to indirect through the symbol to always use the current function definition in the future)
14:00:30
lukego
if you're in the debugger maybe you can patch the function object in the stack frame but that's beyond my ken..
14:02:46
lukego
another fun solution would be to hit the BREAK, get a reference to the old function object in the inspector, and then rewrite the heap to replace all pointers to the old function object with the new one. but I don't know how to do that either :) likely sbcl has some internal API for it..
14:07:09
lottaquestions
yeah, that would be fun. I know how to do somewhat similar things in gdb and C, and woudn't mind getting to the same skill level with sbcl
14:32:51
trittweiler
Clearly slime needs an interactive hex-editor of the dissasembly of function objects :)
14:33:41
trittweiler
lukego: Are you aware of sbcl's 20th birthday celebration in vienna in roughly 2 weeks, and if so, are you going to join?
14:40:40
lukego
I'm totally tying myself in knots with this ql2nix problem now. I've made a new version that hopefully works, but it seems to be failing with the same error, and much earlier than makes sense, so I'm wondering if the new "working" one is trying to bootstrap from the old broken one for some reason
14:50:23
lukego
oh, I see, I'm an idiot :). I have been working in *shell* instead of a real shell and now I remember that I started this Emacs session in a shell environment that had the broken lisp setup. So I've been fighting that old version while thinking I'm fighting the new one. (isn't nix-shell great?)
15:07:36
lukego
Xach: Thanks! Just now what I want to do is force each system in a Quicklisp bundle to be loaded. That's because some packages need to do extra init on the first load and I want that to happen now (before I copy them onto a read-only file system.) Is groveling system-index.txt for names to pass to REQUIRE a reasonable solution?
15:09:27
phoe
lukego: remember that some systems may fail to load if their foreign dependencies are not loaded.
15:10:28
lukego
I'm able to declare foreign libs as dependencies here so long as I can identify them. (McCLIM is complaining it can't find truetype fonts but I'm putting that to one side for now.)
15:11:36
lukego
somehow feels dirty to (intern (string-upcase (pathname-name (pathname line-from-system-index.txt))) :keyword) but this is not the time to be squeamish..
15:28:12
pjb
lukego: you don't need to call pathname, it's called automatically by pathname-name when given a namestring.
15:28:49
pfdietz
In twitter I am told that writing seven lines of CL is "on the verge" of reimplementing the language. The things one learns.
15:30:54
lukego
it it too onomatopoeiac to write (loop for /some/system.asd = (read-line in nil) ...) ?
15:48:33
Xach
phoe: pfdietz's suggestion to use *macroexpand-hook* to work around a problem with code that can't be directly modified was met with "well then i may as well rewrite common lisp"
15:57:25
Xach
lukego: i don't know exactly how i feel about loop variables that look like pathnames but i am feeling something
16:12:01
Shinmera
So a brief test shows that a top-level invocation of a declaration defined with cltl2:define-declaration does not actually get invoked at all on SBCL. If I use the same declaration as part of a function definition it does get invoked. Am I missing something or is that expected behaviour?
16:16:03
Bike
cltl2 doesn't seem to mention this specifically, but it refers to it in the context of augment-environment, which wouldn't be used with a proclamation.
16:17:45
Bike
not based on my quick reading here, but if it did work for proclamations too that seems fine
16:18:26
Bike
i don't know if sbcl is prepared to do that, though. looking at the implementation, user declarations augment the lexenv, but that's not where proclamations go
16:20:46
Shinmera
To give context: I'd like to register functions to be automatically invoked when a change happens elsewhere in my system. Being able to use a declaration for that would have been really nice.
16:21:23
Shinmera
But given that I can't do proclamations, nor get the function name from a declaration being expanded, the only way to do it would be annoying for the user.
16:22:30
Bike
well, i think you could change implementations to have it work for proclamations, if only because cltl2 is used infrequently enough that they won't mind modifications
16:24:28
Shinmera
I don't think I'm patient enough to lobby for that to happen, nor to implement it myself.
16:56:52
pfdietz
I like *macroexpand-hook*. Using it to layer changes onto a code base, without changing that code, can be quite useful. Rejection made me sad. :(