libera/commonlisp - IRC Chatlog
Search
9:30:13
lukego
jackdaniel: thanks yeah that seems good enough for having a simple env to make bug reports from
9:39:01
jackdaniel
for source code distribution. instead of depending on quicklisp in CI you build a bundle artifact you may load with only asdf
9:44:15
lukego
Sounds nice. Generally I'm always looking for moar import/export of well-defined passive data. Often this seems to be hiding behind operational APIs. For example Quicklisp releases are defined by text files listing the source for each package - and CLPM too as I understand it - but these seem to be treated mostly as internal implementation details while for me it's the primary artifact of interest
9:46:21
lukego
i.e. for me the really phenomenal thing about Quicklisp is that it defines a set of relatively up-to-date software versions that are known to actually work together. that's huge -- especially in that often the reason they work is that Xach has given authors a heads-up when they break. The fact that it also has a client to automatically install stuff is just a cherry on the top.
9:50:11
lukego
I suppose that in Nix terms what's needed - and is cobbled together in various out-of-tree packages - is something similar to Bundles but exporting only metadata e.g. the list of packages, the list of dependencies between packages, the list of systems provided by each package, etc. Maybe exporting that could be a Quicklisp (or CLPM or ...) feature one day to cut down on ad-hoc wheel reinventing.
9:58:41
lukego
Sorry to spam, but here is my "duct-tape bliss" workaround to the static-vectors problem, quite satisfactory for the moment:
10:00:35
lukego
which is considerably less nasty when I add a comment linking to a proper bug report on github
10:01:53
lukego
I guess you always have to find your own tribe. This packaging setup already has a few very similar hacks to this so likely it represents my people :)
10:02:26
lukego
in a perfect world I'd prefer something managed by Git but that seems like too much of a headache when you have hundreds of projects that each need to be forked separately
10:03:56
lukego
although now we come to the real yuck for me which is that it will take about 45 minutes to see if this works due to the serial nature of the package updating code
10:03:57
jackdaniel
- does it work? ; - well, kind of. first you need to apply these 2 patches to compile it, then this binary patch to fix a miscompiled magic number - easy peasy
10:06:19
lukego
for me it's comforting to have one central file, in this case https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/lisp-modules/quicklisp-to-nix-overrides.nix, which declares all the obscure rules for installing a consistent set of working packages, including all their FFI dependencies and required patches due to bugs that need to be worked around and so on.
10:06:51
lukego
so yes, you do have all those obscure rules as you describe, but you don't have to remember them because they are collected and automatically applied.
10:07:38
jackdaniel
until you, what, change the package manager? ;) either way this seems to be only remotely related to lisp, sorry for interrupting
10:07:49
lukego
and my BATNA here is to make a fork of static-dispatch, somehow redirect my builds to that instead of the Quicklisp default, and then feel guilty about not having any strategy for keeping that in sync with upstream
10:08:56
Nilby
At least 3 OSs that I use seem to be held together by a lot of such duct tape "bliss" when you look under the hood.
10:10:12
lukego
jackdaniel: I think it's a practical issue too, e.g. I don't get automatic updates of McCLIM on my machine, because I have an older version forked in ~/quicklisp/local-projects/, and I can't easily get rid of that because it contains a dirty hack that's needed to locate fonts on my machine that is too nasty to upstream
10:10:42
lukego
so it would be an upgrade for me if my "distro" would automatically pull the latest mcclim and just automatically remember to apply my patch to that, or complain if it doesn't apply cleanly anymore.
10:13:36
Nilby
Not saying it's good, but even old Lisps you'd have to all apply-all-patches before getting a sane system.
10:13:44
jackdaniel
speaking of fonts, cl-dejavu is available on quicklisp, so if anyone has some free time, then making McCLIM depend on that would be great
10:15:09
scymtym
lukego: do you know whether this https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/lisp-modules/quicklisp-to-nix-overrides.nix#L226 and the next one are nix-specific? i can't immediately see why they would be
10:15:25
lukego
jackdaniel: Sure, but fixing stuff often has to be asynchronous. Particularly on NixOS there is a genre of problem where a package has done things in a way that's not correct but that you can get away with on most distros - e.g. hard-coding a path into /usr/share/xxx and hoping that's where the data is - but that stuff all breaks once you get to NixOS. So it creates a burden on NixOS users to try and figure out the "right" way
10:15:55
jackdaniel
lukego: I'm not preaching you to change your ways, I'm just saying that this looks damn ugly from sidelines
10:17:19
lukego
I spent a whole bunch of energy trying to "fix" the expedient workarounds that Pharo Smalltalk does on NixOS and pretty much burned out on that to the point I don't want to even look at the environment again. I'm trying not to get into that situation with Lisp :)
10:19:24
lukego
I suppose there are different kinds of ugly. This is my preferred one. Central collection of nasty hacks that are applied automatically and systematically and can be shared with other users on that same basis.
10:20:38
lukego
Here's the mcclim code btw: https://github.com/McCLIM/McCLIM/blob/master/Extensions/fonts/fontconfig.lisp#L18-L34. I just did what everyone else apparently does, added my own directory to the list, but my directory is "/nix/store/70rrm133w22d68lmx7gy3pzdp2xmjfar-dejavu-fonts-2.37" and that doesn't seem like something that should go upstream.
10:21:55
lukego
So anyway if I successfully migrate from "random hacked forks in ~/quicklisp/local-projects/" to "random hacked patches in quicklisp-overrides.nix" then I'll at least be getting automatic updates again, both of the actual lisp packages and also crowdsourced expedient hacks from other nix users...
10:23:16
lukego
and on the plus side, I'm not spending at least a whole day on this hacks-curating, and it seems like at the end I'll be able to make a pull request to nixpkgs to share the fruits of it with some small number of other people. I do like the idea of collaborative package-wrangling instead of doing everything in $HOME/
10:26:02
lukego
scymtym: Good question. No idea, it would be nice to have comments on these. I guess at least 'git blame' should work.
10:27:19
scymtym
lukego: i see, thanks. does nix require build steps to produce bit-level deterministic results?
10:27:35
lukego
scymtym: This is the upstream Lisp tooling for Nix and I never really looked at it before this week, having used a downstream one called ql2nix for creating "bundles." I believe there is some asdf-level instrumentation going into the builds, something about caching, and possibly this causes problems that are specific to this environment and need such workarounds
10:29:00
lukego
Generally speaking Nix doesn't do bit-level determinism, though maybe that is a thing now as a more recent advancement. the Nix brand of determinism is just that it was built with the same commands from the same sources, applied recursively throughout all dependnecies including e.g. compiler and libc and so on. If the actual program is non-deterministic, e.g. writes a random number to a file, that's mostly ignored.
10:29:23
_death
the nix introduction says "An important consequence is that operations like upgrading or uninstalling an application cannot break other applications, since these operations never “destructively” update or delete files that are used by other packages." but this can only be true if programs don't interact (anyway)
10:29:32
lukego
I think that the CI might try building packages multiple times and checking for differences, but that would be more of a style-warning
10:31:20
lukego
_death: I'd rate that as "mostly true." If program A depends on program B then it will be pinned to one specific version of B, identified by sha256 hash. If another program depends on B, or B is installed as stand-alone, then those will be separate copies (potentially shared if they are identical.)
10:32:54
_death
lukego: when using a program, a user can feed it output from another program, for example
10:33:21
lukego
_death: I mean in the sense that it uses it to compile, or wants it in the $PATH at runtime, or in $LD_LIBRARY_PATH, etc. But not in the sense of e.g. both attached to the same X11 server and being copy-pasted between
10:34:13
jackdaniel
OK, I'm as guilty as anyone, but let's cease offtopic. unless it is lisp-related please move i.e to #lispcafe
10:35:49
lukego
This is actually Lisp related. This is the way I'm managing my Lisp dependencies, including "shell out" executables like openssl and z3. I declare them as dependencies and Nix generates a line like "export PATH=/nix/store/abcd1234-z3/bin:..." so that the Lisp process always shells out to precisely the command that was declared as a dependency, including its patches and which compiler was used and what compiler arguments and so on.
10:36:52
jackdaniel
discussing inner workings of nix is not though; I sometimes eat when I code in common lisp but this doesn't render sandwitches ontopic
10:37:05
lukego
I have a vague fantasy of integrating this a bit more with Lisp, e.g. for Nix to be able to build a set of stand-alone executables and shared libraries that correspond with Quicklisp, so that the Quicklisp client could automatically fetch known-good working versions of all of those things. Julia ecosystem seems to have this and it works surprisingly well.
15:48:28
minion
lisp123, memo from beach: What I call "concrete data type" is anything data type where there is essentially only one reasonable implementation. An abstract data type is defined entirely by the operations possible on its instance, without reference to its implementation.
17:39:57
jcowan
beach: I don't follow that definition of CDTs. It looks to me like fixnum is a CDT on your account and so is float, but bignum is an ADT and so is string.
19:55:47
lisp123
stacksmith: Thanks. An alternative I found is (do-symbols (s (find-package "my-package")) (shadowing-import (find-symbol (symbol-name s) "my-package")))
19:56:08
lisp123
(If anybody thinks thats a bad idea for importing all internal symbols, let me know)
20:11:42
_death
you could have an implementation package that exports both symbols for external use and internally used symbols that you want to refer to in your tests.. then an external interface package can use it and export only the former symbols
20:12:32
lisp123
_death: Thank you!! That is genius (and solves a problem I was facing with a brute force shadowing-import)
20:14:07
White_Flame
personally, I think it's reasonable for tests to invasively use foo::bar to get at the internals of what it's testing
20:15:36
lisp123
White_Flame: Yeah, I have the same view. And I got tired of typing foo:: for every test
20:15:37
_death
I think it's ok (and not just for testing, as long as you've control of both ends), but may get a bit verbose
20:33:59
Nilby
It turns out I mostly test the external interface, but if not, it's foo:: and the essential multiple-cursors!
20:35:46
shka
lisp123: i write my tests in the same package and i think that this is fine, but you gonna need separate asdf system for tests
20:36:57
lisp123
shka: Thanks, yes I have that. I've ended up keeping the test package separate and doing the above trick to import all the symbols into it
20:37:19
_death
Nilby: indeed.. I often use package inferred systems style, so I can test functionality of internal modules since they export their symbols as external interface for internal use
20:39:18
_death
Nilby: but I did use foo::bar recently, as I have a module for visualization that needs to specialize on some classes and obtain information from their instances, and I didn't want to export those symbols
20:40:07
stacksmith
I try to keep things separate, but when it takes too much effort I remind myself that it is likely that no one else will ever see the crap I am making.
20:43:56
_death
they work very well for some (most) of my use cases.. though I did have one or two projects where a more coarse approach would've been better
20:49:38
Nilby
Many package and systems 1-to-1, but some package and systems relationship status has to be: "It's complicated."