libera/#commonlisp - IRC Chatlog
Search
6:17:38
lukego
beach: True. But that sounds a bit like a fortune-cooking comment that could apply to any project :)
6:19:29
lukego
and in this case I'm not sure that it would be advantageous to have the tooling running inside that Lisp image that has run out of heap space (or e.g. heap corrupted) and landed at the LDB prompt. The fact that it's partitioned into a separate subprocess is kind of handy because I'm restarting the Lisp process multiple times per day but never losing my editor state, since my Emacs is a lot more stable than my Common Lisp :)
6:20:47
lukego
is the long term vision for mcclim ecosystem to run all of the Lisp code together in the same image? Or to partition it into multiple images e.g. the way HEMLOCK had REMOTE stuff to manage an "inferior" lisp process way back when?
6:25:09
lukego
This is btw the main topic on my mind this morning: where should the inter-process/inter-language boundaries in my system be? which parts are best integrated, which best separated, and which doesn't it matter? Interesting in the context of e.g. SMT solvers, machine learning toolkits, data visualization, etc, where the benefits of integration need to outweigh a *lot* of extra work reinventing stuff that's available separately
6:26:33
beach
Right, our Common Lisp implementations are not stable enough, or safe enough, to handle an IDE in the same image.
6:27:08
beach
But it seems like the wrong approach to take for granted that our Common Lisp implementations are not great, and work around it, rather than fixing the implementations.
6:29:16
beach
I don't think that there is an agreed-upon vision for McCLIM. I know what I want, and I believe scymtym shares this vision. I also know that Shinmera (for example) does not share it. He once said something like "I will *never* use an editor that runs in the same image as my Common Lisp system".
6:30:01
lukego
My mental model of Lisp is also "I should restart the image at least once per day or week for the sake of hygeine" but I don't want ot restart my editor so often
6:31:13
hayley
My mental model of Lisp is that, would someone implement htop in CLOSOS for some reason, the machine should always have the exclamation mark that htop adds as a suffix to long uptimes.
6:31:16
beach
That is a weird model, but also probably due to restrictions of our Common Lisp implementations. Just as I don't restart my operating system even once a month, I think I shouldn't be required to restart my Common Lisp system. But the way things currently are, I pretty much have to.
6:33:16
beach
lukego: Here is how I see it. A Common Lisp implementation should have multiple first-class global environments, and instead of restarting your entire system, you might be required to trash the current first-class global environment if you (say) delete an essential function, and create a new one.
6:47:21
lukego
I know. I see lots of benefits to the fully integrated approach, and lots of benefits to the full separated approach. I'm not sure if your model is a best-of or worst-of both worlds approach?
6:48:15
beach
Right, it's hard to tell. But I know of only one way to figure it out, namely to try it and see.
6:49:02
lukego
The expected reward isn't high enough for me, but that's a very uncertain quantity of course.
6:50:48
beach
This vision I have has resulted in 2 papers per year for 8 years, and I can't think of a better line of research than that.
6:57:59
lukego
but this topic is at the top of my mind right now. mapping out my own personal computing ecosystem in terms of lisp, C, R, Julia, python, etc. what to use for which tasks, and how to make them interact? I spend a lot of time bothering about this stuff -- and that's a very real cost that you don't have when you focus on a pure Lisp stack.
7:01:37
lukego
in the machine learning landscape it seems to me like some things benefit hugely from tight integration, e.g. quickly iterating gradient descent optimization over automatically-differentiated functions written directly in the application's language e.g. accessing internal data structures to guide the process. Julia has this via Flux.jl and Zygote.jl and people rave about it.
7:02:45
lukego
I was tempted to write my application in Julia to access that stuff but current feeling is that reinventing it in Lisp would be less work than (inevitably) reinventing lots of Lisp-isms that I would miss in Julia e.g. Emacs integrations.
7:04:31
lukego
On the other hand it seems like MCMC and neural network kind of stuff doesn't necessarily have much advantage of doing inside the application. they are like extremely special-case virtual machines that you configure and run but don't really do any interesting application-specific things inside (from what I can see.) a bit like SMT solving. Maybe that stuff should just get shelled out to an off-the-shelf solver in C++/Python/ec.
7:06:26
lukego
beach: btw I respect the work that you are doing and I recognize the dream, having had the same one a long time ago, but I'm just in a different corner of this gigantic expanding computing universe these days. I regret having these nay-saying interactions with you, but it seems rooted in your value judgements of e.g. Emacs based on your own subjective criteria, so I'm not sure how to refrain from reacting :)
7:15:17
pjb
In any case, eventually this should lead to the same user experience: if you botch an environment in your CL image, you will still kill it and start a new envionment…
7:15:26
lukego
(Maybe I should try harder to avoid naysaying. SICL does look like the foundation for future generations of Lispers from where I sit and that's hugely important. I'm just a bit more myopic and not often seeing much beyond a ~ 5 year time horizon.)
7:35:20
beach
lukego: You can be as much of a naysayer as you like. Like water off a duck's back to me.
7:36:05
beach
lukego: And I am not trying to convince you. What I write is for the benefit of all #commonlisp participants. Otherwise, I would do it in a private exchange with you.
7:40:21
lukego
that's the thing, I don't like being a naysayer. I suppose that I feel baited. I'll take a break on that note :)
10:21:07
greyrat
Lukego: I am migrating my Julia dev env from vscode to emacs, and I am quited pleased with eglot (whose non-REPL experience seems as good as vscode, or better for me since I can customize a lot of stuff in elisp) and Revise.jl (so that I can run a standalone REPL that automatically gets updated when I save my edits, i.e., an alternative to SLIME). (Not to mention, copy-pasting using evil text objects for 'cells' between '##' lines helps
10:21:08
greyrat
a lot with using a non-nitegrated REPL.) (jupyter-emacs and org-babel also used to work somewhat well, but recently have broken on my new emacs 28.)
10:21:58
greyrat
What I still haven't found is how to launch a graphical REPL without firing up the whole vscode Electron machinary, as the TUI REPL isn't as nice for working with plots and images.
10:29:27
greyrat
lukego: another thing I am working on, is to create a general purpose HTTP API using Jupyter kernels that can eval code in any languages. So for example, I'll load up Julia kernels with the needed dependencies, and then I can easily "shell out" to them in any language. This seems the easiest way to interop between different languages that sometimes have expensive startup costs, to me. What are your thoughts? (I currently have a
10:29:27
greyrat
self-implemented (not using Jupyter) HTTP API for zsh, and it has been perhaps the single most useful thing I have ever built. Using that, I can easily execute zsh code in, e.g., elisp, with a syntax as nice as `(z custom-zsh-function (elisp-functions-are-evaled-and-their-results-quoted) (identity some-elisp-var) automatically-quoted-string)`: `(z mkdir -p (identity my-path))` . I currently use some zsh helpers for running pipes (or
10:29:28
greyrat
directly use `(z eval 'produce | consume')`, but it's just a matter of doing the work to support `(z produce | consumer)` directly.
10:31:44
greyrat
(I have a lot of zsh code, so zsh takes ~6 seconds to boot up on my machine, which is my I don't just use `zsh -c '...'`. The same problem applies to Julia; Python starts up fast enough that I have not needed a server for it yet.)
10:59:10
greyrat
beach, pjb: After our previous discussion of the limits of scoping on eval, I have distilled my fundamental question to this: https://stackoverflow.com/questions/69334197/common-lisp-how-do-i-set-a-variable-in-my-parents-lexical-scope
11:00:49
hayley
This is intentional and done to preserve modularity (good luck debugging a function which diddles other stack frames) and the possibility of compilation (to an extent).
11:03:08
hayley
To be fair, I made up the former part, I don't know if it's true (though I suppose it is). But the latter part is a common argument for why you can't do such things.
11:05:14
hayley
Dunno about that, myself and Gnuxie have written substantially about such things. It's plain unreasonable to expect everyone to implement such a feature to me.
11:09:23
hayley
greyrat: The way I see things is that modular design makes individuals _more_ able to manage complex projects, and not less able. So corporations which want to make motivated individuals incapable of replicating corporate efforts would rather promote anti-modular concepts.
11:14:07
greyrat
hayley: Both approaches are valid, I do not say that non-modularity is somehow good. I am saying some places benefit from non-modularity. The costs of non-modularity can be much higher for corporations, and this coupled with their social elements of conservatism and conformity, produces the current world where big corporations are very bad positoned to use such non-modular, complex code. It's all about choosing the right tools for the
11:15:19
greyrat
seok-: No idea, if it's relevant to you, but Nyxt might be of interest. It doesn't even work well on macOS though, so you really want a Linux machine for this one.
11:16:02
hayley
ACTION uploaded an image: (11KiB) < https://libera.ems.host/_matrix/media/r0/download/matrix.org/JIAFDPcycpHnNgJEvfctzKkZ/bruhcha.jpg >
11:18:37
hayley
On the contrary (again), big and established corporations can basically bail out bad design decisions. Whereas for mortals, such as myself, one would be wasting precious time trying to make it work.
11:25:16
greyrat
hayley: "bad design" is by definiton bad. I usually use such scoping hacks for well-behaved abstractions. Kind of like how Rust uses a lot of unsafe code in its internals, but presents a safe API to the user.
11:35:25
hayley
Having written a smattering of unsafe performance hacks, that is still difficult to maintain.
11:42:57
greyrat
xach: the best case scenario is to have different "dialects", ala racket, with different optimization needs and safety enforcement. "Glue" code that does not run in a hot loop can be very liberal with its (non)performativity.
11:45:09
Xach
Sure - if you don't like how the language is defined, you can make one more to your liking.
11:46:30
hayley
I don't think Racket even has the capabilities for such bogosity, unless your language is almost entirely interpreted.
11:59:01
beach
Wow, that is some very nasty code. I guess the fact that it is possible in Python explains the factor 50 performance penalty of Python over Common Lisp.
11:59:40
Xach
It reminds me of how the naive idea of "if I compile 'slow' language X to 'fast' language Y, my program will run faster" is false
11:59:54
beach
I mean, that such things are generally possible. Not that this particular possibility is responsible for it all.
12:01:44
beach
Right, there are language-design decisions that can make it extremely hard to write a compiler that generates fast code.
12:05:00
hayley
It probably could be done efficiently if you tried hard enough (you could use debugging information to diddle call frames, but then you would have to handle every inferred type possible being violated), but it is still a bad idea.
12:05:58
_death
greyrat: one way to do that in a principled way is to pass a closure that sets the variable, (lambda (new-value) (setf my-var new-value)).. you can create a macro so that you write (setter my-var) instead
12:06:13
beach
hayley: But you would also have to prevent the compiler from deleting dead variables.
12:08:15
hayley
beach: Also true. Would it affect performance substantially? You could spill dead variables and be done with them.
12:08:18
Xach
Hmm, can anyone reach https://www.lrde.epita.fr/~didier/ ? I can't get Didier's software.
12:09:48
beach
hayley: Not to bad in this case, as we decided the other day, but variables could also be replaced by strength reduction and such, so you would then have to keep them synchronized in loops.
12:11:53
beach
I think this question is another piece of evidence that it is a bad idea to let people without knowledge of compiler design make decisions about language design.
12:25:32
hayley
With regards to being careful with unsafe code: from experience at best I find I need a piece of paper, in order to convince myself that my code works as intended. At worst, I keep finding mistakes (as I have been today, while testing regular expression compilation), which is not something you want to be doing for long.
12:27:28
hayley
Somewhere in the middle is using external automation tools. For example, with concurrent programs, I implement a model in TLA+ and wait a few minutes for it to check that the program won't break certain invariants. From what I read, the miri interpreter for Rust can do vaguely similar things for unsafe code.
12:29:33
hayley
(The latter does not exhaustively check models, but it can detect races and mistakes in manual memory management. But, otherwise, you need tools outside the programming language to avoid problems that are usually not problems.)
14:26:21
random-nick
there's an emacs package for interaction with scheme called geiser which is in some ways similar to slime
14:30:26
lucerne
pjb: So to confirm my understanding; I'll create a clojure that sets the var in the parent scope, then I can pass this closure around, and anyone can set the aforementioned var?
14:33:19
beach
lucerne: yes, but then there is no violation of modularity. The creator of the closure grants permission to modify the variable.
14:52:28
beach
Right, it didn't occur to me that greyrat perhaps didn't know about special variables.
15:01:07
pjb
lucerne: it's not awsome: we implemented C-like operators, so we get the same problems than in C. eg. cf. example foo* bar* in https://termbin.com/o8at
15:02:04
pjb
beach: it's an old solution, implementation of locative (aka C-pointers) in CL: http://groups.google.com/group/comp.lang.lisp/msg/d141c5636559ba3b http://groups.google.com/group/comp.lang.java.programmer/msg/1c27cc14109db55f
15:03:04
lucerne
beach: I guess what I want is local dynamic vars (the macro pjb has supplied seems exactly this?). I don't like polluting the global env with special vars.
15:03:45
beach
You don't have to "pollute" the global environment. Just declare the variable SPECIAL.
15:03:46
pjb
lucerne: if you want local dynamic variables, then just declare the variable special locally!
15:06:06
pjb
lucerne: it only affects mentions of this variable in the lexical scope of the special declaration.
15:06:50
pjb
special variables name dynamic variables that live in time, while lexical variable live in space.
15:07:06
Nilby
I know beach won't approve of using the words "lexical special" because it's technically incorrect.
15:07:33
lisp123
pjb: special variables name dynamic variables that live in time, while lexical variable live in space. >>> best explanation I have heard yet
15:11:30
Nilby
So what's the proper term? How does differentiate between a defvar and a (let (x) (declare (special x)) ...)
15:11:56
pjb
greyrat: anyways, beach's comment means: Why do you want do that? It's a bad idea, don't do that!
15:13:59
pjb
Nilby: define-symbol-macro is all-encompassing, while (let (x) ...) is a lexical scope.
15:14:16
beach
The difference is whether the variable is special in the global environment or not. Hence my suggested "local".
15:15:40
Nilby
I understand how it works, I just want to use words that won't bother or confuse people.
15:16:23
pjb
Also, lexical scopes are not always contiguous, hence the notion of locality can be topologically strange. Similarly, temporary scopes (dynamic bindings) can be shadowed, and also, when threads are involved, can have strange topologies, since threads represent parallel times.
15:17:07
lisp123
Nilby: Have a think about what lexical means, hopefully that will help explain why that wording doesn't work
15:17:50
pjb
See for example, the lexical scope where functions defined with labels are visible (on the right): http://informatimago.com/~pjb/files/lisp/common-lisp/flet-or-labels.png
15:18:46
Nilby
I do have a fondness for utilizing the thread topologies of "local" or "non-global" specials.
15:22:22
Nilby
lisp123: I don't like thinking. But, these variables are in fact introduced in a lexical scope, and then set somewhat but not totally free, by the declaration, so perhaps one should call them "free-lexicals".
15:23:51
lisp123
I wouldn't do that since the meaning of lexical is most closely associated with the textual region - e.g. definition of lexical scope " Here references to the established entity can occur only within certain program portions that are lexically (that is, textually) contained within the establishing construct. "
15:23:51
Nilby
Of course one doesn't want to imply that they're free in time or space cost, which is the nearly the opposite.
15:24:19
beach
Why would you not use the word "special" to describe a restricted kind of special variable.
15:24:59
beach
An adjective can be used to indicate such a restriction. But to use a completely different noun for it seems just so wrong.
15:29:10
pjb
lisp123: true, but there's an operator that maps between time and space: it's eval (execution in general). The "text" in a lexical scope is evaluated "during" a time period. This is how dynamic environments come alive.
15:30:11
Nilby
I'll just say "local special" from now on. If beach approves, that's good enough for me.
15:31:05
pjb
But granted a symbol that is locally declared special, let you name a dynamic variable temporarily.
15:31:40
pjb
Note that the dynamic variable that is temporarily named depends on the dynamic scope: it can be an outer temporary dynamic variable, or the global dynamic variable.
15:36:10
lisp123
Adding (declare (special g)) gives an error here: (defun compose (f g) #'(lambda (x) (declare (special g)) (funcall f (funcall g x))))
15:40:33
lisp123
Is it because g now has dynamic extent - is it disestablished after the function is defined?
15:43:02
lucerne
lisp123: I don't understand your function at all. Can you explain what it is supposed to do in plain English?
15:45:01
lucerne
pjb: I looked at your examples of failure modes. Indeed, the solution really only works if the names of the original functions are also supplied as strings. And these special vars seem to be just a better fit. I had misremembered `declare special` as global like `defvar`.
15:46:12
lucerne
BTW, can one inspect the "stack" of a special variable? To see what value it has at the grandparent scope, for example.
15:47:04
lisp123
lucerne: I'm not sure if I can explain it that well, https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html goes through it a bit more
15:54:01
beach
lisp123: Yours is a "free declaration" because it is not attached to the lambda-list variable G.
15:54:36
lisp123
beach: Its a bit confusing for me because there are references to both f & g within the funcalls
15:58:39
lisp123
Out of curiosity to see what would happen. From what I understand, f & g have indefinite extent (" The parameter bindings for f and g do not disappear because the returned function, when called, could still refer to those bindings."), so wanted to see what would happen if one adds in a special declaration which gives g dynamic extent? However, both you and Nilby say its a different G, but I can't seem to get my head around that
15:59:43
pjb
lisp123: your lambda is calling now a function that should be bound to a dynamic variable named g. The parameter of compose named g is a lexical variable!
16:00:39
beach
lisp123: Your SPECIAL declaration is not "attached to" the lambda-list variable G, because for that to be the case, it has to be first in the DEFUN body.
16:00:44
pjb
lisp123: if you declare it special: (defun compose (f g) (declare (special g)) #'(lambda (x) (declare (special g)) (funcall f (funcall g x)))) then the lambda will perhaps refer to the same dynamic variable that is now the parameter of compose, but since it's not called WHEN compose is evaluated, it won't be!
16:01:30
beach
lisp123: So it is a "free declaration", and it affects only references, as the Common Lisp HyperSpec page says. Not any binding like the lambda-list variable is.
16:01:35
pjb
lisp123: on the other hand, if you call the anonymous function returned by compose WHEN there is a dynamic variable named g defined that moment (= temporal scope), then then it will be able to call it.
16:01:52
pjb
lisp123: but since in general, random dynamic variables are unbound, you get this error.
16:05:15
pjb
lisp123: you should read more Asimov! eg. "Gimmick Three" https://en.wikipedia.org/wiki/Gimmicks_Three
16:05:54
pjb
lisp123: you cannot emprison the devil in a thick eternal bronze cube. Because he can travel back in time, before the cube existed, and escape!
16:06:14
lisp123
beach & pjb: thanks, it doesn't make too much sense, but if I read it enough times hopefully it will
16:07:01
pjb
lisp123: in the case of this compose function, if you want g to be special, then you don't need it as a parameter to compose.
16:07:52
lisp123
pjb: I was trying to break this "closures can only exist over lexical variables." by making it special
16:12:20
pjb
If you call h in a loop, you can even change the *trace-composed-functions* binding in another thread (or in the body of the loop) >:-}~
16:20:48
Nilby
lisp123: I was first exposed to lisp around the age of 10, and I am very old. I only stopped using other languages about 16 years ago.
16:23:07
Nilby
lisp123: My dad thought it would stop me from getting in trouble to put me in front of a teletype at MIT. He was very wrong.
16:33:31
Nilby
lisp123: In my case it didn't matter since I was born with a severe allergy to academia. It mearly gave me a warped and mostly incorrect ideas about robots, programming languauges, and the future.
16:40:23
etimmons
borodust: I noticed that in CLAW repos you like to include the foreign lib's repo as a submodule. Have you messed around at all with having ASDF build the foreign lib for you?
16:42:25
borodust
etimmons: i do link foreign lib repos in wrapper projects and primary reason is to have specific version of headers
16:43:44
borodust
etimmons: it's hard to get things goind in native way across different platorms (different knobs needed to be triggered to build properly), implementing that in asdf feels like being a total nightmare
16:46:22
etimmons
borodust: That's fair! Kind of just wondering how you ensure the header version matches the lib version if you use the system's lib
16:47:34
borodust
etimmons: btw, also when putting bindings into a quicklisp dist i strip everything foreign - no headers, no c/c++ code, no foreign project as a whole
16:48:18
borodust
meaning user won't be able to build foreign library at all by using system from a dist at all