libera/#commonlisp - IRC Chatlog
Search
9:15:07
kakuhen
beach: Unfortunately, the style warnings do not appear without the RESTART-CASE macro.
9:15:41
kakuhen
So perhaps there is something about that macro that causes this. Interestingly enough, ECL does not complain at all. It's just CCL that complains, and SBCL complains even louder by giving twice as many style warnings as CCL.
9:17:22
pjb
beach: note: this rule about where to signal or re-signal errors from, should be different when you are debugging the implementation. often I have code that will use either handler-case or handler-bind (notably in tests) depending on what you want to debug (eg. the tested code, or the tests).
9:18:54
pjb
I understand. But when you implement the rule, you need to make it optional because the code and conditions are also used by implementers.
9:20:50
beach
Also, I would like to see the backtrace not mention the particular operator that was invoked, because the operator that failed could be the result of a macro expansion, and that would be (is?) confusing to the application programmer.
9:21:20
kakuhen
pjb: hmm... on my side I am getting warnings, though, even when I run the REPL in a terminal rather than SLY
9:21:36
beach
Instead, the tentative plan is to show the source form (as highlighted text in the source code) that caused the problem.
9:31:53
pjb
kakuhen: I use (proclaim '(optimize (safety 3) (debug 3) (space 0) (speed 0) (compilation-speed 3)))
9:33:37
kakuhen
I'm starting to think backquoting restart-case is just giving me more trouble than what it's worth
9:37:04
pjb
kakuhen: https://termbin.com/xr0o2 ; probably the warning occurs when the implementations tries to perform compilation-time optimization that would require the class to be defined, which it cannot do without evaluating the defclass that is in the progn. It cannot do that because it's in a single progn form; but should be able to do, because it's a toplevel progn form, so it could be spliced out.
9:37:52
hayley
Is there much I can do to make it more likely that I can handle a STORAGE-CONDITION (for running out of memory) rather than crashing the Lisp implementation?
9:38:18
kakuhen
pjb: one of my friends speculated this, since sbcl and ccl apparently share some common history with their compilers, and both have very similar behavior when running this macro, yet ECL didn't
9:38:29
hayley
For example, (handler-case (loop collect 2) (storage-condition () 'no)) crashes into LDB rather than signalling on SBCL.
9:38:41
pjb
hayley: one trick could be to allocate some vector, and when you get this storage-condition, to release the last reference to that vector to free some space.
9:39:00
pjb
hayley: that should require a minimum of space to perform, but the question is whether it will be enough to continue.
9:39:59
pjb
hayley: somebody did an evaluation of the handling of storage-condition in various implementation when usenet cll was still a thing, but nothing moved since.
9:40:17
pjb
hayley: somebody would have to provide patches to the implementations. So far it has been easier to just buy more RAM.
9:41:09
hayley
pjb: My current "threat model" is that we are overwhelmed with messages on a server, and so we run out of memory allocating objects for them all. To handle it, we would probably kill threads which try to run out of memory.
9:41:15
pjb
One problem is that the storage-condition handler may require some memory to work. If the heap is already overextended, it may be difficult. So the implementation should reserve some space for the handlers in this situation.
9:42:05
hayley
Given that such message objects would be large, maybe with large strings or byte vectors, it seems I would have a good chance on SBCL.
9:45:37
flip214
hayley: well, you could mmap() your message files and have your OS swap them in/out as needed?
9:46:09
flip214
might not be as easy to handle as strings or ub8 vectors... don't know what you need, though.
9:48:51
hayley
I already have a message size limit, but one could just send more messages at that size limit on multiple connections (and thus threads; I guess every other multiplexing model looks a bit nicer with this problem) and blow the heap that way.
9:50:05
flip214
so if you know how much heap you need per message, and the size of your heap, you can restrict the number of parallel threads and be done
9:50:53
hayley
Right. I don't think I know how much heap I need per message, and it would hurt my head too much to use only TCP buffers.
9:51:09
moon-child
I would change the variable there and say, rather, you know how much heap you need per message, and the number of cores your machine has; so you know the size your heap needs to be
9:51:24
moon-child
hayley: you may not know exactly, but it should not be overly difficult to come up with a conservative estimate, no?
9:52:34
hayley
I don't know what my server will be used for, and so I could only say something between 10² and 10⁷ bytes per message?
9:57:29
hayley
So, taking the maximum limits the throughput substantially if that is not the case, but a more conservative estimate would be too conservative. Hence why I would like a more...I suppose "feedback"-driven approach?
9:57:44
moon-child
I don't think this kind of probabilistic approaches to security is the right one. Is it better that you have (say) an 80% chance of signalling a condition than a 5% chance? Marginally. But if the implementation is unwilling to _reliably_ signal such a condition, I think it is better to accept that as a constraint and look for an alternative solution (or, patch the implementation, as pjb
9:59:02
hayley
I would consider it to be a performance hack, as there is no security (or progress, for that matter) lost if the server crashes outright, but I would prefer to only kill one connection rather than all of them.
9:59:10
moon-child
for instance, separate your server into multiple processes and use the operating system to coordinate them
9:59:21
flip214
hayley: run (TIME (process-one-message)) - the output will tell you how much heap was allocated. round up, and estimate.
10:00:11
flip214
also, you can have a background thread monitoring heap usage - but then you'll need to find out which thread to kill...
10:01:16
flip214
depending on your usecase (needs IO, or not) it might be easier to just limit number of thread == number of cores. then every thread can process a message without any wait time => minimum latency, minimum heap usage.
10:01:45
flip214
works for me with PDF generation - there's no IO involved, just CPU, so there's no reason to have more threads than can run at the same time.
10:03:20
moon-child
I think a server must of necessity do a lot of i/o. But I think the usual approach there is to just have one i/o thread which receives connections and pushes them onto a work queue; and to do writing asynchronously
10:04:11
hayley
I can't seem to understand anything that isn't "straight line" networking code with threads, such as using some callback async library or polling or whatnot. Then writing it is daunting.
10:06:13
flip214
hayley: SB-EXT:CALL-WITH-TIMING can give you a :BYTES-CONSED result, so your threads can (over time) estimate their memory use themselves ;)
10:07:03
flip214
moon-child: I meant disk-IO that would block worker threads. as soon as an HTTP request has arrived, all the work until pushing the result might be possible without any IO.
10:07:38
moon-child
ah, yes. Usual approach for this is coroutines, but I don't think we have any cl implementations that support them
10:08:22
moon-child
(of course, come closos, plain threads will be cheap enough it won't matter. But who knows then that will be)
10:08:45
hayley
Another difference is that I have connections which are expected to be persistent, whereas I can't remember if Hunchentoot supports keep-alive.
10:10:13
flip214
hayley: you can tell HT to ignore a TCP socket - and pass that on to an extra thread.
10:11:27
hayley
moon-child: FWIW my secret plan is to prototype "green threads" in a fork of SICL, which would then be somewhat reusable for a thread implementation in CLOSOS.
10:13:28
flip214
hayley: green threads - with all the features of OS threads? like scheduling priorities, blocking behaviour for OS calls, timeouts, ...??
10:14:03
flip214
I've worked on such a library in C - but it never got up to par on features with kernel threads.
10:16:21
hayley
Well, Erlang has priorities, and in the case of CLOSOS, blocking OS calls could be handled with the usual devices (mostly semaphores I suppose).
10:18:36
hayley
On a Unix system, I think the Erlang and Haskell people both either attempt to use asynchronous operations or a thread pool for synchronous operations, which still use something like mailboxes to unblock the appropriate thread and provide the result.
10:22:32
hayley
Apparently the fastest servers these days use other async stuff, which is backed by some polling technique (epoll, etc, maybe io_uring if you're fancy). Without some intermediate stuff they don't work too well with threads apparently.
10:24:43
flip214
yeah. But if your actual workload (PDFs, for me) take 30-60msec to generate, and I only have a handful of cores (and a handful of threads), the server implementation doesn't actually matter.
10:25:10
flip214
of course, the "hello world ~a" examples need to keep the server as small as possible for their benchmarks...
12:23:23
yitzi
beach: Thanks for the paper reference regarding CL and SICL debugging. Very informative.
12:24:42
yitzi
I noticed that there wasn't a section on CMUCL. From what the user manual and repo says they support stepping and breakpoints, although the stepper was broken until recently.
12:27:14
yitzi
Not sure how well maintained it is though. I can't currently test it in my environment since CMUCL can't load ironclad on my system.
12:28:25
jackdaniel
flip214: kernel threads become troublesome when there are a lot of them (the context switch of a kernel thread is more expensive)
12:29:41
yitzi
beach: If you are curious you can run the Jupyter debugger stuff here: https://mybinder.org/v2/gh/yitzchak/common-lisp-jupyter/add-debugging?urlpath=lab
12:29:42
jackdaniel
also green threads (when pooled in a single native thread) have assured a certain amount of atomicity
12:30:21
jackdaniel
a side-benefit is that if you bother to implement green threads you will probably have delimited continuations for free
12:30:41
jackdaniel
(and with that come numerous interesting programming techniques - i.e generators)
12:31:43
flip214
and with _lots_ of kernel threads you run in all the (already-solved, on the kernel-side) scheduling things - O(1), priorities, fairness, ...
12:37:18
hayley
jackdaniel: I thought of green threads in the terms of undelimited one shot continuations.
12:40:14
yitzi
beach: In addition to debugging, stepping, source and variable inspection there is also symbol completion and help/inspect. Along with a widgets and pile of other stuff.
12:41:32
yitzi
The debugging is still a work in progress. The other stuff is pretty well tested and used by drmeister's group. There is also a maxima-jupyter kernel available based on the same code.
12:44:40
yitzi
beach: Yep. I've already started tweaking some of the debug messaging based on some your ideas. Thanks again!
12:46:15
hayley
I read a delimited continuation was a continuation which was "delimited" to some frame with shift/reset.
12:48:04
jackdaniel
in non-scheme terms, a delimited continuation is a computing context that may be resumed, and after resuming it may yield control again (and be resumed again)
12:48:30
Bike
"one shot" means you can only use it once, like because the stack isn't copied so it will no longer be in place if you try to continue there later. so register saving works.
12:49:56
hayley
I /guess/ we are delimited by THREAD-YIELD (and preemptive scheduling) but it doesn't fit how they are presented in Scheme land in my opinion.
12:52:02
Bike
i don't think threads are really "delimited continuations". the way the continuations given by shift work, they add to the current call stack, so they return like normal functions (to muddle several levels of description)
12:52:05
jackdaniel
so if the stack is not copied and you allow yielding, then how does it work? all threads share the same stack?
12:53:43
hayley
Each thread has a separate stack, and yielding saves state, returns to some scheduler, then loads the new state.
12:53:53
jackdaniel
I didn't say that delimited continuations are threads, I'm saying that cooperative threads may be implemented with delimited continuations
12:55:05
hayley
But my observation was the only green thread implementation with continuations uses undelimited continuations once, as I couldn't spot the delimiting.
12:55:37
hayley
Somehow delimited continuations are harder for me to understand than undelimited even though the former is supposedly much better to use.
12:56:22
hayley
The former does some stack knotting, whereas the latter is a funny first class GOTO.
12:56:22
Bike
they have some interesting advantages over undelimited, but simplicity is not one of them
12:58:00
jackdaniel
I will read on that later, but some loose searching seems to indicate, that one-shot continuations are a certain class of delimited continuations
13:00:27
Bike
and most hagiographies of delimited continuations emphasize their composability which kind of goes with reusability
13:00:28
jackdaniel
continuations that can be called multiple times are full continuations, one-shot may be resumed only once and delimited capture the entire /remaining/ continuation; that's what I've grasped from this quick search
13:01:12
hayley
I think get/setcontext in POSIX-2001 manipulate one shot undelimited continuations, as they are just all the registers as described previously.
13:03:23
hayley
Before I was going to say setjmp/longjmp but those are mere escape continuations as I understand how you are supposed to use them.
13:04:47
Bike
yeah longjmp has the "UB if the function that used setjmp has returned" bit, so they're only an escape
13:42:03
rain3
a 2 hours long video from Atlanta Functional Programming - Common Lisp Study Group youtube channel can be condensed in 1-5 pages of good old text tutorial which in turn can be studied in 10-30 minutes instead of 2-6 hours of listening and watching and pausing and scrolling
13:42:51
rain3
interesting stuff there https://www.youtube.com/watch?v=tcmY7zstig4 but it takes too long to watch a 2h long video for only 10 minutes of essential info
13:45:48
jackdaniel
so while the former purpose is teaching, the latter purpose is socializing (in my understanding)
13:46:49
rain3
socializing around good content which can be summarized as a high quality tutorial , either before or after the meeting
13:48:38
jackdaniel
ah, that's a shame; perhaps you could write a high-quality tutorial or a paper on another topic related to lisp; I'd gladly read it
13:49:08
lotuseater
I skipped through it, is ":filter :something" a method-combination? wasn't aware they can be done that way, but I think because I didn't define ones myself yet
13:50:02
ldb
lotuseater: that's probably optional argument lists, I don't think you need method-combination for that
13:50:49
rain3
jackdaniel: on the other hand your CLIM videos are from beginning to end golden and enjoyable, good and efficient for teaching others
13:51:34
lotuseater
they have something like (defmethod stack-push :filter :number ((stack (eql 'normal))) ...)
13:55:11
jackdaniel
that's nice to hear; either way preparing a tutorial after giving a talk is an extra work - if someone does that it is fine; but expecting it from that person is a bit much
13:57:06
rain3
I am not asking for it.. I am only pointing out that the same thing can be done more efficiently . If you have the tutorial prepared before, then you don't need to waste time thinking and fighting with bugs 'sorry, it didn't work, we'll try again in the next video'
14:51:05
pjb
rain3: totally agree, videos are two slow in general. However it is not necessarily a bad thing: 1- you can listen to them in parallel to some other task. 2- sometimes you're not working at 100% of your capacities yourself, so it's not so bad they're a little slow. 3- it can give you an excuse for such a slow day.
14:51:41
pjb
(and you can always play them at 2x, even if one may sometimes regret there's not an option to play them at 10x).
14:52:59
pjb
Perhaps somebody at google will implement an AI to summarize a video, and transform it into a multimedia stream that could be absorbed by brains in high speed. Or perhaps we'll have to wait for neuralink to provide the brain downloading feature…
14:53:11
rain3
the video above actually was quite good in comparison with others. I've downloaded them so that I can play at 10x with vlc player , then I deleted all of those which only had examples from the readme of the lib
15:36:39
zos
Hello all, I am interested in learning common lisp. I have started working my way through Practical Common Lisp. Right now I am using SBCL with Doom Emacs. In some tutorials I have come across mentions of roswell and am wondering if it is worth setting up from the start.
15:37:47
rain3
it's good to set it up so that you can easily switch between different versions of SBCL when needed
15:40:20
Lycurgus
i believe in n chances, especially if somebody else does the m+1 of them, in my case m is like 2 -4
15:42:57
zos
That sounds like good advice - it is going to take some time before I am likely to need to worry about deployment. Thank you for your help
15:44:07
beach
zos: #commonlisp is not really for newbie questions, in case you want to ask such questions at some point. They are tolerated though, and if they become too trivial or too numerous, you might be asked to go to #clschool.
16:14:56
phantomics
When I eval: (defvar foo 123) (let ((set-foo (lambda (x) (setf foo x))) (foo foo)) (funcall set-foo (setq foo 55)) (* foo 3))
16:16:11
phantomics
Why doesn't the set-foo function defined in the (let) affect the top-level binding of foo? The function is defined before the value inside the (let) form is locally bound to foo
16:18:31
phantomics
Ok, so this method would work if foo was a lexical variable defined in a (let) form containing the code rather than a special variable?
16:19:15
jackdaniel
daily quiz: what's a practical value of (apply foo (car bar) (cdr bar)) compared to (apply foo bar) ;?
16:19:47
beach
phantomics: It depends on your definition of "work", but yes, you can close over lexical variables.
16:24:33
jcowan
Sure, but my claim was that there are no global lexical variables, as you cannot wrap a let around your program (becaause too many things in it insist on being top-level)
16:25:27
jackdaniel
there are file-local-variables (in the standard it is i.e *readtable*, but there is also asdf extension that gives you that)
16:27:10
beach
jackdaniel: I can imagine if foo is something like #'apply, and (car bar) is something like #'reduce, but I am not sure.
16:27:18
jackdaniel
of course one could argue that the practical application I have in mind can't be considered a good style of programming
16:28:33
jackdaniel
however you could specialize the first argument on null as a fallback, and then bar could be nil
16:39:13
pjb
zos: have a look at http://cliki.net/Getting+Started http://cliki.net/Online+Tutorial and in general, http://cliki.net
16:39:56
j-sh
What is the current best way to use the GNUstep GUI toolkit (and maybe its RAD tool Gorm) with common lisp?
16:43:27
pjb
phantomics: yes, it would work with lexical variables, but the question is why did you make a temporary binding of foo in your let?
16:44:38
pjb
phantomics: it's not wrong, that's the point of dynamic variables: they let you shadow temporarily the previous binding, during the execution of a let body. But the point is that if you want to have the previous binding mutated, you should not temporarily shadow it!!!
16:45:14
pjb
(defvar *foo* 123) (values (let ((set-foo (lambda (x) (setf *foo* x)))) (funcall set-foo (setq *foo* 55)) (* *foo* 3)) *foo*) #| --> 165 ; 55 |#
16:46:16
pjb
phantomics: note that defvar declares the variable name as special; this is a global declaration that cannot be removed or shadowed. So it is important to use the earmuff convention foo -> *foo* to avoid damaging the normal variable names.
16:48:29
pjb
j-sh: the first problem is that there are different Objective-C runtimes: Apple's runtime, GNU runtime, cocotron runtime (used on MS-Windows I believe).
16:49:49
pjb
j-sh: the next problem is the FFI from lisp to Objective-C; there are some nice things: the objc runtime is a pure C API. There are some more difficult things: depending on the runtime and the libraries or frameworks, it's not always easy to introspect the API; we still have to process the headers, and this is not easy (failproof) to do it automatically.
16:51:19
pjb
j-sh: ccl provides a FFI that works with Apple's runtime. I don't know how well it could work with GNU Objective-C runtime. There's an Objectiev-C FFI that could work with GNU Objective-C (and possibly cocotron, IIRC), but AFAIK it has bitrotten.
16:52:18
pjb
even ccl uses a ffigen that is a patch of Apple gcc version 4 that is entirely outdated and therefore is completely bit-rotten; I don't know how to generate the cdb files ccl FFI requires to deal nicely with frameworks on current systems.
16:53:42
pjb
In short, it's a mess, and we'd need to re-implement from scratch an CL -> Objective-C FFI that would work nicely for all runtimes, and all frameworks. But it's not a small project.
16:55:19
pjb
j-sh: Have a look at this objcl library (which is a reader-macro to give [recipient message:arg with:arg] syntax over ccl objc FFI: https://github.com/informatimago/lisp/tree/master/objcl
17:01:11
phantomics
pjb: I know I'm using weird conventions in this code, that's because it's for the output of my APL compiler, and APL has some odd conventions for implicitly shadowing variables but also has ways of modifying variables at the top level of their scope
17:06:02
pjb
j-sh: you can still do something. Have a look at https://cliki.net/com.informatimago.hangman
17:07:08
pjb
j-sh: well, http://git.informatimago.com/viewgit/index.php?a=tree&p=public/games/hangman-cocoa-objc&h=6a4c99a8d52e6cdaf08b28dffa67797dab8f9a89&hb=99a80c170985b9b3a753025c00870c63a94cf678 ; I don't know if I left the gnustep version on github.
17:08:06
pjb
(nope. the gnustep/ subdir only contains objective-c code… ; as I said, the problem is difficult and has not been worked on ever)
17:09:00
pjb
j-sh: basically, CL is a not used by a lot of programmers. GNUstep is not used by a lot of programmers. CL with GNUstep is used by a handful of programmers and we don't have the resources to make any progress on the combination.
17:09:21
pjb
j-sh: as beach said, it has been a long time since we talked about GNUstep in #commonlisp
17:37:52
pjb
lotuseater: j-sh: so a first step would be to define a complete FFI to the gnu objective-c runtime, to the apple objectiev-c runtime, and possibly others (cocotron, etc).
17:38:32
pjb
lotuseater: j-sh: next, define a higher level CL to objective-C interface using introspection and working with all runtimes.
17:38:52
pjb
lotuseater: j-sh: finally, use my reader macros to be able to use the nice Objective-CL syntax ;-)
17:40:09
pjb
two problems: 1- dealing with C macros in the objc runtime headers. 2- dealing with the various structure layout (packed/not packed, 32-bit/64-bit, and other #ifdef).
17:41:20
pjb
(perhaps introspecting the debugging symbols in the shared libraries of the run-time, but this adds a new complexity (gdb symbols, dwarf symbols? what to do if stripped without debugging symbols? etc).
17:52:40
j-sh
pjb Thank you for your extensive answer, I didn't know that it would be that difficult to use CL with GNUstep.
17:52:41
j-sh
I personally think that it is a shame that GNUstep didn't get the recognition that something like GTK (it is even older than GTK AFAIK) got, but I guess there aren't many people that want to touch an Objective-c codebase especially since it has the "Apple stigma" to it and even Apple themselves try as hard as possible to deprecate Objective-C
17:56:15
pjb
j-sh: well, it's not that it's difficult. It's labor intensive to do something complete.
17:56:49
pjb
j-sh: if you just want to send a few message, you really need to FFI a few functions in the runtime. You can easily do that.
17:57:35
pjb
j-sh: if there were bindings for other languages to GNUstep, that could indeed make it more popular.
17:59:10
j-sh
pjb There are apparently bindings for Java, ruby and Smalltalk, but I'm not sure how much they are maintained.
18:01:57
pjb
For example, one problem is that the code used to encode the types are defined with #define in the runtime.h header. So CL needs either to parse those headers, or to hardwire the encoding for each compiler/runtime.
18:04:58
pjb
j-sh: For example, you can cffi:defcfun objc_msgSend and sel_getUid, and then already you can send messages to objects.
18:05:32
pjb
but in CL you'd want to be able to introspect the classes and objects. So you'd want the full monty.
18:06:26
pjb
j-sh: try: (defclass foo () ((x :initarg :x) (y :initform 42))) (inspect (find-class 'foo)) (inspect (make-instance 'foo))
18:07:02
pjb
you'd want to do the same in CL for Objective-C classes. (even more, since in Objective-C the methods are attached to the classes, so you'd also want to list the methods).
18:17:11
lotuseater
a C# software developer he could also extend his language or that it couldn't be different in CL ^^ hard to make him clear why not
18:57:38
jcowan
Anyone who asks you "But why would you want to do *that*?" when talking about cross-language comparison should be resolutely ignored: nobody "wants" a feature they've never heard of, because they've learned to work around it or not to want it or are simply clueless. (However, the question does make sense within a single language, because there may well be a different way to achieve the same effect within the language.)
18:59:48
lotuseater
I also tell that's an experience no one can just tell you, you must make it yourself to really get the difference.
19:06:40
lotuseater
haha and just because something is used by more people or there are many (aweful) book about it means it's better (just better marketed)
19:09:35
jcowan
This is a highly speculative question: can anyone see a use case for a separate class hierarchy allowing you to create generic functions that aren't related to ordinary generic functions?
19:10:34
tfeb
lotuseater: 'popular does not mean good: it merely means good at being popular', as someone once said.
19:13:09
aeth
jcowan: type-based generic dispatch in CL. https://github.com/markcox80/specialization-store/
19:13:21
aeth
types allow you to dispatch on numbers/sequences/arrays that don't necessarily have distinct classes
19:13:58
aeth
since you could e.g. (simple-array octet (* *)) as a type for 2D (unsigned-byte 8) arrays (assuming octet is defined as such)
19:14:22
aeth
jcowan: I wouldn't be surprised if there are other distinct generic function systems, too, perhaps more close to what you're looking for
19:15:15
jcowan
Clojure allows uou to create independent class hierarchies, but I can find no explanation of why you'd want to.
19:16:50
aeth
and perhaps you can't do that, if it's a well-specified language in a way that's incompatible with the host language
19:17:13
aeth
but that way at least you can use built-in generics even though you can't use the built-in class hierarchy
19:24:32
pjb
jcowan: independent class hierarchies are usually an illusion, there's normally a TOP class (eg. T in CLOS).
19:26:20
pjb
jcowan: that said, in CLOS, playing with metaclasses, you can probably implement separate subhierarchies according to your wishes.