libera/#commonlisp - IRC Chatlog
Search
6:34:49
phantomics
recordgroovy: Looks interesting at a glance, very neat code. I'll give it a try when I have time
7:25:25
pjb
lotuseater: I've seen big systems defining some dispatching reader macros on $ ; but nowadays, with unicode, you can more often just define a reader macro on a unicode character.
7:46:26
kakuhen
Is it possible for eval-when to implicitly modify arguments of functions like find-class?
7:47:18
kakuhen
I have a macro that transforms a given symbol and creates a class named with this transformed symbol, together with methods specialized on the class.
7:48:20
kakuhen
Say this transformed symbol is X. Both CCL and SBCL throw style warnings that my defmethod's are using the undefined type X, but the class was defined right before I define these methods.
7:49:53
pjb
kakuhen: not eval-when directly, but eval-when implies that the body will be evaluated in different environments, so it's possible that operators that may depend on the environment such as find-class give different results.
7:50:02
kakuhen
So I decided to wrap (defclass X () ...) in an eval-when form. Above in the macro, I check the output (find-class X nil). When I use eval-when, errorp somehow becomes T rather than nil
7:50:09
pjb
It should be the job of the programmer to ensure that the differences are not semantically different.
7:51:10
kakuhen
pjb: Yeah, I suspect eval-when implying a different environment somehow makes the arguments supplied to FIND-CLASS "flip," but it wouldn't explain why this issues goes away when I supply the environment.
7:51:49
kakuhen
anyway, I am trying this because SBCL and CCL are giving me style warnings (ECL doesn't give style warnings), and I assume it could be a bug in both SBCL and CCL's compiler, but I don't want to jump to such conclusions yet.
7:52:14
kakuhen
and I get strange behavior with FIND-CLASS when I include an (eval-when (:compile-toplevel :load-toplevel) ...) in the macro
7:52:26
pjb
You must be doing somethign strange, because a simple toplevel sequence of defclass defmethod should work nicely.
7:53:19
kakuhen
Anyway, I am trying to produce a minimal working example of the style warnings SBCL and CCL produce
7:53:33
kakuhen
SBCL produces twice as many style warnings as CCL, so it's catching(?) something that CCL isn't. Meanwhile ECL just gives zero style warnings.
7:53:48
pjb
kakuhen: so, your macro should expand to (progn (defclass x () ()) (defmethod moo ((x x)) …))
7:55:01
beach
kakuhen: The dictionary entry for DEFCLASS contains a phrase that makes its compile-time behavior very hard to understand.
7:55:37
beach
That might be the reason for the differences in behavior of different implementations.
7:56:23
beach
It is essentially impossible for an implementation to comply with what the standard says.
7:56:28
kakuhen
I see. And, before it gets asked, yes, I set the debug level to 3 before compiling on each implementation.
7:56:38
pjb
"If a defclass form appears as a top level form, the compiler must make the class name be recognized as a valid type name in subsequent declarations (as for deftype) and be recognized as a valid class name for defmethod parameter specializers and for use as the :metaclass option of a subsequent defclass. The compiler must make the class definition available to be returned by find-class when its environment argument is a value received
7:57:57
pjb
kakuhen: But I've seen nothing that implies a need for eval-when in your macro expansion. Just use progn around your defclass and defmethod forms.
7:57:58
kakuhen
The style warnings disappear on both SBCL and CCL once I use (find-class X nil nil) and put the defclass in this eval-when block(?)
7:58:26
kakuhen
but I found an interesting behavior where ERRORP automatically gets set to T (on both sbcl and ccl!) if I just have (find-class X nil) written, and the defclass is in this eval-when block
7:58:47
kakuhen
this behavior mysteriously goes away if I do one of two things: remove the eval-when block, or explicitly provide the environment
7:59:24
kakuhen
I'm trying to produce a minimal working example but so far I haven't been able to reproduce the style warnings, yet.
8:00:43
beach
I would be interested in complete examples of what happens in different implementations and what you expected to happen. But take your time.
8:05:11
beach
kakuhen: It is not enough to say that the definition is in an "eval-when block". The "situations" given in that form are important too.
8:09:46
kakuhen
Ok this will sound really dumb, but I have a macro %FOO and a macro FOO that calls %FOO with a restart-case
8:10:07
kakuhen
Calling (%FOO bar) does not give the undefined type warning. Calling (FOO bar) does.
8:11:35
kakuhen
i understand the importance, but i also want to minimize irrelevant information -- you'll know i've given up in attempting to produce a minimal example when i just send the entire lisp file along with the repl output for each implementation.
8:14:01
beach
You also need to show when the behavior you observe happens, like if it is at compile time, load time, or run time.
8:16:31
beach
lotuseater: Wow, hold on! First errors are not "thrown", they are "signaled". Second, it is not typically the debugger that signals the error, but the application code, which will call the debugger if the error is not handled. Third, no, the same error can be signaled in all three situations.
8:20:21
kakuhen
Expected behavior: I expect the new class' corresponding type to be defined before we define the method specializing on the new class.
8:20:54
kakuhen
ECL does not give any style warnings, but I will nonetheless give output for it soon.
8:23:08
kakuhen
These style warnings disappear when the defclass is compiled and loaded at the top level, that is, we place it inside (eval-when (:compile-toplevel :load-toplevel :execute) ...)
8:23:34
kakuhen
But this opens up a can of worms where FIND-CLASS suddenly sets ERRORP to T despite supplying NIL to it. This goes away if you explicitly provide NIL for both ERRORP and ENVIRONMENT
8:28:31
beach
kakuhen: I see no trace of the argument to FIND-CLASS having been altered. I see essentially the same behavior, i.e. that the class is not recognized. And I think it will take me some time to figure out why. Others here are usually much faster.
8:30:35
kakuhen
Yes. And it seemed fine to me. It was invoking FIND-CLASS exactly as I intended, yet ERRORP would be set to T
8:30:43
beach
kakuhen: And, again, your examples don't show how you compiled and/or loaded the file containing the macro definitions.
8:31:02
kakuhen
ah... I'm running C-c C-c in SLY, and I denoted in the file where I have compiled both
8:35:01
beach
And what makes you think that FIND-CLASS should be invoked as you intended, rather than as the system decided to invoke it?
8:36:55
kakuhen
beach: here is a working example where FIND-CLASS mysterious gets ERRORP set to T http://0x0.st/-WyX.lisp
8:37:04
kakuhen
I will post the output from CCL soon, and this time I am compiling and loading at the REPL.
8:40:31
kakuhen
CCL output http://0x0.st/-WyK.txt and the debugger with a full backtrace http://0x0.st/-Wy8.txt
8:40:59
kakuhen
beach: in the latter link, you will see immediately in the backtrace (FIND-CLASS BAR* T NIL)
8:41:14
kakuhen
I expect (FIND-CLASS BAR* NIL NIL) to be supplied since my macro calls (FIND-CLAS BAR* NIL)
8:42:05
beach
Yes, but I have no reason to believe that this error is a result of your own call to find-class.
8:47:07
kakuhen
Okay, I guess that settles the issue with the FIND-CLASS call. This convinced me that my earlier interpretation is incorrect.
8:50:11
beach
Even if your call to FIND-CLASS is evaluated before the defmethod form, you would not see any output from your find-class call.
8:50:12
kakuhen
I would like to correct myself on what I said at 01:23 PDT. The FIND-CLASS call that ultimately signals an error does not occur when I specify (:COMPILE-TOPLEVEL :LOAD-TOPLEVEL :EXECUTE) in my eval-when block. It occurs when I am missing the :EXECUTE symbol
8:51:07
kakuhen
So this error is more due to myself misusing / not understanding what happens when I use EVAL-WHEN with specific arguments
8:51:40
beach
Also, every new attempt must be done in a fresh image, or else, you may have altered the global state of your environment in previous attempts.
8:52:29
kakuhen
Yes. When I made the separate examples, I made sure to restart my lisp image and delete the FASLs generated by the compiler
9:06:45
kakuhen
lotuseater: The reason I have a macro that calls %foo is so that I can pass symbols as arguments. I originally had FOO as a function in my actual code, but ALEXANDRIA:SYMBOLICATE would then complain that I supplied neither a symbol nor a string.
9:07:26
beach
Anyway, that backtrace that confused you reminded me of a rule I made for SICL system code (but that I have not had the energy to implement entirely yet), namely that if a standard operator A calls another standard operator B, then A should make sure that B succeeds or else capture any errors by B so that the ultimate error is signaled in terms of A.
9:08:18
beach
If that rule had been followed here, you would not have seen a call to FIND-CLASS in the backtrace.
9:09:34
beach
kakuhen: It would be much preferable if you could eliminate a lot of extraneous stuff to provide a minimal example. Those macro combinations, and the call to Alexandria make it harder to follow.
9:11:01
kakuhen
What I provided in http://0x0.st/-Wy-.lisp is enough to reproduce the warning, but I will attempt removing the call to Alexandria right now. Unfortunately, I am unable to reproduce the style warnings unless I have those macro combinations. The style warnings happen when FOO is called, but not when %FOO itself is called.
9:12:52
kakuhen
Yes. In my original code, there is a scenario where a condition I made gets signalled, and the user has two choices to restart.
9:14:34
beach
So, this example again shows that support for debugging is not great, at least in the free Common Lisp implementations.
9:15:07
kakuhen
beach: Unfortunately, the style warnings do not appear without the RESTART-CASE macro.
9:15:41
kakuhen
So perhaps there is something about that macro that causes this. Interestingly enough, ECL does not complain at all. It's just CCL that complains, and SBCL complains even louder by giving twice as many style warnings as CCL.
9:17:22
pjb
beach: note: this rule about where to signal or re-signal errors from, should be different when you are debugging the implementation. often I have code that will use either handler-case or handler-bind (notably in tests) depending on what you want to debug (eg. the tested code, or the tests).
9:18:54
pjb
I understand. But when you implement the rule, you need to make it optional because the code and conditions are also used by implementers.
9:20:50
beach
Also, I would like to see the backtrace not mention the particular operator that was invoked, because the operator that failed could be the result of a macro expansion, and that would be (is?) confusing to the application programmer.
9:21:20
kakuhen
pjb: hmm... on my side I am getting warnings, though, even when I run the REPL in a terminal rather than SLY
9:21:36
beach
Instead, the tentative plan is to show the source form (as highlighted text in the source code) that caused the problem.
9:31:53
pjb
kakuhen: I use (proclaim '(optimize (safety 3) (debug 3) (space 0) (speed 0) (compilation-speed 3)))
9:33:37
kakuhen
I'm starting to think backquoting restart-case is just giving me more trouble than what it's worth
9:37:04
pjb
kakuhen: https://termbin.com/xr0o2 ; probably the warning occurs when the implementations tries to perform compilation-time optimization that would require the class to be defined, which it cannot do without evaluating the defclass that is in the progn. It cannot do that because it's in a single progn form; but should be able to do, because it's a toplevel progn form, so it could be spliced out.
9:37:52
hayley
Is there much I can do to make it more likely that I can handle a STORAGE-CONDITION (for running out of memory) rather than crashing the Lisp implementation?
9:38:18
kakuhen
pjb: one of my friends speculated this, since sbcl and ccl apparently share some common history with their compilers, and both have very similar behavior when running this macro, yet ECL didn't
9:38:29
hayley
For example, (handler-case (loop collect 2) (storage-condition () 'no)) crashes into LDB rather than signalling on SBCL.
9:38:41
pjb
hayley: one trick could be to allocate some vector, and when you get this storage-condition, to release the last reference to that vector to free some space.
9:39:00
pjb
hayley: that should require a minimum of space to perform, but the question is whether it will be enough to continue.
9:39:59
pjb
hayley: somebody did an evaluation of the handling of storage-condition in various implementation when usenet cll was still a thing, but nothing moved since.
9:40:17
pjb
hayley: somebody would have to provide patches to the implementations. So far it has been easier to just buy more RAM.
9:41:09
hayley
pjb: My current "threat model" is that we are overwhelmed with messages on a server, and so we run out of memory allocating objects for them all. To handle it, we would probably kill threads which try to run out of memory.
9:41:15
pjb
One problem is that the storage-condition handler may require some memory to work. If the heap is already overextended, it may be difficult. So the implementation should reserve some space for the handlers in this situation.
9:42:05
hayley
Given that such message objects would be large, maybe with large strings or byte vectors, it seems I would have a good chance on SBCL.
9:45:37
flip214
hayley: well, you could mmap() your message files and have your OS swap them in/out as needed?
9:46:09
flip214
might not be as easy to handle as strings or ub8 vectors... don't know what you need, though.
9:48:51
hayley
I already have a message size limit, but one could just send more messages at that size limit on multiple connections (and thus threads; I guess every other multiplexing model looks a bit nicer with this problem) and blow the heap that way.
9:50:05
flip214
so if you know how much heap you need per message, and the size of your heap, you can restrict the number of parallel threads and be done
9:50:53
hayley
Right. I don't think I know how much heap I need per message, and it would hurt my head too much to use only TCP buffers.
9:51:09
moon-child
I would change the variable there and say, rather, you know how much heap you need per message, and the number of cores your machine has; so you know the size your heap needs to be
9:51:24
moon-child
hayley: you may not know exactly, but it should not be overly difficult to come up with a conservative estimate, no?
9:52:34
hayley
I don't know what my server will be used for, and so I could only say something between 10² and 10⁷ bytes per message?
9:57:29
hayley
So, taking the maximum limits the throughput substantially if that is not the case, but a more conservative estimate would be too conservative. Hence why I would like a more...I suppose "feedback"-driven approach?
9:57:44
moon-child
I don't think this kind of probabilistic approaches to security is the right one. Is it better that you have (say) an 80% chance of signalling a condition than a 5% chance? Marginally. But if the implementation is unwilling to _reliably_ signal such a condition, I think it is better to accept that as a constraint and look for an alternative solution (or, patch the implementation, as pjb
9:59:02
hayley
I would consider it to be a performance hack, as there is no security (or progress, for that matter) lost if the server crashes outright, but I would prefer to only kill one connection rather than all of them.
9:59:10
moon-child
for instance, separate your server into multiple processes and use the operating system to coordinate them
9:59:21
flip214
hayley: run (TIME (process-one-message)) - the output will tell you how much heap was allocated. round up, and estimate.
10:00:11
flip214
also, you can have a background thread monitoring heap usage - but then you'll need to find out which thread to kill...
10:01:16
flip214
depending on your usecase (needs IO, or not) it might be easier to just limit number of thread == number of cores. then every thread can process a message without any wait time => minimum latency, minimum heap usage.
10:01:45
flip214
works for me with PDF generation - there's no IO involved, just CPU, so there's no reason to have more threads than can run at the same time.
10:03:20
moon-child
I think a server must of necessity do a lot of i/o. But I think the usual approach there is to just have one i/o thread which receives connections and pushes them onto a work queue; and to do writing asynchronously
10:04:11
hayley
I can't seem to understand anything that isn't "straight line" networking code with threads, such as using some callback async library or polling or whatnot. Then writing it is daunting.
10:06:13
flip214
hayley: SB-EXT:CALL-WITH-TIMING can give you a :BYTES-CONSED result, so your threads can (over time) estimate their memory use themselves ;)
10:07:03
flip214
moon-child: I meant disk-IO that would block worker threads. as soon as an HTTP request has arrived, all the work until pushing the result might be possible without any IO.
10:07:38
moon-child
ah, yes. Usual approach for this is coroutines, but I don't think we have any cl implementations that support them
10:08:22
moon-child
(of course, come closos, plain threads will be cheap enough it won't matter. But who knows then that will be)
10:08:45
hayley
Another difference is that I have connections which are expected to be persistent, whereas I can't remember if Hunchentoot supports keep-alive.
10:10:13
flip214
hayley: you can tell HT to ignore a TCP socket - and pass that on to an extra thread.
10:11:27
hayley
moon-child: FWIW my secret plan is to prototype "green threads" in a fork of SICL, which would then be somewhat reusable for a thread implementation in CLOSOS.
10:13:28
flip214
hayley: green threads - with all the features of OS threads? like scheduling priorities, blocking behaviour for OS calls, timeouts, ...??
10:14:03
flip214
I've worked on such a library in C - but it never got up to par on features with kernel threads.
10:16:21
hayley
Well, Erlang has priorities, and in the case of CLOSOS, blocking OS calls could be handled with the usual devices (mostly semaphores I suppose).
10:18:36
hayley
On a Unix system, I think the Erlang and Haskell people both either attempt to use asynchronous operations or a thread pool for synchronous operations, which still use something like mailboxes to unblock the appropriate thread and provide the result.
10:22:32
hayley
Apparently the fastest servers these days use other async stuff, which is backed by some polling technique (epoll, etc, maybe io_uring if you're fancy). Without some intermediate stuff they don't work too well with threads apparently.
10:24:43
flip214
yeah. But if your actual workload (PDFs, for me) take 30-60msec to generate, and I only have a handful of cores (and a handful of threads), the server implementation doesn't actually matter.
10:25:10
flip214
of course, the "hello world ~a" examples need to keep the server as small as possible for their benchmarks...
12:23:23
yitzi
beach: Thanks for the paper reference regarding CL and SICL debugging. Very informative.
12:24:42
yitzi
I noticed that there wasn't a section on CMUCL. From what the user manual and repo says they support stepping and breakpoints, although the stepper was broken until recently.
12:27:14
yitzi
Not sure how well maintained it is though. I can't currently test it in my environment since CMUCL can't load ironclad on my system.
12:28:25
jackdaniel
flip214: kernel threads become troublesome when there are a lot of them (the context switch of a kernel thread is more expensive)
12:29:41
yitzi
beach: If you are curious you can run the Jupyter debugger stuff here: https://mybinder.org/v2/gh/yitzchak/common-lisp-jupyter/add-debugging?urlpath=lab
12:29:42
jackdaniel
also green threads (when pooled in a single native thread) have assured a certain amount of atomicity
12:30:21
jackdaniel
a side-benefit is that if you bother to implement green threads you will probably have delimited continuations for free
12:30:41
jackdaniel
(and with that come numerous interesting programming techniques - i.e generators)
12:31:43
flip214
and with _lots_ of kernel threads you run in all the (already-solved, on the kernel-side) scheduling things - O(1), priorities, fairness, ...
12:37:18
hayley
jackdaniel: I thought of green threads in the terms of undelimited one shot continuations.
12:40:14
yitzi
beach: In addition to debugging, stepping, source and variable inspection there is also symbol completion and help/inspect. Along with a widgets and pile of other stuff.
12:41:32
yitzi
The debugging is still a work in progress. The other stuff is pretty well tested and used by drmeister's group. There is also a maxima-jupyter kernel available based on the same code.
12:44:40
yitzi
beach: Yep. I've already started tweaking some of the debug messaging based on some your ideas. Thanks again!
12:46:15
hayley
I read a delimited continuation was a continuation which was "delimited" to some frame with shift/reset.
12:48:04
jackdaniel
in non-scheme terms, a delimited continuation is a computing context that may be resumed, and after resuming it may yield control again (and be resumed again)
12:48:30
Bike
"one shot" means you can only use it once, like because the stack isn't copied so it will no longer be in place if you try to continue there later. so register saving works.
12:49:56
hayley
I /guess/ we are delimited by THREAD-YIELD (and preemptive scheduling) but it doesn't fit how they are presented in Scheme land in my opinion.
12:52:02
Bike
i don't think threads are really "delimited continuations". the way the continuations given by shift work, they add to the current call stack, so they return like normal functions (to muddle several levels of description)
12:52:05
jackdaniel
so if the stack is not copied and you allow yielding, then how does it work? all threads share the same stack?
12:53:43
hayley
Each thread has a separate stack, and yielding saves state, returns to some scheduler, then loads the new state.
12:53:53
jackdaniel
I didn't say that delimited continuations are threads, I'm saying that cooperative threads may be implemented with delimited continuations
12:55:05
hayley
But my observation was the only green thread implementation with continuations uses undelimited continuations once, as I couldn't spot the delimiting.
12:55:37
hayley
Somehow delimited continuations are harder for me to understand than undelimited even though the former is supposedly much better to use.
12:56:22
hayley
The former does some stack knotting, whereas the latter is a funny first class GOTO.
12:56:22
Bike
they have some interesting advantages over undelimited, but simplicity is not one of them
12:58:00
jackdaniel
I will read on that later, but some loose searching seems to indicate, that one-shot continuations are a certain class of delimited continuations
13:00:27
Bike
and most hagiographies of delimited continuations emphasize their composability which kind of goes with reusability
13:00:28
jackdaniel
continuations that can be called multiple times are full continuations, one-shot may be resumed only once and delimited capture the entire /remaining/ continuation; that's what I've grasped from this quick search
13:01:12
hayley
I think get/setcontext in POSIX-2001 manipulate one shot undelimited continuations, as they are just all the registers as described previously.
13:03:23
hayley
Before I was going to say setjmp/longjmp but those are mere escape continuations as I understand how you are supposed to use them.
13:04:47
Bike
yeah longjmp has the "UB if the function that used setjmp has returned" bit, so they're only an escape
13:42:03
rain3
a 2 hours long video from Atlanta Functional Programming - Common Lisp Study Group youtube channel can be condensed in 1-5 pages of good old text tutorial which in turn can be studied in 10-30 minutes instead of 2-6 hours of listening and watching and pausing and scrolling
13:42:51
rain3
interesting stuff there https://www.youtube.com/watch?v=tcmY7zstig4 but it takes too long to watch a 2h long video for only 10 minutes of essential info
13:45:48
jackdaniel
so while the former purpose is teaching, the latter purpose is socializing (in my understanding)
13:46:49
rain3
socializing around good content which can be summarized as a high quality tutorial , either before or after the meeting
13:48:38
jackdaniel
ah, that's a shame; perhaps you could write a high-quality tutorial or a paper on another topic related to lisp; I'd gladly read it
13:49:08
lotuseater
I skipped through it, is ":filter :something" a method-combination? wasn't aware they can be done that way, but I think because I didn't define ones myself yet
13:50:02
ldb
lotuseater: that's probably optional argument lists, I don't think you need method-combination for that
13:50:49
rain3
jackdaniel: on the other hand your CLIM videos are from beginning to end golden and enjoyable, good and efficient for teaching others
13:51:34
lotuseater
they have something like (defmethod stack-push :filter :number ((stack (eql 'normal))) ...)
13:55:11
jackdaniel
that's nice to hear; either way preparing a tutorial after giving a talk is an extra work - if someone does that it is fine; but expecting it from that person is a bit much
13:57:06
rain3
I am not asking for it.. I am only pointing out that the same thing can be done more efficiently . If you have the tutorial prepared before, then you don't need to waste time thinking and fighting with bugs 'sorry, it didn't work, we'll try again in the next video'
14:51:05
pjb
rain3: totally agree, videos are two slow in general. However it is not necessarily a bad thing: 1- you can listen to them in parallel to some other task. 2- sometimes you're not working at 100% of your capacities yourself, so it's not so bad they're a little slow. 3- it can give you an excuse for such a slow day.
14:51:41
pjb
(and you can always play them at 2x, even if one may sometimes regret there's not an option to play them at 10x).
14:52:59
pjb
Perhaps somebody at google will implement an AI to summarize a video, and transform it into a multimedia stream that could be absorbed by brains in high speed. Or perhaps we'll have to wait for neuralink to provide the brain downloading feature…
14:53:11
rain3
the video above actually was quite good in comparison with others. I've downloaded them so that I can play at 10x with vlc player , then I deleted all of those which only had examples from the readme of the lib