libera/#commonlisp - IRC Chatlog
Search
1:33:06
recordgroovy
Hi, I'm wondering if anyone could spare some time to peer-review a library I've been working on. Thanks for any comments :)
1:49:28
moon-child
recordgroovy: I am not so sure of the special-casing of eof. Usually it makes more sense to treat the eof as just another kind of token
6:34:49
phantomics
recordgroovy: Looks interesting at a glance, very neat code. I'll give it a try when I have time
7:25:25
pjb
lotuseater: I've seen big systems defining some dispatching reader macros on $ ; but nowadays, with unicode, you can more often just define a reader macro on a unicode character.
7:46:26
kakuhen
Is it possible for eval-when to implicitly modify arguments of functions like find-class?
7:47:18
kakuhen
I have a macro that transforms a given symbol and creates a class named with this transformed symbol, together with methods specialized on the class.
7:48:20
kakuhen
Say this transformed symbol is X. Both CCL and SBCL throw style warnings that my defmethod's are using the undefined type X, but the class was defined right before I define these methods.
7:49:53
pjb
kakuhen: not eval-when directly, but eval-when implies that the body will be evaluated in different environments, so it's possible that operators that may depend on the environment such as find-class give different results.
7:50:02
kakuhen
So I decided to wrap (defclass X () ...) in an eval-when form. Above in the macro, I check the output (find-class X nil). When I use eval-when, errorp somehow becomes T rather than nil
7:50:09
pjb
It should be the job of the programmer to ensure that the differences are not semantically different.
7:51:10
kakuhen
pjb: Yeah, I suspect eval-when implying a different environment somehow makes the arguments supplied to FIND-CLASS "flip," but it wouldn't explain why this issues goes away when I supply the environment.
7:51:49
kakuhen
anyway, I am trying this because SBCL and CCL are giving me style warnings (ECL doesn't give style warnings), and I assume it could be a bug in both SBCL and CCL's compiler, but I don't want to jump to such conclusions yet.
7:52:14
kakuhen
and I get strange behavior with FIND-CLASS when I include an (eval-when (:compile-toplevel :load-toplevel) ...) in the macro
7:52:26
pjb
You must be doing somethign strange, because a simple toplevel sequence of defclass defmethod should work nicely.
7:53:19
kakuhen
Anyway, I am trying to produce a minimal working example of the style warnings SBCL and CCL produce
7:53:33
kakuhen
SBCL produces twice as many style warnings as CCL, so it's catching(?) something that CCL isn't. Meanwhile ECL just gives zero style warnings.
7:53:48
pjb
kakuhen: so, your macro should expand to (progn (defclass x () ()) (defmethod moo ((x x)) …))
7:55:01
beach
kakuhen: The dictionary entry for DEFCLASS contains a phrase that makes its compile-time behavior very hard to understand.
7:55:37
beach
That might be the reason for the differences in behavior of different implementations.
7:56:23
beach
It is essentially impossible for an implementation to comply with what the standard says.
7:56:28
kakuhen
I see. And, before it gets asked, yes, I set the debug level to 3 before compiling on each implementation.
7:56:38
pjb
"If a defclass form appears as a top level form, the compiler must make the class name be recognized as a valid type name in subsequent declarations (as for deftype) and be recognized as a valid class name for defmethod parameter specializers and for use as the :metaclass option of a subsequent defclass. The compiler must make the class definition available to be returned by find-class when its environment argument is a value received
7:57:57
pjb
kakuhen: But I've seen nothing that implies a need for eval-when in your macro expansion. Just use progn around your defclass and defmethod forms.
7:57:58
kakuhen
The style warnings disappear on both SBCL and CCL once I use (find-class X nil nil) and put the defclass in this eval-when block(?)
7:58:26
kakuhen
but I found an interesting behavior where ERRORP automatically gets set to T (on both sbcl and ccl!) if I just have (find-class X nil) written, and the defclass is in this eval-when block
7:58:47
kakuhen
this behavior mysteriously goes away if I do one of two things: remove the eval-when block, or explicitly provide the environment
7:59:24
kakuhen
I'm trying to produce a minimal working example but so far I haven't been able to reproduce the style warnings, yet.
8:00:43
beach
I would be interested in complete examples of what happens in different implementations and what you expected to happen. But take your time.
8:05:11
beach
kakuhen: It is not enough to say that the definition is in an "eval-when block". The "situations" given in that form are important too.
8:09:46
kakuhen
Ok this will sound really dumb, but I have a macro %FOO and a macro FOO that calls %FOO with a restart-case
8:10:07
kakuhen
Calling (%FOO bar) does not give the undefined type warning. Calling (FOO bar) does.
8:11:35
kakuhen
i understand the importance, but i also want to minimize irrelevant information -- you'll know i've given up in attempting to produce a minimal example when i just send the entire lisp file along with the repl output for each implementation.
8:14:01
beach
You also need to show when the behavior you observe happens, like if it is at compile time, load time, or run time.
8:16:31
beach
lotuseater: Wow, hold on! First errors are not "thrown", they are "signaled". Second, it is not typically the debugger that signals the error, but the application code, which will call the debugger if the error is not handled. Third, no, the same error can be signaled in all three situations.
8:20:21
kakuhen
Expected behavior: I expect the new class' corresponding type to be defined before we define the method specializing on the new class.
8:20:54
kakuhen
ECL does not give any style warnings, but I will nonetheless give output for it soon.
8:23:08
kakuhen
These style warnings disappear when the defclass is compiled and loaded at the top level, that is, we place it inside (eval-when (:compile-toplevel :load-toplevel :execute) ...)
8:23:34
kakuhen
But this opens up a can of worms where FIND-CLASS suddenly sets ERRORP to T despite supplying NIL to it. This goes away if you explicitly provide NIL for both ERRORP and ENVIRONMENT
8:28:31
beach
kakuhen: I see no trace of the argument to FIND-CLASS having been altered. I see essentially the same behavior, i.e. that the class is not recognized. And I think it will take me some time to figure out why. Others here are usually much faster.
8:30:35
kakuhen
Yes. And it seemed fine to me. It was invoking FIND-CLASS exactly as I intended, yet ERRORP would be set to T
8:30:43
beach
kakuhen: And, again, your examples don't show how you compiled and/or loaded the file containing the macro definitions.
8:31:02
kakuhen
ah... I'm running C-c C-c in SLY, and I denoted in the file where I have compiled both
8:35:01
beach
And what makes you think that FIND-CLASS should be invoked as you intended, rather than as the system decided to invoke it?
8:36:55
kakuhen
beach: here is a working example where FIND-CLASS mysterious gets ERRORP set to T http://0x0.st/-WyX.lisp
8:37:04
kakuhen
I will post the output from CCL soon, and this time I am compiling and loading at the REPL.
8:40:31
kakuhen
CCL output http://0x0.st/-WyK.txt and the debugger with a full backtrace http://0x0.st/-Wy8.txt
8:40:59
kakuhen
beach: in the latter link, you will see immediately in the backtrace (FIND-CLASS BAR* T NIL)
8:41:14
kakuhen
I expect (FIND-CLASS BAR* NIL NIL) to be supplied since my macro calls (FIND-CLAS BAR* NIL)
8:42:05
beach
Yes, but I have no reason to believe that this error is a result of your own call to find-class.
8:47:07
kakuhen
Okay, I guess that settles the issue with the FIND-CLASS call. This convinced me that my earlier interpretation is incorrect.
8:50:11
beach
Even if your call to FIND-CLASS is evaluated before the defmethod form, you would not see any output from your find-class call.
8:50:12
kakuhen
I would like to correct myself on what I said at 01:23 PDT. The FIND-CLASS call that ultimately signals an error does not occur when I specify (:COMPILE-TOPLEVEL :LOAD-TOPLEVEL :EXECUTE) in my eval-when block. It occurs when I am missing the :EXECUTE symbol
8:51:07
kakuhen
So this error is more due to myself misusing / not understanding what happens when I use EVAL-WHEN with specific arguments
8:51:40
beach
Also, every new attempt must be done in a fresh image, or else, you may have altered the global state of your environment in previous attempts.
8:52:29
kakuhen
Yes. When I made the separate examples, I made sure to restart my lisp image and delete the FASLs generated by the compiler
9:06:45
kakuhen
lotuseater: The reason I have a macro that calls %foo is so that I can pass symbols as arguments. I originally had FOO as a function in my actual code, but ALEXANDRIA:SYMBOLICATE would then complain that I supplied neither a symbol nor a string.
9:07:26
beach
Anyway, that backtrace that confused you reminded me of a rule I made for SICL system code (but that I have not had the energy to implement entirely yet), namely that if a standard operator A calls another standard operator B, then A should make sure that B succeeds or else capture any errors by B so that the ultimate error is signaled in terms of A.
9:08:18
beach
If that rule had been followed here, you would not have seen a call to FIND-CLASS in the backtrace.
9:09:34
beach
kakuhen: It would be much preferable if you could eliminate a lot of extraneous stuff to provide a minimal example. Those macro combinations, and the call to Alexandria make it harder to follow.
9:11:01
kakuhen
What I provided in http://0x0.st/-Wy-.lisp is enough to reproduce the warning, but I will attempt removing the call to Alexandria right now. Unfortunately, I am unable to reproduce the style warnings unless I have those macro combinations. The style warnings happen when FOO is called, but not when %FOO itself is called.
9:12:52
kakuhen
Yes. In my original code, there is a scenario where a condition I made gets signalled, and the user has two choices to restart.
9:14:34
beach
So, this example again shows that support for debugging is not great, at least in the free Common Lisp implementations.
9:15:07
kakuhen
beach: Unfortunately, the style warnings do not appear without the RESTART-CASE macro.
9:15:41
kakuhen
So perhaps there is something about that macro that causes this. Interestingly enough, ECL does not complain at all. It's just CCL that complains, and SBCL complains even louder by giving twice as many style warnings as CCL.
9:17:22
pjb
beach: note: this rule about where to signal or re-signal errors from, should be different when you are debugging the implementation. often I have code that will use either handler-case or handler-bind (notably in tests) depending on what you want to debug (eg. the tested code, or the tests).
9:18:54
pjb
I understand. But when you implement the rule, you need to make it optional because the code and conditions are also used by implementers.
9:20:50
beach
Also, I would like to see the backtrace not mention the particular operator that was invoked, because the operator that failed could be the result of a macro expansion, and that would be (is?) confusing to the application programmer.
9:21:20
kakuhen
pjb: hmm... on my side I am getting warnings, though, even when I run the REPL in a terminal rather than SLY
9:21:36
beach
Instead, the tentative plan is to show the source form (as highlighted text in the source code) that caused the problem.
9:31:53
pjb
kakuhen: I use (proclaim '(optimize (safety 3) (debug 3) (space 0) (speed 0) (compilation-speed 3)))
9:33:37
kakuhen
I'm starting to think backquoting restart-case is just giving me more trouble than what it's worth
9:37:04
pjb
kakuhen: https://termbin.com/xr0o2 ; probably the warning occurs when the implementations tries to perform compilation-time optimization that would require the class to be defined, which it cannot do without evaluating the defclass that is in the progn. It cannot do that because it's in a single progn form; but should be able to do, because it's a toplevel progn form, so it could be spliced out.
9:37:52
hayley
Is there much I can do to make it more likely that I can handle a STORAGE-CONDITION (for running out of memory) rather than crashing the Lisp implementation?
9:38:18
kakuhen
pjb: one of my friends speculated this, since sbcl and ccl apparently share some common history with their compilers, and both have very similar behavior when running this macro, yet ECL didn't
9:38:29
hayley
For example, (handler-case (loop collect 2) (storage-condition () 'no)) crashes into LDB rather than signalling on SBCL.
9:38:41
pjb
hayley: one trick could be to allocate some vector, and when you get this storage-condition, to release the last reference to that vector to free some space.
9:39:00
pjb
hayley: that should require a minimum of space to perform, but the question is whether it will be enough to continue.
9:39:59
pjb
hayley: somebody did an evaluation of the handling of storage-condition in various implementation when usenet cll was still a thing, but nothing moved since.
9:40:17
pjb
hayley: somebody would have to provide patches to the implementations. So far it has been easier to just buy more RAM.
9:41:09
hayley
pjb: My current "threat model" is that we are overwhelmed with messages on a server, and so we run out of memory allocating objects for them all. To handle it, we would probably kill threads which try to run out of memory.
9:41:15
pjb
One problem is that the storage-condition handler may require some memory to work. If the heap is already overextended, it may be difficult. So the implementation should reserve some space for the handlers in this situation.
9:42:05
hayley
Given that such message objects would be large, maybe with large strings or byte vectors, it seems I would have a good chance on SBCL.
9:45:37
flip214
hayley: well, you could mmap() your message files and have your OS swap them in/out as needed?
9:46:09
flip214
might not be as easy to handle as strings or ub8 vectors... don't know what you need, though.
9:48:51
hayley
I already have a message size limit, but one could just send more messages at that size limit on multiple connections (and thus threads; I guess every other multiplexing model looks a bit nicer with this problem) and blow the heap that way.
9:50:05
flip214
so if you know how much heap you need per message, and the size of your heap, you can restrict the number of parallel threads and be done
9:50:53
hayley
Right. I don't think I know how much heap I need per message, and it would hurt my head too much to use only TCP buffers.
9:51:09
moon-child
I would change the variable there and say, rather, you know how much heap you need per message, and the number of cores your machine has; so you know the size your heap needs to be
9:51:24
moon-child
hayley: you may not know exactly, but it should not be overly difficult to come up with a conservative estimate, no?
9:52:34
hayley
I don't know what my server will be used for, and so I could only say something between 10² and 10⁷ bytes per message?
9:57:29
hayley
So, taking the maximum limits the throughput substantially if that is not the case, but a more conservative estimate would be too conservative. Hence why I would like a more...I suppose "feedback"-driven approach?
9:57:44
moon-child
I don't think this kind of probabilistic approaches to security is the right one. Is it better that you have (say) an 80% chance of signalling a condition than a 5% chance? Marginally. But if the implementation is unwilling to _reliably_ signal such a condition, I think it is better to accept that as a constraint and look for an alternative solution (or, patch the implementation, as pjb
9:59:02
hayley
I would consider it to be a performance hack, as there is no security (or progress, for that matter) lost if the server crashes outright, but I would prefer to only kill one connection rather than all of them.
9:59:10
moon-child
for instance, separate your server into multiple processes and use the operating system to coordinate them
9:59:21
flip214
hayley: run (TIME (process-one-message)) - the output will tell you how much heap was allocated. round up, and estimate.
10:00:11
flip214
also, you can have a background thread monitoring heap usage - but then you'll need to find out which thread to kill...
10:01:16
flip214
depending on your usecase (needs IO, or not) it might be easier to just limit number of thread == number of cores. then every thread can process a message without any wait time => minimum latency, minimum heap usage.
10:01:45
flip214
works for me with PDF generation - there's no IO involved, just CPU, so there's no reason to have more threads than can run at the same time.
10:03:20
moon-child
I think a server must of necessity do a lot of i/o. But I think the usual approach there is to just have one i/o thread which receives connections and pushes them onto a work queue; and to do writing asynchronously
10:04:11
hayley
I can't seem to understand anything that isn't "straight line" networking code with threads, such as using some callback async library or polling or whatnot. Then writing it is daunting.
10:06:13
flip214
hayley: SB-EXT:CALL-WITH-TIMING can give you a :BYTES-CONSED result, so your threads can (over time) estimate their memory use themselves ;)
10:07:03
flip214
moon-child: I meant disk-IO that would block worker threads. as soon as an HTTP request has arrived, all the work until pushing the result might be possible without any IO.
10:07:38
moon-child
ah, yes. Usual approach for this is coroutines, but I don't think we have any cl implementations that support them
10:08:22
moon-child
(of course, come closos, plain threads will be cheap enough it won't matter. But who knows then that will be)
10:08:45
hayley
Another difference is that I have connections which are expected to be persistent, whereas I can't remember if Hunchentoot supports keep-alive.
10:10:13
flip214
hayley: you can tell HT to ignore a TCP socket - and pass that on to an extra thread.
10:11:27
hayley
moon-child: FWIW my secret plan is to prototype "green threads" in a fork of SICL, which would then be somewhat reusable for a thread implementation in CLOSOS.
10:13:28
flip214
hayley: green threads - with all the features of OS threads? like scheduling priorities, blocking behaviour for OS calls, timeouts, ...??
10:14:03
flip214
I've worked on such a library in C - but it never got up to par on features with kernel threads.
10:16:21
hayley
Well, Erlang has priorities, and in the case of CLOSOS, blocking OS calls could be handled with the usual devices (mostly semaphores I suppose).
10:18:36
hayley
On a Unix system, I think the Erlang and Haskell people both either attempt to use asynchronous operations or a thread pool for synchronous operations, which still use something like mailboxes to unblock the appropriate thread and provide the result.
10:22:32
hayley
Apparently the fastest servers these days use other async stuff, which is backed by some polling technique (epoll, etc, maybe io_uring if you're fancy). Without some intermediate stuff they don't work too well with threads apparently.
10:24:43
flip214
yeah. But if your actual workload (PDFs, for me) take 30-60msec to generate, and I only have a handful of cores (and a handful of threads), the server implementation doesn't actually matter.
10:25:10
flip214
of course, the "hello world ~a" examples need to keep the server as small as possible for their benchmarks...