libera/#commonlisp - IRC Chatlog
Search
9:14:31
jackdaniel
but if you want to call the unix function, then you need to use an extension (either builtin or via ffi)
9:25:28
jackdaniel
if you are going to use uiop, please please don't forget to add it in depends-on section of your asd file
9:34:11
AadVersteden[m]
So, I was really happy with the profiling tools I have in Common Lisp, but i'm somewhat stuck in my experiments right now.
9:35:42
jackdaniel
why include uiop or why AadVersteden[m] is happy with their experiments? (what experiments? :)
9:40:15
nij-
(ql:quickload :zmq) raises the subprocess error - it didn't include the correct files so that <zmq.h> was not found. How to alter its behavior so it compiles zmq successfully? Details: https://bpa.st/YSRA
9:47:15
AadVersteden[m]
Sorry, kids got in the way of the conversation ^_^. So, the profiling I ran shows I need less computing cycles in Lisp than in Elixir. However, when I make the roundtrip through an HTTP connection (which I haven't benchmarked but don't know a faster lib than what I use now), then the latency is substantially higher.
9:48:16
AadVersteden[m]
I went for Fukamachi-ware as my understanding is that that's rather fast, although it may have caveats.
9:49:44
AadVersteden[m]
sb-prof indicates most time being spent on functions I've benchmarked before, but those are faster than their Elixir counterpart so I'm clearly not measuring everything with that.
9:51:16
AadVersteden[m]
I'm basically receiving an HTTP request (from another Lisp Docker container), transform it, send a request to another HTTP service (a SPARQL endpoint, also in a Docker container), and send back the response. The transforming bit can be quite elaborate
9:52:03
nij-
In the ASD file, there is something mysterious to me: (cffi-grovel:grovel-file "grovel") .. among other (:file ..). It seems to be the failing point.
9:52:25
AadVersteden[m]
Looking at CPU cores, I'm not CPU bound in either of the approaches but latency is still a real issue.
10:03:07
nij-
For those who could help, I have a better detail description typed up here (https://bpa.st/U5TA). In short, I do not understand CFFI-GROVEL which seemed to make a mistake while compiling.
10:03:25
beach
thuna`: Would it go through the entire (potentially infinite) stream and delete all those characters or bytes, so that they don't show up when a read is called?
10:04:21
thuna`
beach: Oh, no, that's not how I meant it. So for example delete-char would delete the character that you would get with read-char, delete-line with read-line, and so on
10:07:32
beach
What function in the operating system do you imagine Common Lisp would call in order to accomplish that?
10:07:40
AadVersteden[m]
thuna`: You'd have to read and write the file. If you have multiple of these operations then I suspect you'd want to group them rather than to write to disk each time (or at least, asking to write it each time). It doesn't sound like a very complex primitive to write and I guess people would understand it when they read it.
10:08:36
beach
thuna`: Every character after the one deleted would have to be moved, potentially on disk or some other external device.
10:08:40
thuna`
AadVersteden[m]: So go through the file with :overwrite and just not write the sections that I want to "delete"?
10:09:57
beach
thuna`: You understand that something like that can be very expensive in terms of performance.
10:10:03
jackdaniel
in lispos file blocks should have size of one character, that would solve that! :)
10:11:47
AadVersteden[m]
for a database-startup thing I did we kept a list of changes to be applied to the datastructure. If an OS would offer such thing then this could possibly be performant but I don't know of any offering this. Regardless of the performance, if that's the functionality you need, you may just take the performance with it :-)
10:12:16
beach
thuna`: Imagine a hypothetical OS system call that deletes a character from a file. Since files consist of blocks of characters, that system call would have to move every character following the one you delete. And that, in turn, means reading the entire file and writing it back out with one fewer characters.
10:13:45
thuna`
beach: maybe the stream could buffer those deletions, with something similar to force-output
10:15:27
AadVersteden[m]
jackdaniel: the changes are akin to a transaction log, yes. The log is incomplete as changes are materialized based on heuristics and the log itself can be compacted. State is base state + changes.
10:16:41
AadVersteden[m]
Are there any low-latency libraries for HTTP server and/or HTTP client anyone here knows of? I went with Fukamachi's code but the interface to it is fairly limited and I'm in tryout stage. I don't know which of these two steps is causing the higher latency and am unsure how to benchmark.
10:18:54
AadVersteden[m]
jackdaniel: I wouldn't classify it in the big data realm I've been part of earlier, this is normally running on a single (but sometimes quite beefy) server.
10:19:34
jackdaniel
I'd just go with something with convenient interface then (and not grasp for "the fastest")
10:20:24
AadVersteden[m]
jackdaniel: oh no, speed is a real issue. Based on production data. That's what I'm aiming to tackle.
10:27:05
AadVersteden[m]
There's two parts: the processing time on Elixir is too high. We want to do more advanced processing and mutable objects will seemingly be key for achieving that. So there's a real need to shift to something allowing for that (discussed with people with intricate knowledge of BEAM, the runtime). I see that we can do the processing itself fast enough and that there's leeway, but it seems there is a latency in http connection
10:27:05
AadVersteden[m]
that overshadows current processing. I don't know if I should look at incoming or outgoing but I can plug in alternatives. I'm tinkering with it either case.
10:28:48
AadVersteden[m]
Cfr the amount of requests, I haven't checked but suspect it to be in the broad neighborhood of a million requests per day. It's more latency than throughput for the non-processing part.
10:36:06
AadVersteden[m]
_death: I have not, but I wouldn't expect me to write a faster http client/server than what was built before 🤔
10:43:16
_death
I see.. well, probably best dig deeper into the cause of the performance issue first.. on linux there are tools like perf
10:56:03
nij-
Hmmm.. I have a usocket client (https://bpa.st/4YMQ) connected with a server setup with `nc -l 127.0.0.1 12321`, but whatever I typed into the lisp repl doesn't show up on the terminal. I'm sure it's connected, because when I kill lisp, that `nc` stops to listen.
10:56:15
pjb
thuna`: POSIX only offers a very rudimentary kind of files: sequences of octets. You cannot insert or delete octets in those sequences. Only append octets. You may replace octets in those sequence, but only block by block (blocks are often something like 512 or 4096 octets).
10:57:02
pjb
thuna`: other (older) operating systems had other kind of files. For example, sequences of records. You could then delete and insert records. There was also various kinds of indexed record files.
10:57:52
pjb
thuna`: on POSIX system, when you what that kind of sophistication, you must use or implement a file access method library. For example, db3, sqlite, isam, etc. for character-by-character insertions and deletion, those libraries are called text editors.
10:58:49
pjb
thuna`: you could want to develop a library exporting only the text file editing API, instead of a whole editor like climacs or hemlock.
10:59:06
AadVersteden[m]
_death: Yeah, at least try to figure out if the http server or the http client is slow. Perf indeed, that's a good guess. I've been bitten my FOSS marketing claiming the FOSS library is fast XD
11:00:08
pjb
thuna`: and forget about streams, streams introduce a temporal aspect that is incompatible with editing (insert or delete) until you have a time machine. (think socket streams, protocols, remote, etc).
11:04:47
_death
AadVersteden[m]: I think lukego and flip214 did some work to help with exposing lisp names to perf
11:14:22
jackdaniel
preliminary support for emscripten: https://gitlab.com/embeddable-common-lisp/ecl/-/merge_requests/277 (contributions welcome, i.e if someone would provide a nice widget for a repl instead of default emscripten thing)
11:15:20
AadVersteden[m]
If they read this, I'd love to get pointers. I'm trying to zone in based on what I know so far. It seems likely the issue is in the outgoing request and I think I might not be sending the request to the other endpoint in the exact same way so who knows.
11:15:22
_death
nij-: you need to pass the socket stream to force-output (or finish-output).. also, no need for the your unwind-protect since with-client-socket already does that
11:18:47
_death
AadVersteden[m]: I believe the code snippet is https://github.com/phmarek/sbcl/blob/master/contrib/sb-perf/export-syms.lisp
11:33:07
AadVersteden[m]
Thanks for the suggestion and motivation all. It seems there's something strange going on with the connection-pool in dexador. Without the connection pool the lisp variant has about 2x the speed of the old system (which does a tad more, but I think that'll fit in the speed budget).
11:54:52
pdietz
This causes confusion with Paul H. Dietz, who has done interesting work but is not me.
11:56:56
nij-
Does anyone know how burgled-batteries work (https://github.com/pinterface/burgled-batteries)? How does it call python without launching a python process?
12:02:30
nij-
I suppose. In the low level code, the dynamic stuff doesn't get embedded. But instead, it's called?
12:04:50
nij-
and that means if other programs want to call sbcl, it doesn't have to wait for the full sbcl to launch?
12:05:45
jackdaniel
that means that you may link your program with a runtime, but usually you will still need to call some "initialiation" function before using functions of such managed runtime
12:06:19
jackdaniel
so I'd say, that you'll have to wait for the "full <lisp> to launch", but not by calling a binary, but rather invoking the "launch" function after the library is loaded
12:06:28
nij-
but that still means a lisp is launched, though it doesn't seem to be a separated process?
12:18:08
McParen
hi, if a macro generates the argument to a function, for example (defmacro foo (&body body) (let ((bar (gensym))) (lambda (,bar) &body)), is there a way to refer to the gensym-generated bar from the body? is there a way to get the lambda list of the function i am in?
12:20:43
pjb
McParen: also, you may give access to it thru a function, or local symbol-macro: (defmacro foo (&body body) (let ((bar (gensym))) (lambda (,bar) (flet ((get-argument () ,bar)) ,@body))))
12:21:00
pjb
(defmacro foo (&body body) (let ((bar (gensym))) (lambda (,bar) (symbol-macrolet ((the-argument ,bar)) ,@body))))
12:21:37
pjb
(defmacro foo (var &body body) (let ((bar (gensym))) (lambda (,bar) (let ((,var ,bar)) ,@body))))
12:21:45
jackdaniel
this gensym+symbol-macrolet is kind of like sticking right hand in a left pocket, no?
12:22:49
jackdaniel
the variable itself may be optimized out, so merely being able to access the symbol does not guarantee that you can access the variable
12:23:02
McParen
I am tring to not name the argument to the lambda, let the macro name it through gensym, but still be able to access it in some way from the body.
12:24:12
McParen
I'm trying to hide a repetitive variable that has to be there, but is seldomly used.
12:26:22
_death
e.g., (defmacro foo ((&key (my-var (gensym))) &body body) `(lambda (,my-var) ,@body))
12:27:14
pdietz
The only thing you'd worry about here is polluting packages with otherwise never used symbols. But uninterned symbols (as created by gensym) would solve that.
12:28:46
McParen
pjb: in both cases, flet and symbol-macrolet, I am introducing new variables that will be present when the macro is expanded, what i was trying to avoid.
12:29:17
McParen
_death: that seems to be a nice solution. name the variable when needed, otherwise use a gensym.
12:30:54
pjb
McParen: macroexpansion time is compilation time. if it's not used, it's not compiled in.
12:32:20
pjb
Try: (defmacro foo (&body body) (let ((bar (gensym))) `(lambda (,bar) (flet ((get-argument () ,bar)) (declare (inline get-argument)) (symbol-macrolet ((the-argument (get-argument))) ,@body))))) (disassemble (compile nil (foo 'hi)))
12:48:31
McParen
pjb: naming symbol-macrolet still pollutes the namespace, so i cant (defparameter the-argument 1) and then (funcall (foo (print 'hi))). What I tried to find out was whether it is possible to access a lambda list from a function itself, so I can let gensym name the argument. but thanks.
12:49:42
McParen
_death's suggestion to only name the variable when needed and other wise gensym it is good enough for my case.
12:59:55
_death
note that you can use &key's suppliedp feature to decide whether to declare ignore the parameter later
13:26:22
mfiano
If I wanted to do some MOPery to make a class final, such that defining any new class in any package with the final class name symbol in the superclasses list and C-c C-c would result in an error at compile time. Is this possible, and where should I start looking?
13:29:58
pjb
mfiano: I'd do: (defmethod initialize-instance ((object the-final-class) &key) (if (eql (class-named 'the-final-class) (class-of object)) (call-next-method) (error "final class")))
13:30:37
pjb
at compilation-time, apart from undefining the final class, I don't see. Perhaps with a special meta-class?
13:33:35
pjb
Then a method on ensure-class should let you do the same, at compilation-time, I'd guess.
13:34:18
mfiano
Doesn't ensure-class happen at runtime to finalize if need be, when make-instance is called though?
13:35:39
mfiano
I'm wanting to get the error when a class that might not even be final attempts to build itself up from a restricted class
13:46:46
thuna`
"OPEN :IF-EXISTS :NEW-VERSION is not supported when a new version must be created" Isn't that the entire point of :new-version?
14:12:53
thuna`
Alright, so I decided to use :rename-and-delete, now https://0x0.st/o6eP.txt is signalling an error opening output, saying the file exists.
14:22:18
McParen
is there an opposite to (declare (special foo))? A declaration that within the following lexical scope, foo is not special even if it was previous defined special?
14:24:10
McParen
or does the shadowing by a let or lambda list implicitely consider a local variable non-special?
14:24:10
pdietz
Not needed.  "A special declaration does not affect inner bindings of a var; the inner bindings implicitly shadow a special declaration and must be explicitly re-declared to be special. special declarations never apply to function bindings."
14:25:27
pdietz
"When used in a proclamation, a special declaration specifier applies to all bindings as well as to all references of the mentioned variables."
14:27:58
pjb
in (let ((x 42)) (defvar *x* x)) defvar is not a toplevel declaration. the compiler may signal undefined variable errors.
14:28:22
pdietz
The defvar still creates a global declaration there, it just doesn't do it at compile time.
15:01:51
nij-
Anyone familiar with cffi-grovel? I got a problem while installing cl-zmq.. https://bpa.st/6OCA
15:08:49
jackdaniel
ffi bindings always require the library in question, and the groveller requires also header files I think
15:11:23
kakuhen
i've avoided homebrew precisely because it "installs" a library and cant even at least add ${prefix}/lib to the ld path or install pkg-config entries
15:12:39
nij-
but the problem is I need to add certain flags (as shown in the link above (here again https://bpa.st/6OCA))
15:13:37
kakuhen
seems like a homebrew issue (unsurprisingly); this path is standard and would work on my mac
15:14:26
nij-
kakuhen I suppose the problem is actually here https://github.com/freiksenet/cl-zmq/blob/master/zeromq.asd#L20
15:15:50
yitzi
That is what I use for common-lisp-jupyter...although I have comtemplated writing native Lisp for zeromq several times.
15:18:31
yitzi
nij-: I'm not sure about your setup, but I've used homebrew+pzmq plenty without issues. Not with roswell, though.
15:18:43
nij-
However, if I add the flags `-I/opt/homebrew/Cellar/zeromq/4.3.4/include -I/opt/homebrew/Cellar/libsodium/1.0.18_1/include ` then it's fine
15:25:18
_death
if it's just for your own use, you can add them in the asd file (:cffi-grovel-file "grovel" :cc-flags "...")
15:25:50
kakuhen
just installed zmq from macports and quicklisp compiled the entire system just fine
15:25:56
kakuhen
ACTION uploaded an image: (197KiB) < https://libera.ems.host/_matrix/media/v3/download/kotoura.moe/McDTLcZYFmcHpQrlCGQQuNpY/%E3%82%B9%E3%82%AF%E3%83%AA%E3%83%BC%E3%83%B3%E3%82%B7%E3%83%A7%E3%83%83%E3%83%88%202022-11-11%207.25.30.png >
15:27:27
kakuhen
no idea since I've never used zmq (only rabbitmq) and I stay as far away as I can from anything CFFI
17:15:56
White_Flame
(old american cartoon started that way. also, I thought this was another channel :) )
17:19:39
Josh_2
I say a method but actually its an old method before I changed its name, clearly I forgot to modify this instance ;(
17:23:17
Josh_2
I changed service-name to version and somewhere service-name is still being called :(
17:29:26
_death
you have something that swallows the error? then maybe (defun service-name (...) (break "debug me"))
17:30:31
Josh_2
I was just thinking that. I might also be able to just specialize the gf and then use M .
17:37:43
jackdaniel
beach: http://www.lispworks.com/documentation/HyperSpec/Body/26_glo_s.htm#specialize
17:37:54
jackdaniel
"specialize v.t. (a generic function) to define a method for the generic function, or in other words, to refine the behavior of the generic function by giving it a specific meaning for a particular set of classes or arguments. "
18:53:01
Josh_2
Postgres does not support ratios without me creating two integer columns and then recombining them when I restore the data into cl. Considering this is there any benefit to using ratios to represent monetary values?
18:55:01
nij-
Could someone help explain the behavior of with-open-file? The scenario: I have created a FIFO, and have written something into it.
18:55:21
nij-
I then call this function (https://bpa.st/GAKA). Immediately, step 1 and 2 are fulfilled.
18:56:15
nij-
Anyway, now Lisp pauses here. I then write to the FIFO once again. Then step 3 is fulfilled, but the function doesn't return here.
18:56:24
jackdaniel
or, if you have fixed precision, as i.e an integer is then multiplied x100, or x1000 etc
18:58:51
Catie
I'd think that multiplying the value you get back from the database by 100 would give you an integer that can be used for fixed-point calculations
19:01:42
nij-
Yes, I found some other workarounds.. but this bug literally caused me 4 hours.. and I hope I can understand it before moving on.
19:03:20
nij-
even with what I have in the code, I don't see why step 3 isn't fulfilled immediately after calling the function.
19:05:01
Josh_2
(with-open-file (s "~/.emacs")(format t "beep")(force-output)(sleep 5) (uiop:read-file-string "~/.emacs")) -> beep ... 5 seconds ... <.emacs>
19:10:42
Josh_2
jackdaniel: Thanks for your input, this was the reason I chose to use ratios in the first place. Just a bit of faffing to get it all to work nicely with postgres
19:11:36
Catie
nij-: Sure enough, it hangs on my machine as well until I write something into the FIFO
19:11:53
Josh_2
Can't decide if I should use dedicated columns for numerator and denominator or just stick with what I have now which is JSONB
19:11:58
nij-
Catie yeah, but why should that be the case? It's not clear why in CLHS http://clhs.lisp.se/Body/f_open.htm
19:12:13
Josh_2
for ease JSONB is better, but for DB performance in the future dedicated columns will surely be faster.
19:16:03
Catie
I'm gonna see if I can reproduce it in C as well, but it's gonna take a bit because I'm out of practice
19:19:23
nij-
https://github.com/sbcl/sbcl/blob/master/src/code/fd-stream.lisp#L2549 this seems to be where it calls unix open
19:21:59
Catie
Here's the code I'm using, I'm not _entirely_ certain it's equivalent, just a quick-and-dirty test: http://0x0.st/o6_C.c
19:23:05
Catie
:DIRECTION defaults to :INPUT, which corresponds to SB-UNIX:O_RDONLY, which is the flag I used, so that's good
19:24:37
nij-
***** So for folks who'd like to help, here's the simplified question: (open FILE) hangs for fifo FILE but not regular FILE. I wonder why. This is observed on sbcl and ccl, but open() in C doesn't block. The relevant code in sbcl is here: https://github.com/sbcl/sbcl/blob/master/src/code/fd-stream.lisp#L2549***
20:09:34
pjb
Josh_2: https://www.postgresql.org/docs/current/datatype-money.html is too limited for a lot of usages. To small a range, not enough precision, for certain applications. I'd advise to use VARSTRING.