freenode/#lisp - IRC Chatlog
Search
6:36:00
phoe
These two aren't disjoint. There's binary data that also decodes to a simple-base-string.
6:39:04
drmeister
In every message except for the very first one - the identity(ies) is a uuid string.
6:39:29
drmeister
In the very first message - the identity isn't provided by the jupyter notebook framework. Ident = None
6:40:52
drmeister
https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/session.py#L598
6:42:11
phoe
I'm looking at https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/session.py now to see if the implementation shows anything to me.
6:42:14
drmeister
As a historical note, ZeroMQ v2.2 and earlier use UUIDs as identities. ZeroMQ v3.0 and later generate a 5 byte identity by default (0 + a random 32bit integer). There's some impact on network performance, but only when you use multiple proxy hops, which is rare. Mostly the change was to simplify building libzmq by removing the dependency on a UUID library.
6:43:29
drmeister
The very first message I'm getting has a 5-byte identity that can not be translated into a simple-base-string using the UTF8 encoder - in cl-jupyter it errors out saying this.
6:44:31
drmeister
cl-jupyter has been generating this error for two years - Fredrick Peschanski apologized to me about it two years ago when I started using cl-jupyter. I don't understand why cl-jupyter worked at all - but I suspect it was some fluke of the protocol.
6:45:06
phoe
I don't understand how the identities are generated and then how they are told apart after they are sent.
6:45:35
phoe
The whole message looks like [ident1, ident2, ..., DELIM, HMAC, p_header, p_parent, p_metadata, p_content, buffer1, buffer2, ...]
6:46:14
drmeister
A couple of weeks ago it stopped working. I've been debugging this for three days and it's pretty clear that because this first identity can't be encoded into a simple-base-string that the return message from the kernal is missing the identity and so zeromq is dropping it.
6:47:03
drmeister
Yes - the identities come before the DELIMiter. The DELIMiter is the string "<IDS|MSG>"
6:48:21
drmeister
So - if I were to store anything before the DELIM as (array (unsigned 8)) and pass that through to the return messages - everything would (hopefully) be fine.
6:48:50
drmeister
Most of the time though - the identity(ies) are text UUID's - so it's fine to store them in simple-base-strings.
6:49:28
phoe
"As a historical note, ZeroMQ v2.2 and earlier use UUIDs as identities. ZeroMQ v3.0 and later generate a 5 byte identity by default (0 + a random 32bit integer)."
6:49:38
drmeister
The original cl-jupyter code converted every message part as a string - I think that was the problem.
6:51:45
drmeister
Note how he uses (cffi:foreign-string-to-lisp (pzmq:msg-data msg) :count (pzmq:msg-size msg) :encoding encoding) to try to convert the foreign-string into a lisp string with the default encoding (UTF-8) and then if it fails it prints a warning and uses LATIN-1 and says it's ugly.
6:52:03
phoe
I don't know the full context, but now it smells to me that there was a type assumption that these identities are text.
6:52:16
drmeister
It's more than ugly - it's mangling the identity and then I can't send a response using the mangled identity because zeromq drops the message.
6:52:46
phoe
The first byte of zmq3.0 identity is #x00 - that's not a textual character in any charset I know.
6:53:45
drmeister
I am very grateful for your feedback - I am not at all convinced that I'm on the right track.
6:55:20
Shinmera
The standard's charset is pretty irrelevant to anything we do nowadays and have been doing for the past twenty years
6:56:28
phoe
Theoretically, (flex:octets-to-string #(0)) doesn't error; in practice, I'd freak out seeing the resulting "^@" anywhere in my program.
6:57:08
drmeister
The tricky thing here is that zeromq switched to using these 5-byte 0x0 W X Y Z (where W X Y Z are 4 bytes of an int) identities when the user doesn't provide a UUID identity. UUID identities can be coerced to simple-base-string. 0x0 W X Y Z can not be - so I thought I would coerce it to an (array (unsigned 8))
6:58:26
drmeister
But I would only coerce it to an (array (unsigned-byte 8)) if it contains any characters for which graphic-char-p is NIL.
7:00:05
drmeister
Because the cl-jupyter code needs to work with simple-base-strings for everything that isn't this one identity.
7:00:40
drmeister
I don't think it ever looks into the contents of identities - it only passes them around and uses them to generate responses for requests
7:00:45
phoe
Then it seems that cl-jupyter was based on the false assumption that identities are strings.
7:01:20
phoe
If it's just passing them around, then it doesn't seem that there are so many call sites that would need to be edited to make it work with arrays.
7:02:32
phoe
But, in their case, you could completely throw out the try-to-coerce-into-string-or-whatever functionality and go for a much simpler one: read five bytes, write five bytes.
7:04:12
drmeister
I think the big problem is that zeromq switched to these binary identities by default when previously they used uuid's
7:06:34
drmeister
I'm a lot confused by this. These components of the wire protocol are converted to JSON.
7:08:25
phoe
From Python docs: `Bytes literals are always prefixed with 'b' or 'B'; they produce an instance of the bytes type instead of the str type. They may only contain ASCII characters; bytes with a numeric value of 128 or greater must be expressed with escapes.`
7:08:59
phoe
So basically, 'asdf' in Python is "asdf" in Lisp, but b'asdf' in Python is (flex:string-to-octets "asdf") in Lisp.
7:09:21
Shinmera
Or, if your implementation supports it, a static-vector with element-type base-char.
7:12:05
Shinmera
Or, if you're SBCL, (sb-sys:with-pinned-objects (string) (.. (sb-sys:vector-sap string)))
7:12:20
Shinmera
drmeister: static-vectors is a library that implements lisp vectors for which you can retrieve a pointer that you can pass to C
7:12:24
phoe
drmeister: there's a library called STATIC-VECTORS that allocates Lisp vectors suitable for CFFI usage.
7:13:44
phoe
Basically, you get a chunk of memory that is guaranteed to act the same when read/written from Lisp (via Lisp primitives) and read/written from C (via pointers). It gets funny when the memory is allocated as a base-char vector, because in C you get bytes, but in Lisp you get base-chars.
7:15:01
drmeister
Rather, I started using cffi because pzmq only accepts lisp strings and foreign data objects.
7:17:15
drmeister
The wire protocol dictates python byte arrays are used. So I send and receive (array (unsigned 8)) arrays.
7:18:14
drmeister
Then I pass those parts to wire-deserialize - and it converts what it needs to into lisp strings.
7:18:55
drmeister
Later I pass dictionaries containing lisp strings to wire-serialize - that converts the strings into (array (unsigned 8)) arrays and then I send those back to zeromq.
7:20:15
drmeister
Identities are not encoded by wire-serialize or wire-deserialize - they are only prefixed or removed from the start of the message.
7:20:53
phoe
Identities are basically tokens that you receive and send but don't anyhow interact with, right?
7:24:34
drmeister
I think my current approach will work - but it's weird because for one thing - it encodes strings immediately when things come in to recv -
7:30:03
phoe
drmeister: that process is weird, yes. Instantly encoding everything into strings is weird, especially that these things aren't strings.
7:30:58
unanimousarc
I have a Q, I'm going through Land of Lisp and there's this part where the lisp reader creates an instance of a structure directly from the printed representation: "(defparameter *that-guy* #S(person :name "Bob" :age 35 :waist-size 32 :favorite-color "blue"))" However, I get an error when I do this in SBCL
7:36:22
unanimousarc
I wrapped the defstruct but that doesn't seem to change it, should I wrap everything in the eval-when?
7:38:43
beach
phoe: So why does it work when he evaluates it? And why does the error mention "dump"?
7:39:27
trittweiler
Yes but it's still what beach hints at. In the case of #S(...) the person struct will be created at read time, and the object will have to be dumped into the fasl, to be bound to *that-guy* at load time
7:43:35
trittweiler
unanimousarc, Yeah, there's a way around that involving make-load-form but it's really quite advanced,I would suggest you to just go with MAKE-PERSON instead of the #S(...) notation. Note that the #S(...) notation would also create a read-only object.
7:45:26
phoe
There are several "times" in Lisp that are basically reader/compiler/etc. passes from raw text to actual Lisp code and data.
7:46:07
phoe
The file compiler's role is to grab Lisp forms, turn them into code and dump all of the resulting code into compiled files.
7:46:56
phoe
#S(person ...), when it's read, actually results in a Lisp *instance*. The reader turns it into an instance of PERSON.
7:47:42
phoe
The file compiler does just fine with lists, code, functions, and so on. But it chokes on actual instances of objects, for a good reason.
7:47:57
unanimousarc
hm, I'll have to read more about this compilation stuff eventually, for now I'll just treat lisp like an interpreter
8:28:19
no-defun-allowed
AeroNotix: [here is cl-decentralise in its god awful hackish form](https://gitlab.com/Theemacsshibe/cl-decentralise)
9:53:28
phoe
AeroNotix: you should make it possible to declare what the delimiter is. It should be possible to customize the delimiter, just like in HTTP content-type multipart/form-data. It has a custom boundary.
10:20:16
beach
v0|d: Because most modern garbage collectors don't touch objects that are no longer live. Such collectors could not call any destructor for dead objects.
10:21:30
beach
v0|d: Some Common Lisp implementation have what is known as "finalizers" that can accomplish some of what a destructor does. However, it is still not predictable when the garbage collector will detect that an object is no longer live.
10:21:32
Zhivago
The more important point is that by the time the object is collected it is no-longer reachable by a destructor. If it were, it could be re-linked into the graph by the destructor. Leading to things like resurrecting objects from the dead (once only) in Java.
10:22:55
Zhivago
Finalizers get around this by not operating on the dead object, but usually share some (still live) substructure.
10:24:57
heisig
v0|d: You could, however, implement your own generic function (destroy-instance X), which would internally do a (change-class X 'destroyed). Then CLOS would have destructors :) Of course this would be unrelated to the GC.
10:34:06
phoe
the basic use case for a destructor is freeing resources - Lisp has garbage collection instead.
10:34:27
phoe
another basic use case is closing closeable resources when they are no longer used - Lisp has WITH-* macros for that.
10:35:51
phoe
yet another use case is closing closeable resources that isn't limited to the dynamic scope of an object, so WITH-* macros can't help you there - and, for that, you can write an ordinary function in Lisp.
10:37:19
heisig
You forgot another use case - accidentally breaking your program. That is why I proposed DESTROY-INSTANCE :)
10:37:48
v0|d
Zhivago: Is it possible to detect for a compiler that a user defined finalizer can ressurect objs?
10:40:24
Shinmera
Because the finalizer stays alive due to the object and the object stays alive due to the finalizer
10:40:56
Shinmera
GCs protect against double frees and forgetting to free, but you can still create memory leaks.
10:41:27
phoe
the finalizer must be registered in the GC-accessible location somewhere, so GC knows to fire it when the collection occurs
10:41:42
phoe
And if the finalizer is registered in a live location, then the object it closes over is also live
10:42:17
phoe
if A closes over B and B closes over A, then both of them are dead if neither of them is accessible
10:42:46
phoe
so "the finalizer stays alive due to the object and the object stays alive due to the finalizer" is not enough to proclaim the object or the finalizer alive.
10:45:57
Shinmera
It's not necessarily the case that a finalizer is registered as a root, it just needs to be treated specially since, yes, usually cyclic references in objects are freed just fine.
10:46:49
phoe
Yep - it doesn't really have to be registered anywhere, but the GC needs to know where it is and how to fire it.
11:10:28
beach
no-defun-allowed: It's because, for some reason, people don't use Emacs abbrevs. Don't ask me why.
11:14:48
heisig
From the spec: "A girlfriend is a function whose behavior depends on the classes or identities of the arguments supplied to it." Well put :)
11:25:17
phoe
No doubt that Lisp isn't all that popular if Lisp programmers finalize their girlfriends
11:25:45
beach
jackdaniel: Then they should use the equivalent of Emacs abbrevs in the IRC client they are using.
11:34:56
heisig
Does anyone know a good library for type inference in Lisp? I'd like to have something like (infer-type 'cl:floor 'double-float) -> '(integer double-float).
11:35:58
no-defun-allowed
There's a chatroom with two other lispers, me and her and she hasn't been converted yet.
11:38:37
v0|d
no-defun-allowed: according to the definition, try supplying a proper argument to her.
11:56:11
TMA
no-defun-allowed: and then the old girlfriend would be garbage collected and possibly finalized. may I suggest CHANGE-CLASS instead?
13:01:50
hjudt
why do some programmers use keyword prefixes in loop? e.g. (loop :with manual-names...)
13:21:30
heisig
phoe: I already considered using SBCL internals. But I also like to have things portable. Maybe a mix of both.
13:25:01
pjb
hjudt: this is because loop keywords are not exported from CL. Therefore they can be exported from other packages.
13:26:06
pjb
hjudt: in that case, if you write a loop in your package without using KEYWORD, you intern symbols with those names in your package. Then if you use-package the package that exports symbols with the same name, you get collisions!
13:28:24
phoe
(loop #.(make-symbol "FOR") i #.(make-symbol "IN") '(1 2 3 4 5) #.(make-symbol "DO") (print i))
13:28:29
pjb
and for the people who find that loop is not lispy enough, using KEYWORDS instead of symbols makes it look more or less like an operator with &key arguments :-)
13:41:20
phoe
Something that would allow me to (trivial-ftype:function-ftype #'phase) ;=> (FUNCTION (T) (VALUES (OR (DOUBLE-FLOAT -3.141592653589793d0 3.141592653589793d0) SINGLE-FLOAT) &OPTIONAL))
13:44:11
AeroNotix
It's already one of the most unreadable forms you can use, anything to make it more readable helps
13:50:41
heisig
AeroNotix: In Petalisp (https://github.com/marcoheisig/Petalisp), I JIT-compile array definitions to fast specialized code. To do so, I need to know that adding floats produces floats and so on.
13:52:52
heisig
Ideally, I could also use type inference to prevent code like (+ "foo" 5) from ever being run. Run-time errors in distributed systems are not pretty.
15:43:48
ym
Is there any tool to visualize AST/ASG of LISP code so that it would be displayed as interactive graph like in modern visual programming DSLs?
15:45:38
slyrus1
scymtym: thanks! do you have a preference for merging the PR vs just pushing a (cleaned up) commit to master?
15:45:48
phoe
ym: I somehow don't find the right side of https://raw.githubusercontent.com/cuichaox/visual-cells/master/demo/slime-screenshot.png readable at all
15:46:55
phoe
it's a good way of turning four lines of readable code into half a screen of unreadable graph, as you can see above.
15:47:21
phoe
AudioMulch graphs look like https://dt7v1i9vyp3mf.cloudfront.net/styles/news_large/s3/imagelibrary/A/AudioMulch_02-gbYbMLm8pnWEMgVxdy.HhJMb4U0R8x_l.jpg - this looks not like Lisp code at all.
15:51:07
pjb
The thing is that things are un flux. There are various user interface toolkits, but they're usually hard to use natively (despite CL libraries). There's CLIM, but AFAIUI, McCLIM is being rewritten.
15:52:25
pjb
And if you wanted to put the GUI on a tablet, then we don't have many CL implementations running there either, and it's awkward.
15:55:50
ym
I'm aware of McCLIM. Don't like it's framework nature, too overthinked protocol. I want just zoom in/out the depth of code represented in network graph in respect to packages/function nesting for example. For me it's most comfortable way of getting a fast glance of system structure.
15:58:25
pjb
If you don't need sophiticated user interaction, people often use secundary programs to render the graph, such as GraphViz, displaying it as a bitmap.
16:02:35
ym
Some good guy from here shared his graph visualization program (cl-frame.lisp). That's enough to start with, but I'm just can't get why such useful features aren't widespread.
16:09:18
pjb
ym: that's because, as I've said, there has been and still is a lot of changes in the graphic and UI technologies.
16:11:51
pjb
Also, you'd have to choose a platform for it. (asuming you cannot write versions for all the platforms). A lot of lispers use macOS. Another big share use linux.
16:12:27
pjb
More of them use emacs. Perhaps this would be the right choice of a platform for such tools.
16:16:56
ym
Well, I don't get this software "evolution" tendency either. I thought that layers of abstraction should hide implementation details and provide simple interface. Like 20 years ago I was able to write SCREEN 12 CRICLE (320, 240), 32 in QBasic to ouput a circle. And now I have to struggle with X protocol, CLX, write over 20 SLOC to make the same.
16:30:13
gendl
Hi, what different conditions or parameter settings can cause (format nil "~t" some-variable) to give different amounts of spacing?
16:31:37
gendl
I think wrapping (with-standard-io-syntax ...) around it will make it work the same in both cases I'm seeing. But without with-standard-io-syntax, the ~t seems to be generating different amounts of whitespace.
16:33:35
pjb
ym: you're right but in the case of UI, there are yearly fads, it's not software, it's fashion.
16:34:40
gendl
phoe: I'll try. I'm just looking at static customer code right now, without the ability to run it in their environment...
16:35:04
pjb
ym: strugglying with X11 would be nothing (and very dated!) Nowadays, you'd have to do this in 3D with OpenGL, etc…
16:37:48
ym
Oh, no. OpenGL is way more horrible thing in this perspective. There is known reason why CS/IT stuff getting worse, but that's off-topic.
16:50:01
gendl
Ok i have an example which shows different output with & without with-standard-io-syntax
16:51:55
gendl
So when I wrap this with w-s-io-s, the output has more whitespace (I can post the actual output if that helps)
16:57:57
phoe
gendl: can you give me the result of (list *print-lines* *print-miser-width* *print-pprint-dispatch* *print-right-margin* *print-level* *print-length* *print-lines* *print-circle* *print-escape* *print-pretty*) ?
16:58:47
pjb
gendl: on the other hand, you have newlines and tabs in your format strings. The newlines are ok but you may prefer explicit ~%. The tabs are bad since they're so implementation dependent.
16:59:42
gendl
looks like *print-pretty* is different, and the *print-pprint-dispatch* is different.
17:00:42
pjb
gendl: In general, I would avoid with-standard-io-syntax. Instead, I would define my own macro setting all the variables as I want them.
17:01:25
gendl
customer is running ANSI mode, I happen to be running modern-mode here (probably bad, I know)
17:03:10
gendl
The customer is comparing output from our previous version which was Allegro CL 9 (32-bit), vs. our new version they're trying to get into production, which is Allegro 10.1 (64-bit).
17:04:16
gendl
phoe: In my function `try' there is a lot of whitespace to the left of all but the first line.
17:04:49
gendl
Should the ~1t force it into column 1, even with all the leading whitespace in the format string?
17:06:26
gendl
But you're right, that doesn't look right. The ======... should obviously be just below the title.
17:07:21
gendl
pjb: It's not a matter of what I want -- this is customer code, i have no control over it. (I can make recommendations to them, though).
17:08:00
phoe
If the behavior of that code changed between different versions of Allegro CL, then I'd ask Franz for clarification.
17:09:36
pjb
gendl: the principle of software is that it should be soft and easily (cheaply) modifiable.
17:10:23
gendl
phoe: Before I go blaming Franz I have to make sure we didn't inadvertently change *print-miser-width* or *print-pretty* in our stuff.
17:10:25
pjb
gendl: otherwise, I would write the code to format tables from raw data and headers, without pre-conceived format. The format would be computed automatically from the data.
17:11:35
pjb
gendl: notice in my example, that you only have 2 formats: ~A for strings, which should not depend on *print- vars, and ~6,3f. If you get your 6 characters, then this format should not depend on the other *print- variables either.
17:12:32
phoe
...and if this is code written by the customer, make a shameless suggestion to them to patch it.
17:16:57
pjb
gendl: since you only have absolute ~T directive, preceded only by literal strings (and absolute ~T directives), (apart for the ~F and ~A that are trivial), then I would say that if you get bad results, it's a bug in the implementation.
17:18:53
gendl
*print-pretty* is supposed to be nil by default, right? I'm noticing in Allegro CL 10.1, *print-pretty* is t, out of the starting gate.
17:19:14
gendl
And the difference between having it t and nil is causing the differences which the customer is reporting.
17:22:25
phoe
The default values for *print-miser-width* and *print-pretty* are implementation-dependent.
17:24:26
gendl
But if you look at the actual documentation for *print-pretty* and *print-miser-width*, they both do say the initial value is implementation-dependent.
17:25:41
trittweiler
there are more instances where that's the case. The initial readtable versus (copy-readtable nil), perhaps. (Haven't checked)