freenode/#clasp - IRC Chatlog
Search
14:37:42
drmeister
The kernel gets the identity and I believe that I now faithfully capture it and send a message with the correct identity to the middleware
15:02:40
drmeister
Bike: The bind worked when I passed it (pzmq:bind socket (format nil "tcp://127.0.0.1:9343"))
15:05:15
drmeister
I have a python script that sends a message and then waits for a response and a Common Lisp script that waits for the message and sends the response.
15:36:36
drmeister
Say something like "Chris - could you evaluate (core::expand-deftype 'babel:unicode-string) and paste the response"?
16:33:03
Bike
it wasn't anything recent. probably back when i changed deftype months ago. we haven't noticed since coerce with a variable argument is a little unusual.
17:08:01
drmeister
I have the test case working in sbcl now. tcpdump indicates that the response is not making it out of the Common Lisp code.
17:27:59
drmeister
I just read that I'm supposed to send one. I've tried "", #(), nil - non seem to behave like a delimiter
17:47:12
drmeister
I can do... (make-array 3 :element-type 'base-char :initial-contents (list #\a #\a #\a))
18:05:10
drmeister
I've got really limited mental bandwidth at the moment. I'm trying to write a pair of functions encode a vector of bytes as a string with utf8 (I guess) and then back again
18:05:38
drmeister
This freaking cl-jupyter was broken so badly - I have to rewrite the communication code
18:09:08
Bike
in other news, after those stupid bugs it's gone pretty well. i think i can eliminate the use of vaslists for normal functions
18:11:50
drmeister
Is this sort of thing reasonable? (position #(1 2) (vector #(2 3) #(1 2) #(5 6)) :test #'equalp)
18:19:09
drmeister
message-recv and message-send are now returning and accepting only list of (array (unsigned-byte 8) *)
18:20:10
drmeister
How can I get (array (unsigned-byte 8) *) to print as something with asc characters?
18:20:56
Bike
though hopefully you mean vectors and not multidimensional arrays? I dunno how it would work with those
18:22:01
Bike
i would just write a quick function that does like what shinmera said, and use that to show the debug output.
18:23:31
Shinmera
(defun print-ub8-array-as-string (array &optional (stream *standard-output*)) (write-char #\" stream) (loop for i across array for c = (code-char i) do (when (char= c #\") (write-char #\\ stream)) (write-char c stream)))
18:32:34
Bike
llvm has seen fit to turn allocasinto twelve-clause phi nodes, but that's probably good for it
18:33:40
Bike
keyword parsing is still impossible to understand, but i think that's inevitable, and now at least the generating code is commentated
18:37:38
Bike
a vaslist is a va_list plus the count of arguments remaining, but we don't actually need to update that count as we go, or store it in a structure
18:41:06
Bike
the main thing is the not updating, so e.g. if you have three required arguments (and not in registers) the ir just goes "%r1 = va_arg ...; %r2 = va_arg ...; %r3 = va_arg ...;" lickety split
18:41:39
Bike
we actually had (well, have, i can't push this for a while) two counts, one up and one down
18:43:48
drmeister
I'm getting the same damn result over and over and over and over and over and over again.
18:44:25
drmeister
No f*cking permutation of inserting null anywhere is getting the response to pass into the middle ware.
18:45:11
Bike
So you determined that the messages weren't passing /out/ of lisp, did i have that right? rather than the middleware getting them and then dropping them or something?
19:15:35
frgo
drmeister: I am back - sorry, got distracted by family - my wife and our dog requested their fair share of frgo time ;-)
19:17:01
drmeister
I'm stuck - I have two programs - one Python (called middle.py) that sends a simple request to the other in Common Lisp (kernel.lisp) that gets the request and is supposed to send a response to middle - but the response is lost/dropped whatever
19:17:25
drmeister
I'm looking for a pair of python examples that do the same thing but can't find one.
19:18:11
frgo
Could you post all source code involved somewhere? I'd like to see both kernekl.lisp and middle.py ...
19:27:47
drmeister
Because when I use the Common Lisp version of kernel.py - the response never makes it to the middle.py
19:37:15
drmeister
My mind is boggling wrt how that can impact here - everything is bytes in between.
19:38:07
frgo
Well, obviously middle.py does not see a "message end" expecting at least a different message end "signal".
19:38:49
drmeister
middle.py doesn't see it when it's sent from kernel.lisp - but it does see it when it is sent from kernel.py. Why?
19:39:43
frgo
And we need to find the difference between the message sent by lisp and by python - byte by byte
19:40:47
drmeister
Sure - but how? I used tcpdump - no response comes out of the kernel.lisp - but I do see the request from middle.py
19:52:07
scymtym
drmeister: is the DO loop in MESSAGE-SEND right? have you tried TRACEing PZMQ:SEND?
19:53:23
drmeister
Yes - we have an unrelated problem with clasp that bike may have fixed - but I want to work with something more reliable at the moment.
19:58:02
scymtym
i'm asking because it could be zmq waiting for more message parts before handing anything off to the kernel socket
20:01:51
drmeister
Excuse me while I go slam my head repeatedly into the hole that I've opened up by slamming my head repeatedly into a wall.
20:05:23
drmeister
Previously cl-jupyter was encoding everything coming from zmq as strings - I've changed it so that it returns vectors of octets and sends out vectors of octets.
20:06:59
drmeister
I had some other problems as well with null's and identities that I understand now.
20:08:04
drmeister
I see zmq documentation when I close my eyes and I had a dream about zmq default identities.
20:10:09
frgo
Hehe - I've seen this more than once - Mismatch between languages when protocol stuff is involved ... It took me 7 months in 1992 to debug a mismatch between a C API for IBM SNA on HP-UX and a PL/I Programm running on an IBM mainframe ...
20:13:08
drmeister
Sorry for dropping away - I rushed back into my docker image to see if this fixes the issue - it is getting the message now - but jupyterlab is still not responding. There are still problems down the line - I have a better understanding of everything though.
20:14:09
drmeister
Yeah - now the conversation between cl-jupyter and the middleware is going much farther. I can't say yet if it's correct.
20:19:41
frgo
Ah - don't really know if it matters, but there are messages being protocol version 5.1 (e.g. line 31), some being version 5.3 (e.g. line 85) and some being version 5.2 (e.g. line 184) ... Never used it that way.
20:20:49
drmeister
frgo: I'm not sure if it matters either. I think as long as everything is at least 5.1 and the headers have 'date' fields - it should be ok. But I could be wrong about that.
20:21:15
drmeister
The jupyterlab works fine if I get a evaluation going before the middleware gets the kernel_info_reply
20:26:21
drmeister
They were the start of the problem. the original cl-jupyter tried to convert them to strings.
20:26:43
drmeister
zmq creates an identity when you don't provide one that has the form #(0 w x y z)
20:27:46
drmeister
It turns out - the best way to deal with this is to not convert the wire protocol info into strings at all. To hold off and only convert the parts that are convertable.
20:28:05
drmeister
That requires rewriting the message-send and message-recv functions and the functions they rely on.
20:30:31
drmeister
They really should - and they shouldn't use strings. The strings are very deceptive.
20:30:31
Bike
if i remember correctly, the string case just allocates the foreign data for you. no reason you couldn't do the same with byte arrays. wouldn't even need to worry about encoding crap
20:31:33
frgo
There's cffi:with-foreign-object https://common-lisp.net/project/cffi/manual/html_node/with_002dforeign_002dobject.html
20:31:48
drmeister
Right - so does everyone agree - the pzmq library send should accept simple vectors of unsigned bytes and recv should return simple vectors of unsigned bytes?
20:32:56
Bike
https://github.com/orivej/pzmq/blob/master/c-api.lisp#L514-L520 the string case uses cffi:with-foreign-string. at least i think it's cffi
20:35:01
Bike
this is unrelated, but do you remember where you found out about the va_arg internal structure?
20:44:48
Bike
ok. the reason there's a register save area and stuff is to keep the calling convention reasonably uniform. i think i see. maybe i don't
21:03:55
drmeister
The register save area is to allow va_list's to work when the start of the va_list includes arguments that are passed in registers.
21:04:52
drmeister
The standard decided they would make va_list access slow because it's relatively rare.
21:05:42
drmeister
A better choice from the lisp perspective would be to do what sbcl does - leave open space in the argument list
21:53:18
drmeister
What would you think if we print simple-vector-byte8-t as strings with escaped characters?
21:54:01
Bike
i don't think i understand why you want to work this into core machinery, instead of just having a print-ub8-vector-nicely function you can call in your debug printing
21:55:22
drmeister
Actually - I'm not trying to work it into the core. I had written a special printer for that class long ago and it was FUBAR
21:57:04
drmeister
There is a method in the C++ code that is printing byte arrays in backtraces differently and its screwed up.
22:25:53
Bike
&va-rest might actually be a problem. &rest can be compiled as (list* farg3 (gather-va-rest)) or whatever, but we can't do much with va_lists
22:26:50
Bike
the best i can think of is having gatherVaRest take extra arguments for the registers. think that would work, just look kinda silly