freenode/#lisp - IRC Chatlog
Search
0:28:02
aeth
cl-json and yason choose perhaps the most incorrect choice, of treating nil as null. st-json and jsown and jonathan produces [] and com.gigamonkeys.json produces {}
0:28:37
aeth
And some of these are configurable, e.g. cl-json, but defaults matter. And you shouldn't be able to do absolute nonsense like nil -> null or nil <-> null.
0:33:07
aeth
Adding a keyword "tag" to a list isn't hard because you just cons on a :foo tag and then when you're dealing with it as data, take the cdr of it. So that's the approach I'll use for distinguishing alists, plists, and regular lists. But by default I'll have JSON read into hash-tables and vectors because those are unambiguous and don't need to be "tagged".
0:36:29
aeth
I do have a separate false instead of treating nil as false when I work with embedded programming languages within CL, such as my still-incomplete cl-scheme. But that's imo totally unnecessary for JSON when you can just not support bare lists and have a :null keyword.
0:37:51
aeth
peterppp: My advice for you if you're not going to make your own JSON parser would be to find a configurable library and configure it to actually have reasonable cl <-> json.
0:38:18
aeth
It's unfortunate that none of them have what I would consider reasonable defaults because defaults are extremely important in software, and are often neglected.
0:41:23
aeth
Imo, the best mapping of the difficult parts is #() <-> [] and (make-hash-table) <-> {} and nil <-> false and :null <-> null, if you're willing to work with vectors and hash tables instead of lists for the part that interfaces with JSON.
0:42:35
aeth
:false <-> false would be even more unambiguous, but is probably unnecessary as long as you don't serialize plain lists (which create the three-way ambiguity when there's an empty list)
0:44:10
aeth
If you can configure that in an existing library, you can go cl -> json -> cl or json -> cl -> json losslessly
0:46:06
pjb
Funny that you should mention it. I can guarantee you that if you change the order, this will break some components somewhere!
0:46:43
pjb
Of course, JSON dictionaries are not ordered, but it's a 100% certainty that some software out there expects entries in a given order.
0:48:53
aeth
When I write my own JSON library, it'll support alists and plists prefixed with :json-object-plist and :json-object-alist and lists prefixed with :json-array. So (:json-array 1 2 3 4) -> [1, 2, 3, 4] and (:json-object-plist :foo 42 :bar 37) -> {"foo" : 42, "bar" : 37} and (:json-object-alist (:foo . 42) (bar . 37)) -> {"foo" : 42, "bar" : 37}
0:49:13
aeth
Of course, optionally reading it into alist or plist instead of hash tables in case the order needs to be kept
0:50:22
aeth
I'll probably have to also have a configuration for how keywords <-> strings is handled. By default, it'd be upcase to CL and downcase to JSON, but I'm sure some JSON is case sensitive and that won't work.
0:52:52
aeth
On top of the JSON library, I might want to add some YAML support, too. There are apparently 3 YAML libraries in Quicklisp, but I don't think there were any when I was looking for one years ago. YAML is built on top of JSON. I probably don't need to deal with XML.
1:02:38
pjb
aeth: I already indicated the right way to do it: you need to have a model of the JSON ontology in lisp (implemented eg. with CLOS classes). Then you can provide niceties such as converting p-list or a-list to and from instances of this JSON model ("DOM"). But at least, the JSON serialization and deserialization between these ontological objects can be done with 100% fidelity and clean roundtrips.
1:03:16
pjb
And if your lisp data is not of the kind expected in an application, you can always instanciate those DOM classes to create or parse exactly the JSON you have to process.
1:08:56
aeth
pjb: I was thinking about something more like this: https://gitlab.com/snippets/1690468
1:10:14
aeth
And for alists and plists, no need for the overhead of CLOS objects, just make the car :json-object-plist or :json-object-alist, which is just a trivial (cons :json-object-plist some-plist) before passing it into the JSON generator. For nested JSON generation it'd be a bit trickier, though.
1:10:25
pjb
The question I ask, is whether a CL hash-table is entirely semantically equivalent to a JSON dictionary?
1:12:07
peterppp
I just played around with this library https://github.com/fukamachi/websocket-driver
1:12:48
peterppp
but apparently the server has no way of finding out which client an incoming message came from
1:13:39
aeth
pjb: My current design is assuming that the CL user is going to try to generate sane JSON and that the JSON is where the potential incompatibilities are. Sanitizing the CL symbols, etc., is going to really hurt the performance, so even in the final thing, I'd leave that off by default.
1:15:06
aeth
pjb: I could replace it with a (format t "~S" string), though, and that will escape things automatically afaik. So then the issue is with edge cases in string escaping.
1:15:44
pjb
aeth: before saying that you will write your own buggy implementation, why don't you read the specification for JSON?
1:16:14
pjb
we already have at least 3 different JSON libraries in quicklisp, I don't think it's worth writing yet another one.
1:16:47
aeth
pjb: The problem is that all of the JSON libraries' defaults are shit. They all fail the nil test, i.e. nil <-> something-in-json is an absurdly bad choice in all of them. And defaults matter.
1:18:31
aeth
pjb: There are almost certainly bugs in all of them if the authors couldn't even be bothered to establish a reasonable cl <-> json mapping in the basic library design, which is the starting point of a JSON library, not some detail to work out later.
1:19:00
pjb
peterppp: this is the clisp function to find the IP of the client connected to a socket.
1:23:04
aeth
pjb: That could perhaps produce a cleaner result, but not a faster result. I need JSON support to read in 3D model formats that at least partially use JSON because JSON's trendy at the moment. So I will sacrifice some good design for performance, which seems to be the norm for game engines.
1:23:53
aeth
My personal design considerations are (1) a better cl <-> json mapping and (2) better performance
1:24:25
peterppp
maybe I can modify the library so that an object representing the socket is passed to the message handler
1:24:30
pjb
Just serialize directly your model, no need to go thru even lisp generic data structures.
1:25:09
aeth
So it'll probably have a bunch of optimizations that only make sense for glTF and any other JSON-based model formats that I happen to add support for later
1:25:40
aeth
At the moment, I'm going to handle .obj models and glTF models for import/export, in addition to a binary internal 3D format.
1:26:03
aeth
For textures, I'm just going to handle PNG because there's a good library for that and I don't have to write my own.
1:26:48
pjb
aeth: but if you write floating points, take into account the format specified for json floating point numbers!
1:26:56
aeth
The only other media format I think I'll have to support is sound... and I'm not going to even deal with sound yet. I'll probably need to support Opus and OGG Theora and maybe flac.
1:27:26
aeth
And the only other data format I'm going to mess with is an s-expressions-as-data format, which is fairly easy.
1:28:42
aeth
whoman: (1) their defaults are bad, (2) they usually focus on flexibility over performance
1:29:19
aeth
whoman: I want reasonable defaults and if I control the library, I can cheat and patch the library to meet all of the performance requirements necessary for the particular glTF JSON that I will be dealing with
1:29:59
aeth
And I really hope that between OBJ and glTF I don't have to support any other import/export for models... because there are so many 3D model portability formats. More than a dozen.
1:30:46
aeth
The older ones tend to use XML because that was trendy at the time. The newer ones tend to use JSON because that's trendy now.
1:32:27
whoman
used to basically design software in that syntax anyhow back in the day in C/c++/objc lands
1:33:03
whoman
ACTION loves NSDictionary also. which is also QuakeEd level editor map format and original DoomEd BSP format ........ so maybe a bit biased
1:35:05
aeth
whoman: An id tech 3 (or perhaps earlier) asset importer would be cool. There are a lot of cool maps, etc., in id tech 3. That would be a much lower priority thing to do.
1:39:49
aeth
pjb: Does JSON require a more formal specification than the diagrams on its website? I've always just used the diagrams. https://json.org/
1:41:26
aeth
ah, I see, there's an actual PDF. https://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf
1:43:07
pjb
aeth: the diagrams are formal enough for the syntax; remains some elements of the semantics, but since it's a data representation language, there's not too much of it. (like meaning of order in various places, what about duplicates, etc)
1:44:39
|3b|
sabrac: looks like it is just data files from unicode.org with an asd file so ql can download it. http://cldr.unicode.org/
1:46:27
aeth
So what I'm probably going to do is support: data s-expressions (.sxp), json (going to write this with the glTF support in mind), and maybe yaml. For 2D, just PNG for now (fortunately, I don't need to write this, at least for reading PNG files). For 3D, OBJ and glTF (requires JSON). For audio, probably Opus and Ogg Theora and *maybe* Flac. And everything else will be internal custom formats.
1:47:04
sabrac
|3b| yes, that is what it looks like. Just wondering who did it. I like to give credit where credit is due.
4:29:15
dmiles
whould this be adequate semantically? (defmacro pushnew (item place &rest rest) `(unless (member ,item ,place ,@rest) (push ,item ,place)))
4:30:27
dmiles
i am looking at where everyone else passes in an environment.. that is required isnt it?
4:47:23
Bike
"Can't specify the actual keywords [in the macro lambda list] since, apparently, non-constant keywords should be accepted." is an interesting if only tangentially related comment
4:54:17
dmiles
what makes me slightly crazy is that the package that keys go into or come from is so random
4:59:01
dmiles
i kinda liek how specializations happen in SICL .. i might end up using that simular trick
5:12:01
dmiles
i keep meaning to ask.. is there ever a time that "(eval-when (:compile-toplevel :load-toplevel :execute) t)" would not be t?
5:20:32
pjb
if it's a toplevel form then it always return T, but since it's a toplevel form, this T is just "compiled-in".
5:22:34
dmiles
ok so when inside a defun its compiled-in as t (since it contains :execute) i can get rid of eval-when wrapper?
5:23:02
dmiles
more like: ok so when inside a defun its compiled-in as t. Next since it contains :execute, i can get rid of eval-when wrapper?
5:24:02
dmiles
oh i was reading it as ? (eval-when (:compile-toplevel :load-toplevel :execute) (eval-when (:execute) t))
5:26:10
pjb
If you compile-file or load the compiled file then a toplevel (eval-when (:compile-toplevel :load-toplevel :execute) t) returns t (but as I said, it wouldn't matter); If you load the source file, then it would return t too. If it wasn't a toplevel form, then in the first two cases, it would return nil.
5:27:38
dmiles
i'll have to test the: (defun is-it-t () (eval-when (:compile-toplevel :load-toplevel :execute) t))
5:29:14
Bike
(defmacro eval-when ((&rest when) &body body) (if (or (member 'eval when) (member ':execute when)) `(progn ,@body) nil)) for non top level
6:50:38
beach
Is it possible to implement the #= and ## reader macros portably? For standard objects, it is possible to use the MOP to traverse them, but what about structs?
6:54:50
Zhivago
Wouldn't those take effect at the surface of the syntax of the constructing expression, as they are reader macros?
6:55:28
cryptomarauder
when you're far from your loved ones, especially your children, you'll not preocupy about priviledge so much in those moments. Rather christmas becomes something that you wish you could enjoy simply because it's an excuse to step out of the drudgery of apathy and disconnect we generally share most days. To me it's a good thing to practice even if you don't consider yourself a religious person. I for one am not religious.
6:55:29
cryptomarauder
I just say merry christmas becasue I hope you guys don't have to live far from your loved ones like I do. Just this.
6:56:07
beach
Zhivago: Consider #1=#(1 2 #1#). When you build the vector, you have to stick in a temporary object that you then fix up later, once the vector is built.
6:59:53
beach
I guess the same problem happens when you have to deal with instances of custom subclasses of CLASS.
7:04:11
beach
I guess for maximum portability, I could write the fixup routine as a generic function, and have the implementation specialize on their own classes.
7:05:17
Zhivago
If you can change the evaluation semantics to include proxies it should be straight-forward.
7:08:31
Zhivago
crypto: Fair enough. My recommendation would be to just skip the drudgery bit in the first place.
7:10:04
Zhivago
Ah, it has an opt-out clause -- but other than that it looks like it has the same sub-structural issues.
9:51:57
ym
shka, do you see how optimized ring buffers can be implemented on top of existing standard?
9:53:44
ym
I can see how it can work as sbcl extension, but on top of existing vectors, for example, this won't be nor portable, nor optimized.
9:59:34
shka
ym: more importantly: there is NOTHING that prevents you from writing optimized ring buffers in common lisp
10:15:16
ym
I got what bothered me. I want lisp machine reimplemented for FPGA board and able to self-modification. In that way I could build "hardware"-optimized ring buffers.
10:20:19
FareTower
ym: with two FPGA chips, you could have a duo-processor where one updates the other at a time.
10:22:45
ym
Or many such machines could be implemented with 9p support and distribute computing power via network for example.
10:23:00
FareTower
ym: for extra fun, read my essay "who (p)owns your computer" http://fare.tunes.org/computing/reclaim_your_computer.html
10:45:19
Shinmera
For those with a raspberry pi, here's a preview build of Portacle: https://twitter.com/Shinmera/status/945243615021977600
11:46:08
shka
i actually starting to write my own implementation of double ended queue before i reminded myself that there is very decent implementation just lying around