libera/commonlisp - IRC Chatlog

1:26:23 Guest74 getting closer I think. Pretty sure the diagonals are indicative of some error on my part. https://youtu.be/W0-eUFMgJ-o

2:24:21 Spawns_Carpet[m] do you guys think the lexer should be responsible for handling escapes in strings in source code?

2:47:50 Alfr clhs 2.4.5

2:47:50 specbot Double-Quote: http://www.lispworks.com/reference/HyperSpec/Body/02_de.htm

2:48:46 Alfr Spawns_Carpet[m], that's a strange question to ask for common lisp; reading strings is done using a reader macro function associated with " . see ^

2:49:47 Alfr Spawns_Carpet[m], for standard read table and when you're using CL's read.

2:50:20 hayley What lexer?

2:51:32 hayley The Common Lisp specification states that you read forms with a simple dispatch technique, rather than a lexer and parser. I'm aware that fancier readers, such as Eclector, are more sophisticated, but I don't know if they lex and parse.

2:53:35 hayley ACTION saw the specbot message first. Can't afford causality?

2:55:40 akater[m] no-defun-allowed: What section of the spec are you talking about re: read?

2:55:48 Alfr hayley, if you consider the turning streams of characters into tokens, in some sense read does that, though you immediately get the denoted objects. (But 2.2 talks about accumulating tokens.)

2:55:56 hayley clhs 2.2

2:55:56 specbot Reader Algorithm: http://www.lispworks.com/reference/HyperSpec/Body/02_b.htm

2:56:52 hayley Right, I guess there is a concept of a "token", but it is usually consumed immediately, and there is no separation between reading tokens, and producing nested structure from tokens.

3:27:03 Spawns_Carpet[m] what is the difference between a lexer and simple dispatch?

3:32:16 Spawns_Carpet[m] i am scanning and building up tokens and then linking those together in an SExpr data structure in rust which is a struct with car, cdr, both being an SExpr Value which is a tagged union of Symbol, String, Int, Char and SExpr which contains the next SExpr like a linked list

3:33:50 hayley The in-memory representation of a Lisp object is not a S-expression.

3:34:06 hayley An S-expression refers to what is written on a character device.

3:34:57 Spawns_Carpet[m] what do you mean by that? I have another LispValue union that contains 'compiled' lisp values later on

3:35:03 Spawns_Carpet[m] does that sound right

3:35:07 hayley Not at all.

3:35:37 hayley A lexer, at least to me, scans over an entire file or string or something, and produces a sequence of tokens, which a parser then consumes in another pass.

3:35:40 Spawns_Carpet[m] what is a "lisp object" in this context

3:35:57 hayley Anything that the Lisp system manipulates.

3:36:18 hayley ACTION reads

3:36:42 Spawns_Carpet[m] do you consider the compiler part of the lisp system?

3:36:46 hayley Well, the data structure with a CAR and a CDR is usually called a "cons cell".

3:36:47 hayley Yes, I do.

3:37:28 Spawns_Carpet[m] so what should the parser be emitting in this case

3:38:28 hayley There is usually a READ function which produces a Lisp object.

3:39:12 Spawns_Carpet[m] my line of thinking was that the bytecode compiler should work on an AST, and in lisp the AST is an sexpr, so thats what I was parsing and feeding into the compiler, then the compiler emits the actual lisp objects for the interpreter to work with

3:41:59 hayley Well, okay, but it seems weird to call the same objects both an "sexpr" and "lisp objects", and I am wondering if there is something I've missed.

3:42:47 Spawns_Carpet[m] no i think I am confused and at some points using the wrong terms as well

3:43:23 hayley Say, we have some code (F 'X). Presumably the whole list won't appear in bytecode, but the symbol X will.

3:45:55 Spawns_Carpet[m] right, it would do something like EVAL(F), PUSH_SYMBOL(X), CALL 1 is my guess

3:46:18 Spawns_Carpet[m] but the bytecode part I am still very shaky on

3:46:34 Spawns_Carpet[m] i only got basic math operations working, like a calculator pretty much

3:47:42 hayley What does the EVAL instruction do? It would be...odd for the bytecode VM to have an instruction which does a plain EVAL.

3:47:53 Spawns_Carpet[m] in this case eval would load a lisp object from the environment and put it on the stack

3:48:01 hayley Okay, sure.

3:48:18 Spawns_Carpet[m] i guess LOAD_VALUE or something would be a better name?

3:48:27 Spawns_Carpet[m] since eval already means something else

3:48:31 pillton Spawns_Carpet[m]: You might enjoy the book Lisp in Small Pieces by Christian Queinnec.

3:48:49 Spawns_Carpet[m] i am going to check that out pillton

3:49:17 hayley The one bytecode VM I did turned everything into De Brujin indices, i.e. every local variable and constant had a number. Then I might emit GET-ENV 0; GET-CONSTANT 0; TAIL-CALL 1;

3:49:19 Bike i would think that you'd have like PUSH_SYMBOL(F) LOOKUPF or something, meaning look up the global function named F

3:49:38 Bike "lookupf" for "lookup function", not related to "F"

3:51:26 Spawns_Carpet[m] that probably makes more sense yeah, especially since everything at the start should be a function

3:51:46 Spawns_Carpet[m] for the most part at least? might be exceptions

3:52:08 Spawns_Carpet[m] *no-defun-allowed*: that sounds really cool, I've never heard of that before

4:02:32 spiaggia Good morning everyone!

4:04:10 hayley Depending on how you handle the environment, especially if functions and other bindings reside in the same "namespace" or not, having a LOOKUP-FUNCTION that is different to LOOKUP-VALUE may or may not make sense. Because this is #commonlisp I am legally obligated to assume you have separate namespaces, of course.

4:12:51 Spawns_Carpet[m] haha, yeah I was planning on keeping one namespace for functions and variables but I am not super dedicated to either method

4:13:21 Spawns_Carpet[m] so far I haven't gotten that far though

4:15:23 spiaggia Spawns_Carpet[m]: The difficult part of a Lisp system is the evaluator. You can save yourself a lot of trouble if you write your system in Common Lisp, and use the host READ to parse your code. Then you can concentrate on the hard part.

4:15:52 Spawns_Carpet[m] i am considering that strongly

4:16:29 Spawns_Carpet[m] the whole text processing step is sort of tedious anyways

4:16:52 spiaggia And what you are doing is wrong in terms of how Lisp is typically parsed.

4:17:33 Spawns_Carpet[m] what do you think I should change

4:18:13 Spawns_Carpet[m] remember that I am pretty new to this, so bare with me

4:18:17 spiaggia It is done as described in chapter 2 of the Common Lisp HyperSpec.

4:18:38 Spawns_Carpet[m] even if I am not implementing common lisp itself?

4:18:41 spiaggia clhs 2.2

4:18:41 specbot Reader Algorithm: http://www.lispworks.com/reference/HyperSpec/Body/02_b.htm

4:19:52 spiaggia Spawns_Carpet[m]: Sure. It is very hard to give you advice if you are implementing "a Lisp" that is not one that has a specification. Then anything you do is obviously "right" because that's how you have decided that things work in this Lisp.

4:20:46 spiaggia Spawns_Carpet[m]: The best we can do is to tell you how the Common Lisp HyperSpec says that things must be done, since this is #commonlisp, after all.

4:21:26 Spawns_Carpet[m] i am having trouble trying to figure out with the reader algo is an actual concrete idea of what the reader should return in terms of rust data structures

4:21:50 Spawns_Carpet[m] or a c struct or whatever

4:22:16 Bike you should probably just have it return lisp-manipulable objects. whatever that means in your system.

4:22:40 spiaggia Spawns_Carpet[m]: Most code elements in Common Lisp would result in lists in the form of CONS cells, and atoms such as symbols and numbers.

4:22:44 Bike as hayley said, a READ function is often exposed to the lisp, and that's what you're implementing. since it's a function lisp can call, it should return a lisp object.

4:23:44 Spawns_Carpet[m] okay that makes a lot of sense when you say it like Bike, i know exactly what you mean now. I actually debated for a while whether it should return the actual internal representation of lisp data types or not, and asked around elsewhere and some people said to not do it, and others said to do it

4:24:36 hayley Who did you ask?

4:24:50 Spawns_Carpet[m] I asked on discord in the rust server

4:24:58 hayley Well, there's yer problem.

4:25:02 Spawns_Carpet[m] LOL

4:25:46 Bike also, even if you don't expose READ, you will presumably have an EVAL function that takes a lisp object and evaluates it into another object, and you'll want to pass the result of your parse to EVAL, so

4:26:46 hayley I mean, it can help to have some other representation with source tracking information, so that any compile-time errors (for example) can reference where the error is in a file, but that is more complicated, and I don't know what they were thinking of instead.

4:27:07 Spawns_Carpet[m] that is the reason they gave actually

4:28:12 hayley Okay then. But it is more effort, and I wouldn't bother for my first (few) interpreter(s).

4:29:11 Spawns_Carpet[m] to be clear also, the bytecode compiler should be processing the actual lisp objects themselves and not the AST-like thing

4:29:35 Spawns_Carpet[m] since the lisp objects pretty much are the source code I guess that makes sense

4:30:07 Alfr Spawns_Carpet[m], one thing nice in most(?) lisps is that read returns a tree made of conses, that those conses are also directly manipulable using functions in lisp and certain (valid for the language in question) trees can be passed to eval to be treated as code and executed. Also those trees usually are more or less AST you asked for above.

4:34:17 Spawns_Carpet[m] the good thing is that it will be super easy to change the reader to emit actual lisp objects. My "AST" data structure actually was pretty much exactly how I represented the lisp objects anyways, except it had a different name

4:34:30 Spawns_Carpet[m] it also was purely immutable

4:35:32 Spawns_Carpet[m] thanks for the information by the way

4:41:26 Spawns_Carpet[m] what happens when the reader has to read multiple top level lists by the way? should I wrap the entire program in an invisible progn?

4:41:54 Spawns_Carpet[m] or just emit each list on its own

4:43:17 Bike usually a file loader will call READ repeatedly, advancing the input stream/port/whatever you call it as it goes

4:43:23 hayley The reader only reads one form at a time.

5:02:24 Spawns_Carpet[m] is the interpreters representation of lisp objects always going to have to be the same as the lisp objects emitted from the reader? that is the idea right

5:02:43 hayley Yes.

5:03:28 Spawns_Carpet[m] that is strange that the compiler uses the same structure that the interpreter uses, but also really cool

5:03:57 hayley You never mentioned a compiler.

5:04:34 hayley As a rule of thumb, the compiler definitely does not use the list-and-symbol-based representation of Lisp code.

5:04:55 Spawns_Carpet[m] oh shoot now I am confused again

5:05:22 hayley The reader produces a form that can be interpreted, but the compiler usually converts this form into another representation more suitable for analysis.

5:12:51 spiaggia Though, a very simple compiler for a very simple language could probably generate code directly from the S-expression representation of the program.

5:13:14 spiaggia It would have the same structure as the evaluator, but instead of computing a value, it would emit code to compute that value.

5:14:17 hayley Sure.

5:15:53 spiaggia Spawns_Carpet[m]: What hayley is referring to is that most serious compilers for serious languages do a lot of optimizations to the code, and those optimizations typically require some intermediate representation that is neither the S-expressions nor the final instructions (bytecode or native).

5:16:31 Spawns_Carpet[m] that makes sense sure

5:17:42 Spawns_Carpet[m] yeah I am not worried about that right now, needing an IR would be a good problem because that means your language it advanced enough and working well enough to benefit from optimizations !

5:17:54 Spawns_Carpet[m] would be a 'good problem to have'

5:18:27 hayley Depends on your definition of "advanced".

5:18:43 hayley Anything that isn't assembler probably can be optimized.

5:19:38 spiaggia Spawns_Carpet[m]: So maybe you need to think about your long-term goals with this thing. Those goals could determine how much you need to learn about compiler design and what data structures to choose.

7:04:53 jackdaniel uh oh, so many next steps on asdf-devel, my feet are tired

7:04:56 jackdaniel hey o/

7:05:19 spiaggia Hello jackdaniel.

7:14:34 spiaggia jackdaniel: I'll be back on #clim tomorrow or Saturday I hope. This dinky laptop does not encourage participation is very many IRC channels.

7:14:47 spiaggia s/is/in/

7:19:36 jackdaniel sure, it is lively as always :)

7:20:00 jackdaniel but the golden principle of a >9000 engineer is "no news is good news"

7:20:28 jackdaniel (>9000 is a reference to the internet meme involving an anime "dragon ball")

7:52:30 lukego Hey does anyone happen to know why cffi on #+unix tries to load libffi.so.7, libffi.so.6, libffi.so.5? Can't it just use libffi.so? I ask because I need to support libffi.so.8 and it feels a bit cargo-cult to add that as the next special case. This tradition seems to go back at least a decade.

8:18:59 jackdaniel the reason for that is mentioned in the commit message

8:19:02 jackdaniel Use soname with major version for libffi on unix as advocated on

8:19:03 jackdaniel http://lispcaveats.tumblr.com/post/13259176455/ffi-linking-against-shared-libraries.

8:19:36 jackdaniel as of whether it is a valid reason I don't know

8:20:16 hayley Apparently that post doesn't exist, nor does the blog. Thus there are no Lisp caveats!

8:21:11 jackdaniel indeed! :)

8:22:47 jackdaniel and that's a cautionary story to store your rationale notes in the repository ,)

8:22:54 jackdaniel there: https://web.archive.org/web/20130830192134/http://lispcaveats.tumblr.com/

8:26:05 lukego "the" commit message? I went spelunking with 'git log -S libffi' and so on and turned up lots of commits but not a reason

8:26:20 jackdaniel git blame is your friend

8:26:32 jackdaniel then you may introspect individual lines

8:26:44 jackdaniel magit is another handy tool to make it easier

8:27:00 lukego I'm using magit and git blame. user error perhaps, but a comment might be more appropriate than a commit message...

8:27:47 jackdaniel it might, perhaps the commit author would agree with you

8:28:50 lukego ffs, I'm just trying to solve a problem here, preferably in a way that's future proof and helps other people. I don't get what's with all these sphynx-like answers. anyway.

8:29:25 jackdaniel hm, I've helped you to find the rationale, digged the wayback archive, and still I'm getting punched on the face

8:29:28 jackdaniel ACTION gets back to work

11:16:34 theothornhill How do I get (asdf:test-system :system) to yield an exit code of 1 when tests fail in rove? I am using `sbcl --non-interactive --load test.lisp` in a `test.sh` to run on ci, but it alwas passes with error code 0

11:19:56 jackdaniel test-system always returns t

11:20:24 jackdaniel so you need to either use whatever the /actual/ testing framework gives you to run the testing function (and take the result)

11:20:47 jackdaniel or you may try some fancy specialization on the method operate specialized on your system and the test-op

11:21:16 semz shouldn't test-system signal an error if tests fail? or was that just a convention

11:21:20 theothornhill Ah right. Was hoping I could use test-system for this

11:21:36 theothornhill semz: I was thinking the same

11:22:39 jackdaniel it certainly doesn't fail for me when I run 5am tests, perhaps when the framework itself signals an error then test-system doesn't handle it?

11:23:01 theothornhill Now the :perform inside the testsystem is (test-op (o c) (symbol-call :rove '#:run :gql/tests))

11:24:46 jackdaniel the most silly solution (much to my liking) would be: (test-op (o c) (handler-bind ((tests-failed (lambda (c) (signal 'vroom))) (symbol-call …))) and then (handler-case (test-system ...) (vroom () (exit :code 1)))

11:24:50 jackdaniel or something in this spirit

11:25:17 jackdaniel or (if rove signals say tests-failed) then (handler-case (test-system ...) (tests-failed () (exit :code 1))

11:26:54 theothornhill it looks like it just returns two values, one list with successes, and one with fails. So I guess I could just check for nil in the second value

11:27:45 jackdaniel I wonder why test-system doesn't return whatever perform returns

11:28:24 theothornhill Yeah, that must be a bug, or at least a missing feature?

11:31:21 jackdaniel it might be aesthetics - the function returns t on success (instead of - the function returns undefined value(s))

11:31:52 jackdaniel because T was deliberely put in the function body after calling operate

11:32:32 theothornhill hmm. Well, so this works: (multiple-value-bind (foo bar fails) (rove::run :gql/tests)

11:32:33 theothornhill (when fails

11:32:33 theothornhill (sb-ext:exit :code 1)))

11:32:52 theothornhill sorry for the paste

11:34:32 theothornhill But that makes test-system useless, which is a shame

11:36:22 jackdaniel theothornhill: you may always signal some non-serious condition that may be optionally handled from above test-system

11:38:20 theothornhill You mean wrapping the test-system call in a handler-bind or something like that?

11:39:04 jackdaniel even handler-case

11:39:46 jackdaniel but yes, that's what I mean, and signal from operate. then when you call test-system interactively without handlers, it will simply return t

11:39:56 jackdaniel but you will be able to return the exit code from a test script

11:41:09 theothornhill Right, I see what you mean. Yeah, that is better that what I do now, at least

12:00:10 theothornhill jackdaniel: thanks, btw :)

12:09:11 phantomics question about ABCL: my understanding is that it supports some CFFI functionality, but when I try to load a system with a simple CFFI dependency, the dependency breaks. Does CFFI actually not work with ABCL?

12:13:30 jackdaniel theothornhill: sure

12:42:51 lisp123w Does asdf re-search all directories every time one runs asdf:load-system or only on startup?

12:44:15 jackdaniel it doesn't, it caches results and uses timestamps to determine whether something has changed

12:45:13 lisp123w If I create a new .asd after loading SBCL, will it get picked up?

12:45:38 lisp123w (brand new one, not updating an existing one)

12:46:21 jackdaniel asdf will do its best I'm sure ,)

12:49:14 X-Scale` ** NICK X-Scale

12:50:02 lisp123w jackdaniel: great, thanks :)

12:51:59 jackdaniel sure

12:52:27 etimmons phantomics: ABCL needs Maven installed in order to use CFFI

12:53:17 jackdaniel perhaps it is possible for cffi to signal error like "Please install Maven or something" when it is not installed

12:54:51 etimmons I think ABCL does that, asking with a restart to install it, but I forget right now how it's triggered

12:55:03 jackdaniel ah, OK

12:55:05 jackdaniel good to know

12:55:08 etimmons s/asking/including/

12:55:56 phantomics thanks etimmons

12:56:01 jackdaniel (cerror "Whatever.." "Something went wrong!")

12:56:26 phantomics Is there a way to check whether Maven is installed within ABCL? That way I can implement a fallback option in case it's not there

12:59:44 rotateq Java hell :)

13:00:14 scymtym lisp123w: if you create a new system definition file after ASDF has already been initialized, (ASDF:LOAD-SYSTEM "new-system") will signal an error. in the context of that error, there will be a restart CLEAR-CONFIGURATION-AND-RETRY which instructs ASDF to re-scan the filesystem and discover the new system definition file

13:00:25 jackdaniel Hava(na) Jell(y) - a nice codename for an application

13:00:47 rotateq jackdaniel: :D

13:01:15 rotateq and they still think they have real legacy

13:01:49 jackdaniel but they do, systems written in java will be maintained for years to come

13:02:08 jackdaniel even _if_ it loses traction

13:02:13 yitzi Our Jelly in Havana

13:02:52 rotateq wasting many man years of work for nothing over and over :)

13:04:26 etimmons phantomics: I'm pretty sure there is, but I don't recall how. You can likely find it if you require :abcl-contrib and then apropos for maven or mvn

13:04:46 jackdaniel I don't think that Java deserves such harsh opinion, but I'll keep quiet I'm getting into offtopic

13:12:23 puchacz hi, using trivia / optima - can I match the same value in 2 or 3 places in a list?

13:12:46 puchacz pattern like (list :foo x x) is interpreted as an attempt to rebind a variable already bound

13:12:59 puchacz I want to match (:foo 3 3) for example

13:13:51 phantomics puchacz: you'd need to match them separately, like (list :foo (guard first3 (= first3 3)) (guard second3 (= second3 3)))

13:14:12 puchacz phantomics, so if I have 5 places, it means 5 guards?

13:14:21 phantomics Yes, afaik

13:14:28 puchacz thanks

13:14:42 phantomics I went looking for a way in trivia to do things like "match however many elements you find that fulfill this criteria" but couldn't find any