freenode/#lisp - IRC Chatlog
Search
21:07:37
vms14
I have no idea how to convert a string to a list of symbols, I just take the first element
21:09:48
aeth
vms14: So the problem is that you're reading in a line into a string, and you're turning "Hello world" into |Hello world| instead of (HELLO WORLD)
21:11:17
vms14
I'm trying to parse input, I just want read every symbol from the input until the user press enter
21:13:37
aeth
vms14: One thing you could do, and it's quite a hack, is (read-from-string (read-line))
21:13:53
vms14
the thing is usually I won't know how many symbols will be, and the delimiter is the newline
21:14:04
aeth
vms14: the second value in read-from-string is where it left off so you can loop on that second value
21:14:25
pjb
vms14: or: (loop for element = (extract-one-item (read-line)) until (eof-element-p element) collect element)
21:15:03
pjb
vms14: using READ or READ-FROM-STRING, you allow input to do whatver it wants with your lisp image, by default.
21:15:11
aeth
vms14: (read-from-string (read-line)) for a line "hello world" will return (values HELLO 5)
21:16:30
aeth
vms14: you can then do (read-from-string (read-line) nil nil :start 5) to get (values WORLD 11)
21:16:53
pjb
So you would want to bind *read-eval* to NIL. but other reader macros can be problematic: (read-from-string "#8931289312839012*") for example, could DOS your system by trying to allocate all its RAM. (or just signal a condition, depending on the implementation).
21:17:19
pjb
another thing is that reading symbols will intern them, so if there's a loop, the input could fill your memory with useless symbols.
21:17:47
pjb
So you might want to intern the symbols in a throw away package that you can delete-package when you're done.
21:17:47
aeth
vms14: The "correct" (safe) way to do things is to parse the string, perhaps with cl-ppcre
21:18:15
aeth
By the time you add in the validation pjb is talking about, the parse solution probably becomes more concise than the elegant solution that pjb and I both said simultaneously
21:18:57
pjb
vms14: (split-sequence #\space (read-line) :remove-empty-subseqs t) is usually all you need.
21:19:36
aeth
Which to use is debatable. split-sequence is a smaller dependency, but if you're doing additional parsing, you might be using cl-ppcre anyway
21:19:40
pjb
(ql:quickload :split-sequence) (use-package :split-sequence) (with-input-from-string (*standard-input* "Hello world! How do you do?") (split-sequence #\space (read-line) :remove-empty-subseqs t)) #| --> ("Hello" "world!" "How" "do" "you" "do?") ; 27 |#
21:21:23
aeth
If you wanted "absolutely 0" overhead, you can get that. Well, not quite 0, you'd have to track start and end positions for each substring. String/sequence functions take in start and end so you can just work like that.
21:22:50
pjb
(com.informatimago.common-lisp.cesarum.array:positions #\space "Hello world! How do you do?") #| --> (5 12 16 19 23) |#
21:24:58
pjb
(let ((string "Hello world! How do you do?")) (loop :for start := 0 :then (1+ end) :for end :in (com.informatimago.common-lisp.cesarum.array:positions #\space string) :collect (cons start end) :into result :finally (return (nconc result (list (cons end (length string))))))) #| --> ((0 . 5) (6 . 12) (13 . 16) (17 . 19) (20 . 23) (23 . 27)) |#
21:25:07
aeth
You could store positions in an array with the :element-type alexandria:array-index, which will probably round up to fixnum or "unsigned fixnum" (it will show up as some strange looking unsigned-byte size like (unsigned-byte 62)) or (in 64-bit implementations) (unsigned-byte 64)
21:25:43
pjb
And then you can use (foo string :start (car pos) :end (cdr pos)) with most sequence functions to process the substrings. Or (subseq string (car pos) (cdr pos)) when you need to extract it.
21:28:18
aeth
You could also do that as two vectors or two lists, one for start position and one for end position. (I think to make the vector, the best solution would be to walk the string twice, first to get the length for the allocated vectors and then to set the elements)
21:28:21
pjb
vms14: Notice that displaced arrays just abstract those (car pos) (cdr pos) bounds. So instead of subseq, you can use (make-array (- (cdr pos) (car pos)) :element-type (array-element-type string) :displaced-to string :displacement-offset (car pos))
21:31:21
aeth
the alternative is to allocate a list or vector of positions, or, as I recently noticed, two sequences instead of one
21:34:57
aeth
splitting isn't the standard way to think about things, the standard way to think about things is with positions, which is why every built-in (and every well-behaved library) has start/end or start1/end1/start2/end2
21:36:55
aeth
the easiest no-library way to do it is probably read-line and do position tracking, but read-char will probably be the most efficient solution
21:38:38
aeth
Thinking about lists can be done with splitting without a library, but only in one direction, splitting the front parts off and keeping the tail.
21:39:27
pjb
Depending on the size of the string and the substrings, displaced arrays may spare a lot of RAM. However, in the substrings are short, then subseq will be more efficient both in time and space. (eg. on a 64-bit system, we can assumme that strings up to 8 or 16 bytes (2-4 unicode characters) are better created rather than (list* string start end) or displaced arrays.
21:42:07
aeth
vms14 might not need a subseq/displacement at all, if it's about determining what to do based on user commands.
21:43:26
pjb
But don't write the state machine by hand! Write a state machine compiler from a high level description!
21:44:07
vms14
yeah, I want to make a transpiler to c, starting with easy stuff like create a variable, output the value, etc
21:45:56
pjb
vms14: or you may have a look at: https://github.com/informatimago/lisp/tree/master/common-lisp/html-generator
21:46:09
pjb
Have a look at https://github.com/informatimago/lisp/blob/master/common-lisp/html-generator/html-generators-in-lisp.txt
21:46:23
aeth
I have a partially complete GLSL generator so I can already essentially transpile to C if I spent a few weeks on it. Very similar syntax.
21:46:42
aeth
Generally, people avoid the parsing problem altogether when generating another language and just work directly in s-expressions
21:48:42
aeth
vms14: the problem is that 90% of the cases where you'd need parsers in other languages, people just avoid them altogether in Lisps and start with s-expressions, so there's probably less work on parsers than you might expect
21:51:39
aeth
vms14: Almost every "transpiler" in Common Lisp starts with s-expressions. If you don't want to start with s-expressions, you should probably act like you're doing the exact same thing as the normal transpilers and use this as the intermediate format.
21:52:42
vms14
what I had is a function wrapping the input from read-line with parens using concatenate 'string xD
21:52:48
aeth
Lisp itself was written in this way. m-expressions were the next step. https://en.wikipedia.org/wiki/M-expression
21:56:56
aeth
This sort of thing in Lisp is always done in at least two stages, where the first stage parses to s-expressions and the last stage turns a direct (or near-direct) s-expression mapping into strings like (:+ 1 2 3) into "(1 + 2) + 3"
22:00:43
aeth
In fact, + is probably one of the harder ones. Mostly you just go (:foo 1 2 3) to "foo(1, 2, 3)" with the only real difficulty being the way to generate the names (e.g. does foo-bar become "fooBar"?)
22:51:25
grewal
vms14: Wouldn't (read-from-string (read-line)) do what you want (read-delimited-list #\Newline) to do?
23:04:05
pjb
vms14: you could make read-delimited-list #\newline work. For this, you need to copy the character syntax from #\) to #\newline.
23:06:55
pjb
theorically. It stil doesn't work :-( (let ((*readtable* (copy-readtable))) (set-syntax-from-char #\newline (character ")") (with-input-from-string (*standard-input* (format nil "hello world~%How do you do~%")) (values (read-delimited-list #\newline) (read-delimited-list #\newline)))) #| ERROR: Unexpected end of file on #<string-input-stream :closed #x3020025DED1D> |#
23:26:28
vms14
and there are more things I'm missing about format, I need to practice a bit with things like ~:* and so on
23:35:59
vms14
I shouldn't be coding yet, but I want to get used to lisp, and the best way is coding
23:36:51
vms14
grewal: I mean I should be reading and doing test stuff and wait a bit to make this program
23:38:03
vms14
also I still thinking the On lisp book should teach me nice things, but I need to understand lisp better before this book, or I'll miss some important stuff
23:49:30
pjb
vms14: loop is nice because it's versatile. Instead of having loops for, while, until, etc, loop does everything. (loop :while … :do …) (loop :do … :until …) (loop :for i :from 0 :to 10 :do …) and other variants: (loop :while … :do … :until …) (loop :do … :while … :do …) etc.
23:50:45
pjb
vms14: note that the :finally clause is jumped to as soon as one terminating clause is validated. So (loop … :until … :do … :finally …) doesn't evaluate :do when the :until condition is true.
23:54:09
aeth
What makes LOOP good for reading is its behavior for :for ... := ... is different than DO's behavior when you do not have an iteration step. With LOOP, it will do the thing initially and then repeat it, with DO it will only do it once so you wind up having to repeat yourself twice (once for the initial value and once for the step) unless you abstract over this with a custom macro.
23:55:39
aeth
So even if you're primarily using DO and/or DO* in your coding style, this is one of those good exceptions where you should use LOOP
23:56:40
aeth
(correction for the nitpickers, you repeat yourself once, which is writing the same code twice, you don't "repeat yourself twice")
23:58:57
pjb
And it's safe: (with-input-from-string (input " #.(delete-file \"~/.bashrc\")") (read-token-list input)) #| --> ("#.(delete-file" "\"~/.bashrc\")") |#
23:59:27
aeth
vms14: Imo, you shouldn't think in terms of "list of atoms being read from input" imo. That's eval()-style behavior (CL's EVAL is different, and eval("1 + 1") in other languages is closer to (eval (read-from-string "(+ 1 1)")) in CL)
23:59:45
aeth
vms14: You should be thinking in terms of what kind of syntax you want to support, and parsing that syntax.
0:00:16
aeth
None of this that we've been talking about is strictly necessary with a sufficiently restrictive syntax
0:01:53
aeth
e.g. you could require the user write things like "foo 42\nbar 43\n" (replace \n with newlines in your head; IRC is limited to one-line-per-message) in which case you don't technically need any intermediate strings.
0:07:19
pjb
and macros wouldn't be unsafe (and worse, unhygienic) if CL's macros weren't so powerful.
0:10:39
aeth
I guess my point is that for untrusted user input you don't want power, so you wind up having to write your own (or use a library) functionality. Shortcuts here are bad.
0:12:22
aeth
read-line vs. read-char is up to you (unless you *need* to not hang, then you have to use read-char-no-hang)
2:40:39
Arylla
Is there any way to set up SBCL so that it's possible to (attempt to) recover from heap exhaustion by aborting to the REPL?
2:41:12
Arylla
(or from just having a relatively low amount of heap space; I'm okay with having to abort while there's still some heap left)
3:15:53
remexre
coming from Scheme and Haskell (as my most lisplike languages), what does Common Lisp have over e.g. Haskell?
3:20:06
Arylla
Common Lisp is quite similar to Scheme in a lot of ways, so if you're familiar with the comparative advantages of Scheme and Haskell, you'll probably find the comparison between CL and Haskell similar
3:22:00
remexre
also, minor inflammatory statement, a large part of my attraction to scheme is that guile is installed on school machines and ghci isn't
3:23:24
remexre
I've worked on a python project from hell before, which pretty much scared me into types-for-everything-land
3:23:32
estest
remexre: Common Lisp has types and there are a number of multi million line projects created with it.
3:25:10
Arylla
I think that a large part of it is macros and the domain-specific-languageiness of Lisp
3:25:32
estest
There are a lot... one that might be up your ally is simply declaring your types. SBCL will give you warnings at compile time when it detects type issues.
3:26:01
Arylla
when you look at any particular function or something, which is written with macros, it's
3:26:35
Arylla
usually expressed in terms of what the person meant, rather than in terms of whatever constructs the language already supports
3:27:54
Arylla
so that way, it's a lot easier to keep track in one's head of what code is actually trying to do, so a strong, static type system is somewhat less necessary
3:30:00
remexre
but there, there are all sorts of problems getting them to work together (e.g. embedding code in another DSL within some place in yours)
3:30:32
remexre
since the inner DSL might be expecting certain invariants to be true that are broken by the outer one
3:31:54
remexre
wait nvm actually read past the first page and it doesn't describe the type system much :p
3:34:14
Arylla
I'm not completely sure; I haven't had a whole lot of issues with this in Lisp, but then again I've never done very much DSL-based programming in Haskell
3:34:33
Arylla
or really run into such problems before, so it's totally possible that I just don't have that experience
3:35:10
Arylla
also if you want to learn about the details of the type system, the Hyperspec is probably the best resource: http://clhs.lisp.se/Body/04_.htm
3:35:28
remexre
afaik the canonical example is that nondeterministic choice doesn't compose with pretty much any other control flow effect
3:37:21
estest
remexre: The second reference is better for understanding the type system, and the hyperspec will be the most standard-compliant reference... SBCL goes beyond the standard with its type inference (and use of type information to compile faster code). I believe it's based on Kaplan-Ullman inference, like this: https://web.archive.org/web/20181107011706/http://home.pipeline.com/~hbaker1/TInference.html
3:37:50
p_l
remexre: Common lisp is strongly-typed but with static analysis being, unfortunately, mostly an optimization method
3:38:17
p_l
CMUCL pioneered advanced static type derivation in Common Lisp, SBCL derives from CMUCL thus also having it
3:38:53
p_l
one of the problems with CL type system is that it's explicitly turing complete, which means you can't run exhaustive static analysis
3:40:03
remexre
e.g. rust and haskell's type systems are turing-complete, but it's pretty rare to infinite-loop the typechecker there
3:40:26
p_l
remexre: Depends on how you code. For a lot of applicationss, heavy use of classes makes things easier, as classes are also types
3:42:59
Arylla
remexre: hmm I haven't heard much about that before; would you mind linking a resource of some sort which describes the difficulties with nondeterministic choice and other control flow?
3:43:02
p_l
it's however easy to make them form assertions in your code, and debugging facilities even on poorest CL environments are pretty great in comparison
3:44:10
remexre
Arylla: https://wiki.haskell.org/ListT_done_right is basically what I'm talking about
3:49:49
remexre
arguably monads can replace macros as a DSL system most of the time; e.g. http://wall.org/~lewis/2013/10/15/asm-monad.html as a crazy example
3:50:20
remexre
and the free monad is super-nice for being able to write your logic in your DSL, then test it against multiple implementations of the DSL itself
3:50:52
p_l
remexre: I just find it funny that the biggest hack of Haskell became it's most recognizable feature
3:51:24
p_l
though I'll admit that I learnt of how hacky Monads are in uni, few years after I started any experience with Haskell
3:51:33
Arylla
hmm usually in Lisp different macros and other such constructs do tend to play nicely together, as a result of the main method of DSL creation involving substituting in forms provided by the user
3:52:18
p_l
Arylla: the moment you stop pushing for mathetically pure functions, Monad stops making sense
3:54:41
Arylla
and also, macros also allow you to do the part where you write your logic in your DSL and test it against multiple implementations
3:55:38
Arylla
for a big example, look at CLIM (Common Lisp Interface Manager); both McCLIM, an open-source implementation, and implementations derived from Allegro CLIM have the exact same semantics
3:58:12
Arylla
yeah they're basically code that generates code; think more "Template Haskell" than "C macros"
3:59:27
p_l
the main difference in contract of MACRO-FUNCTION from FUNCTION is that MACRO-FUNCTION gets arguments unevaluated
3:59:35
Arylla
except that they're easier to write than Template Haskell, owing to the simpler AST of Lisp
4:00:17
remexre
yeah, I only dipped my toe into TH before running for the hills because of the complexity of the AST :P
4:00:59
remexre
at this point I do things at runtime and spam INLINE and RULES annotations when performance matters