freenode/#lisp - IRC Chatlog
Search
6:14:57
pjb
verisimilitude: but forgetting about EQ frees some neurons for better things, and you don't have to think whether you can use it or not when you write code, and you don't have to ponder whether something special occurs with numbers and characters when you read it.
6:15:09
verisimilitude
You'll mostly be using EQL, though, didi. When in doubt between EQUAL and EQUALP, just use EQUALP if you're too lazy to look at the documentation.
6:15:41
verisimilitude
Sure, but why are you using Common Lisp instead of something else, at that point, pjb?
8:08:36
White_Flame
let-me-google^Wcliki-search-that-for-you.com: https://cliki.net/site/search?query=csv
8:09:16
White_Flame
including the recommended pages, pointing to cl-csv: https://cliki.net/Current%20recommended%20libraries
8:09:31
shka__
White_Flame: do you understand question "what is the fastest correct CSV parser for CL?"
8:10:11
shka__
i know that clicki exists, i know that parser exists, i just want to know if there is something faster cl-csv
8:10:11
jackdaniel
as you have noted, "correct" doesn't have much sense, "practical" is better; and I daresay all published libraries are practical to some degree
8:10:36
jackdaniel
as of "spped" you should benchmark it instead of asking. someone may tell you "this is the fastest" - how do you know it really is? maybe he didn't mbenchmark?
8:12:31
splittist
presumably a csv library has to deal with arbitrary csv, but your problem domain is narrower, so you could tune your own for higher performance (whatever that might mean)
8:13:38
shka__
splittist: i actually need something general, but maybe i can investigate cl-csv performance
8:14:10
White_Flame
I guess if you read-line, then have each field be a displaced-array into that line, that could work
8:14:37
White_Flame
however, if you have quoted fields with embedded quotes, eg "foo \"bar\" baz", you'd still need to re-render that string
8:15:05
White_Flame
presumably copying out those fields is slow, but read-line might also be a performance pig. You'll have to profile
8:15:51
White_Flame
also if you have quoted fields with embedded newlines, that mucks with using read-line cleanly
8:16:21
jdz
Also the strings that are enclosed in double double quotes so that the double quotes inside don't have to be escaped?
8:16:57
shka__
furthermore, even if i read the whole file into string and build stream on top of that, it does not improve performance dramaticly
8:17:25
jackdaniel
shka__: you claim that it is not I/O bound, so what is the bottleneck? I don't see cl-csv::check-and-dispatch symbol in said package
8:18:40
shka__
jackdaniel: https://github.com/AccelerationNet/cl-csv/blob/3eba29c8364b033fbe0d189c2500559278b6a362/parser.lisp#L359
8:19:05
Jachy
shka__: You could try fare-csv and compare? I'm curious how slow cl-csv is compared to, say, Python's csv module.
8:19:12
White_Flame
jdz: in some data, we ended up using ASCII field separators from the 0-31 control group
8:19:50
White_Flame
Jachy: when doing some comparisons like that, you never know if the python module is implemented in native code, too
8:20:50
jdz
Oh, maybe what I remember is a field that starts with a double quote, which basically translates to it starting with three double quotes.
8:22:23
shka__
anyway, there is stuff like this https://github.com/AccelerationNet/cl-csv/blob/3eba29c8364b033fbe0d189c2500559278b6a362/parser.lisp#L69 that makes me think that there is some potential for performance improvements
8:23:23
White_Flame
I don't think too many people have had need for high-performance csv reading; sounds like you've got a niche to fill
8:26:12
White_Flame
of course, some of the other libs on cliki might be faster for lack of abstracted features
8:31:06
White_Flame
given that the regex should be able to give you the index of the last character of the match, from which you can parse the next field
8:33:23
White_Flame
of course, if the field starts with a doublequote you'd probably still have to search for embedded quotes and perform a transform
8:33:47
shka__
i will try to tune cl-csv first though, writing my own csv parser is considerable undertaking
9:32:27
scymtym
is there an easy way to use gitlab.common-lisp.net's mandatory two-factor authentication without a phone or additional hardware?
12:53:09
Xach
Hmm, is this ok? (defun starts-with (subseq seq test) (and (<= (length subseq) (length seq)) (every test subseq seq))?
12:53:41
Xach
I've done endless variations of a starts-with thing and I don't remember using this. (e.g. mismatch, :start and :end with string= and stuff, etc)
13:56:57
moldybits
sbcl gives me a style warning for (error "invalid thing: " thing) ... oh, nevermind. :)
15:30:48
phoe
A naïve way causes an error to be signaled: "cannot insert multiple commands into a prepared statement"
16:30:11
dim
you can use my code for that from pgloader at https://github.com/dimitri/pgloader/blob/master/src/utils/read-sql-files.lisp
16:30:46
dim
phoe: the reason why it fails is that Postmodern is using the PostgreSQL extended query protocol which works with a query at a time, with out-of-line parameters (good to avoid SQL injections)
16:31:03
dim
the trade-off is that you then need to make sure in the app that you're sending one query at a time using that protocol
16:32:05
dim
https://github.com/dimitri/pgloader/blob/master/src/utils/read-sql-files.lisp implements a simple hand-written parser for SQL queries delited with a semi-colon (;)... not that simple, it knows about dollar-quoting and other forms of escaping
16:32:58
dim
select E'\';' as ";"; is an example of a corner-case when parsing SQL queries from a file
17:02:14
dim
jackdaniel: http://paste.debian.net/1074118/ if you want to have a look, but don't allocate time for that, I was just curious about building pgloader with ECL that's all
17:02:27
jmercouris
why does (string-downcase nil) return "nil", is there a more sensible function that will actually return nil?
17:03:19
dlowe
strings designators (which include symbols) are converted to strings if they are not strings
17:05:18
Bike
if you want to test whether something is a string and signal an error if it's not, you can do it yourself
17:30:25
pjb
(defun string-downcase⊥ (x &rest keys &key start end) (declare (ignore start end)) (when x (apply (function string-downcase) x keys)))
17:48:36
shka_
i will discuss with maintainer if everything is fine, i was aggressive with my approach so it may be a little bit controversial.
17:50:24
shka_
declaim inline here and there, replaced few defmethod with defun, replaced one class with struct, changed few internal functions
18:08:32
phoe
dim: does it mean that I likely won't get a (postmodern:execute-file #p"/home/phoe/foo.sql")?
18:09:18
phoe
it doesn't make much sense to me - in order to execute a SQL file, I need to either parse and digest it, or drop all the way down to bash to execute "psql < foo.sql"