freenode/#lisp - IRC Chatlog
Search
8:17:25
jackdaniel
shka__: you claim that it is not I/O bound, so what is the bottleneck? I don't see cl-csv::check-and-dispatch symbol in said package
8:18:40
shka__
jackdaniel: https://github.com/AccelerationNet/cl-csv/blob/3eba29c8364b033fbe0d189c2500559278b6a362/parser.lisp#L359
8:19:05
Jachy
shka__: You could try fare-csv and compare? I'm curious how slow cl-csv is compared to, say, Python's csv module.
8:19:12
White_Flame
jdz: in some data, we ended up using ASCII field separators from the 0-31 control group
8:19:50
White_Flame
Jachy: when doing some comparisons like that, you never know if the python module is implemented in native code, too
8:20:50
jdz
Oh, maybe what I remember is a field that starts with a double quote, which basically translates to it starting with three double quotes.
8:22:23
shka__
anyway, there is stuff like this https://github.com/AccelerationNet/cl-csv/blob/3eba29c8364b033fbe0d189c2500559278b6a362/parser.lisp#L69 that makes me think that there is some potential for performance improvements
8:23:23
White_Flame
I don't think too many people have had need for high-performance csv reading; sounds like you've got a niche to fill
8:26:12
White_Flame
of course, some of the other libs on cliki might be faster for lack of abstracted features
8:31:06
White_Flame
given that the regex should be able to give you the index of the last character of the match, from which you can parse the next field
8:33:23
White_Flame
of course, if the field starts with a doublequote you'd probably still have to search for embedded quotes and perform a transform
8:33:47
shka__
i will try to tune cl-csv first though, writing my own csv parser is considerable undertaking
9:32:27
scymtym
is there an easy way to use gitlab.common-lisp.net's mandatory two-factor authentication without a phone or additional hardware?
12:53:09
Xach
Hmm, is this ok? (defun starts-with (subseq seq test) (and (<= (length subseq) (length seq)) (every test subseq seq))?
12:53:41
Xach
I've done endless variations of a starts-with thing and I don't remember using this. (e.g. mismatch, :start and :end with string= and stuff, etc)
13:56:57
moldybits
sbcl gives me a style warning for (error "invalid thing: " thing) ... oh, nevermind. :)
15:30:48
phoe
A naïve way causes an error to be signaled: "cannot insert multiple commands into a prepared statement"
16:30:11
dim
you can use my code for that from pgloader at https://github.com/dimitri/pgloader/blob/master/src/utils/read-sql-files.lisp
16:30:46
dim
phoe: the reason why it fails is that Postmodern is using the PostgreSQL extended query protocol which works with a query at a time, with out-of-line parameters (good to avoid SQL injections)
16:31:03
dim
the trade-off is that you then need to make sure in the app that you're sending one query at a time using that protocol
16:32:05
dim
https://github.com/dimitri/pgloader/blob/master/src/utils/read-sql-files.lisp implements a simple hand-written parser for SQL queries delited with a semi-colon (;)... not that simple, it knows about dollar-quoting and other forms of escaping
16:32:58
dim
select E'\';' as ";"; is an example of a corner-case when parsing SQL queries from a file
17:02:14
dim
jackdaniel: http://paste.debian.net/1074118/ if you want to have a look, but don't allocate time for that, I was just curious about building pgloader with ECL that's all
17:02:27
jmercouris
why does (string-downcase nil) return "nil", is there a more sensible function that will actually return nil?
17:03:19
dlowe
strings designators (which include symbols) are converted to strings if they are not strings
17:05:18
Bike
if you want to test whether something is a string and signal an error if it's not, you can do it yourself
17:30:25
pjb
(defun string-downcase⊥ (x &rest keys &key start end) (declare (ignore start end)) (when x (apply (function string-downcase) x keys)))
17:48:36
shka_
i will discuss with maintainer if everything is fine, i was aggressive with my approach so it may be a little bit controversial.
17:50:24
shka_
declaim inline here and there, replaced few defmethod with defun, replaced one class with struct, changed few internal functions
18:08:32
phoe
dim: does it mean that I likely won't get a (postmodern:execute-file #p"/home/phoe/foo.sql")?
18:09:18
phoe
it doesn't make much sense to me - in order to execute a SQL file, I need to either parse and digest it, or drop all the way down to bash to execute "psql < foo.sql"
19:25:05
phoe
dim: https://github.com/phoe/postmodern-execute-file that's a hack that works well for me.