freenode/#lisp - IRC Chatlog
Search
9:15:55
beach
I keep forgetting about DEFSTRUCT since I have absolutely no intention of using it in SICL code. :)
9:16:32
beach
And even if I *did* use it, it would have the absolute same representation as if I had used DEFCLASS.
9:17:45
dmiles
yeah in my impl DEFSTRUCT and DEFCLASS become the same representation .. jsut with DEFSTRUCT i create extra support of slot numbers
9:42:18
dmiles
rather than doing what i and many others do.. chekcing for environmental overrides and then going to symbol
10:07:32
beach
In fact, I save some space that way, because very few symbols will have a non-empty property list, so no need to represent them in the environment. In a typical implementation, this cell is allocated in every symbol, whether needed or not.
10:08:12
beach
The same thing goes for the function cell, the value cell, etc. In SICL, a symbol has only a NAME and a PACKAGE.
10:10:08
beach
I can even imagine some implementations having two function cells in every symbol, one for the name itself and one for (SETF <name>).
10:10:58
beach
Worse (in terms of maintainability), some implementations have a function cell for the name itself, but store the SETF name somewhere else.
11:16:01
sirkmatija
Hello, I am building a small web game and I am wondering how to serve a file (not a string) by ningle. Thanks for your help.
11:21:23
sirkmatija
I tried reading file into string and then serving it, but then it doesn't find js script file, even if I provide full (not relative) path
12:17:33
JuanDaugherty
sirkmatija, y u gotta use that no name thing when everybody wants you to use the lisp majors (hunchentoot, etc)?
12:28:36
jackdaniel
if you are more interested in shipping whole directory of files consider using (:static :path "/public/" :root #P"/my/dir/") as parameter to lack:builder (instead of just making ningle:<app> instance)
12:33:36
sirkmatija
thank you, it serves the file now, but the script isn't found - when i inspect source it say URL /script.js was not found on this server.
12:34:35
jackdaniel
are you sure it is under /script.js ? not for instance my-host:8080/my-app/script.js ?
12:35:40
jackdaniel
computers are not magic, usually they do exactly what you tell them to do (and when they *do* magic for you, then it is usually a game over [opinionated])
12:36:38
sirkmatija
I thought I don't have to serve the files separately if they are in same dir, a stupid mistake
13:10:37
Shinmera
Just a quick announcement, the emacs package to interact with plaster.tymoon.eu is now on Melpa: https://melpa.org/#/plaster
13:30:21
beach
Shinmera: I don't use "fukamachiware", because it doesn't solve the problems I want to solve, but from what I hear, people are very impressed with it, so I assumed it was high-quality stuff. Not so?
13:35:17
makomo
oh beach, i have a couple more typos to report (hopefully you won't mind me snooping around :-))
13:36:01
beach
makomo: Not at all. If stuff is freely available, it is there for you to read and comment on.
13:37:18
makomo
beach: http://metamodular.com/Software-engineering/developer-productivity.html "First, let is"
13:37:30
makomo
beach: http://metamodular.com/Psychology/continuous-improvement.html "I use computer every day"
13:39:02
beach
makomo: Let's say it's currently the only one. But who knows, maybe loke and others will create ALS one day.
13:39:32
makomo
beach: http://metamodular.com/Essays/you-know-you-have.text "you use more then" (not sure why i read this one but there you go :-))
13:41:25
makomo
beach: i agree very much with your essay on pyschology and people being inefficient at the stuff they do (talking about software specifically)
13:42:50
makomo
however, i managed to get one of my uni friends to try out emacs and he's been actually using it for the last week or so
13:43:52
makomo
i've helped him install slime and he's reading pcl currently (started with land of lisp but then i intervened)
13:48:03
makomo
beach: yeah, true. i figured that it's good to try to spread the word and work on "recruiting" new people into the whole hacker culture
13:48:45
makomo
there's only so much you can do of course, but still spreading the word is part of "giving back" imo
13:51:23
makomo
as much as i'd like to share knowledge, you still have to pick people who you're sharing with carefully
13:52:32
Shinmera
ACTION feels like he's wasting his effort on this channel more often than he'd like to admit
13:54:37
beach
makomo: Now if you could snoop around in my software documentation as well, that would be great! :)
13:56:00
makomo
beach: but i do like your approach to documentation -- nicely written and formatted whitepapers, etc.
13:57:10
makomo
of course that's just my opinion, but i had a weird feeling when i was reading LL (i didn't finish it completely though)
14:51:11
dim
code is at https://github.com/qitab/qmynd/blob/master/src/mysql-protocol/response-result-set.lisp#L197
14:52:11
dim
I have column-type 252, which should be eql to (defconstant +mysql-type-blob+ #xfc) I think, and (column-definition-v41-packet-cs-coll column-definition) is 63, which should be eql to (defconstant +mysql-cs-coll-binary+ 63), and, sorry what?
14:54:10
dim
since https://github.com/qitab/qmynd/commit/39e17d0fdcf111f7b081d8aff316af084476f198#diff-a7f8d7f40f13983046fa6d0c5bfb2272
14:56:02
beach
I suppose the logic behind how CASE works is that the compiler should be able to build a hash table, perhaps with perfect hashing, to make the choice real fast.
14:56:12
dim
(case (#. (list +mysql-type-string+ +mysql-type-var-string+ +mysql-type-tiny-blob+ +mysql-type-medium-blob+ +mysql-type-long-blob+ +mysql-type-blob+) ...)) doesn't seem to cut it
14:56:44
schweers
Does anyone use cl-launch with CCL? I’ve been using cl-launch with sbcl for a few months now and wanted to try my code on CCL too. When I start a repl, load and run my code from there it works. So thats a good start. When I use cl-launch I am dropped in a repl too (which doesn’t happen with sbcl).
14:57:06
jackdaniel
alexandria has switch, but it doesn't allow specifying many elements (just one, but it is evaluated)
14:57:30
beach
jackdaniel: That is not enough if the CASE would take longer if a hash table were used.
14:57:41
schweers
also, when I try to run my code from this cl-launch induced repl CCL bails out because the heap is exhausted (which does not happen when I use a repl directly).
14:58:09
dim
but the #. (list +...+ ) doesn't seem to work, so either I tried it wrong again, or maybe I am testing the code I didn't edit
15:00:31
beach
I suppose if the labels are rational numbers, a technique similar to my fast generic dispatch could be used. That could be a big win.
15:01:26
jackdaniel
sbcl makes constants available at read time (if defined at the same file), ccl doesn't - both behaviors are conforming
15:01:46
jackdaniel
to make code portable (while having defconstant) using eval-when is the solution
15:02:04
beach
_death: Yeah, I wasn't going to bring it up, because I saw no trace of the definitions in the same file, so I assumed they existed already.
15:03:02
beach
With 11 cases, it could definitely benefit from a technique like my fast generic dispatch, provided the entries are numbers, of course.
15:03:58
_death
a tree case analysis or some perfect hash-table thingy may work, if it's even needed (profile first)
15:05:31
beach
_death: A tree (or rather binary search using constants) is what my fast generic dispatch uses, simply because it is faster than trying to store some data structure in memory.
15:07:01
beach
It makes sense even when the number of cases is large, because the number of comparisons only grows logarithmically.
15:07:27
beach
So you would need thousands of cases in order for it to be slower than a few memory accesses.
15:08:18
beach
That's what I am saying. My technique does not access memory, but a perfect hashing scheme would.
15:09:37
dim
I don't think that code path is the most prominent to fix in qmynd btw, but it'd be nice to improve it some more
15:10:47
beach
This discussion did give me the idea that I should implement CASE like that in SICL (for the case of real numbers).
15:12:11
_death
beach: it reminds me of pkhuong's string-case.. and I wouldn't be surprised if I encountered more case operators like it in the past
15:12:14
dim
by the way with the fix the MySQL read now takes 2s while writing to PostgreSQL takes the 27s (in another thread)
15:12:45
dim
so my current performance problem is to do with format-vector-row if some of you guys want to play around
15:13:27
dim
dead easy to improve, but last I tried I broke the whole of pgloader, for wanting to do too many optimizations at once
15:18:14
_death
should split it into several functions.. and btw you can (reduce ... :key ...) instead of (reduce ... (map ...))
15:29:06
dim
I must have also considered :key for class/struct slot accesses, in my mind, but the API is way more generic than that
15:30:28
dim
then there's a huge optimisation (I think) to be done here by only considering UTF-8 convertions for text based datatypes
15:31:28
dim
and then another idea is to avoid batching altogether in the db-2-db case for pgloader, as error handling then should be taken care of at the source db rather than with a reject file
15:40:21
dim
is there any point in building a single vector of bytes compared to a vector of vector of bytes, on a pure performance standpoint?
17:07:51
rumbler31
with (setq *print-radix* t), (CCL:MAKE-SOCKET :REMOTE-HOST "beta.quicklisp.org" :REMOTE-PORT 80) fails with bt https://pastebin.com/1ZQK1qQu
17:12:51
rumbler31
looking to run the fast-io tests. I'm not familiar with many testing frameworks yet, but it doesn't look like an asdf test-op is defined for fast-io. How do I go about... running the tests?
18:35:17
flip214
Can I pass multiple :long-name options to CLON:DEFSYNOPSIS, so that any one of the long names can be used? Ie. aliases?
18:38:45
sjl
rumbler31: as far as I know that's the intended way to run tests for an ASDF system. if fast-io does it differently, they should document it.
18:39:20
rumbler31
so i've been looking for machinery for that, I expect that the test-op function to call would be explicitly defined in the .asd file right?
18:42:49
sjl
the define the tests in a separate system, like a lot of systems do, to avoid having to load the unit test framework to use the main system
18:43:07
sjl
usually when you do that, you add an :in-order-to to the main system like this https://github.com/sjl/cl-digraph/blob/master/cl-digraph.asd#L11
18:45:01
sjl
and then does (checkl:define-test-op :fast-io :fast-io-test) to I think add the in-order-to?
18:45:46
sjl
I dunno, it's weird and non-standard and won't work like most people expect -- (progn (ql:quickload :fast-io) (asdf:test-system :fast-io)) won't work without other stuff
18:47:21
Xach
I also find the way to set it up a bit cumbersome. Kind of a throwback to the (asdf:oos 'asdf:load-op ...) days.
18:50:08
rumbler31
so the answer seems to be that the authors usage of checkl in the -test.asd means that the correct answer to run tests is to invoke (run-all). Although the output doesn't suggest success or failure on my system. Thanks all
19:03:54
rumbler31
minion: memo for antonv: theoretically, if you wish to send a udp packet to a multicast address, you shouldn't need any more os support. The magic comes when you try to tell the os that you care to receive packets from certain multicast addresses
19:05:35
rumbler31
minion: memo for antonv: there were some patches for ccl on the mailing list long ago that added support for multicast sockets. Sending to multicast addresses *may* *just work* though
20:39:30
dim
ok, I simplified https://github.com/dimitri/pgloader/blob/master/src/pgsql/copy-format.lisp a lot, made several simpler functions and all, and I won some low-% of perfs at that
20:39:50
dim
so now that it's easier to review, I hope some perfs guru will get interested into either giving advices or sending patches ;-)
20:46:18
AeroNotix
Wonder if simple application of type declarations and safety 0 would get anything.
20:47:34
AeroNotix
dim: why not just filter the list of funs prior to the loop? https://github.com/dimitri/pgloader/blob/master/src/pgsql/copy-format.lisp#L52
20:50:42
dim
AeroNotix: bencmarking consists of using pgloader to load data into PostgreSQL or migrate a whole database from MySQL, the function format-vector-row is called with every single row that's loaded
20:51:14
dim
but anyway, some people had comments earlier on the previous version of the function, that I addressed, and it was good comments ;-)
20:51:38
AeroNotix
dim: Yes I know, i was reading before, thought I'd read through it now you've cleaned the code up. Before it was rather unwieldy to read.
20:55:56
AeroNotix
dim: does it need to be modified in an in-memory vector rather than streaming directly to file?
20:56:53
dim
the idea is that we use the high performance COPY protocol of PostgreSQL, which allows streaming data (in and out of the database)
20:57:09
dim
the thing is that COPY is (of course) transactional, so any error aborts the whole transaction
20:57:35
dim
pgloader works around transaction rollback and replay the good input, rejecting the bad one, so that you can get your data in
20:58:07
AeroNotix
ohh, ok. So maybe to retain that feature you can't stream it directly. Might be a good option to provide though :)
20:58:22
dim
when that happens we have ready-to-go data in memory, and also we know that the COPY offset reported in the PostgreSQL is the array offset in the batch
20:59:07
dim
now I'm thinking that in the case when you migrate from a source database to PostgreSQL, you don't want any reject file, you'd rather --on-error-stop (the option exists)
20:59:28
dim
and then you either fix your load command (casting rules, mostly), or fix the input database
21:00:38
dim
so I'm thinking that --on-error-stop should 1. be the default if the source of the data is a database server and 2. make it so that we don't use the in-memory batches at all but rather stream the data to PostgreSQL directly
21:02:12
AeroNotix
I've come across pgloader before a few times. Glad to see it's still being worked on.
21:03:36
dim
well the White Paper has been downloaded some already, which proves interest, and also that's tooling for the Enterprise, not much use at home
21:03:40
AeroNotix
dim: you already have a good place to filter functions here: https://github.com/dimitri/pgloader/blob/3bb128c5db9fc7b8926586113aa1b97c59a7b457/src/pgsql/copy-from-queue.lisp#L124-L126
21:03:58
AeroNotix
I don't see why https://github.com/dimitri/pgloader/blob/3bb128c5db9fc7b8926586113aa1b97c59a7b457/src/pgsql/copy-format.lisp#L52 needs to be conditional
21:04:46
dim
I could use #'identity I guess, but I'd be surprised if it came to a benefit in terms of performance
21:05:30
AeroNotix
I was going to mention #'identity. My thinking is that it's part of your inner loop, could be ran quite a lot on a huge input
21:11:05
AeroNotix
didn't read the whole scrollback but is using a vector like this optimal? Even ignoring what i mentioned before about streaming directly to PG, is it not faster to stream bytes to an in memory buffer instead of writing bytes to vector offsets and incrementing an index?