Search
Friday, 5th of July 2019, 16:14:37 UTC
18:34:14
faheem
makomo: Weird. I kind of came here to ask exactly that questoin.
18:34:40
faheem
The only thing I'm aware of is a library that Tamas Papp wrote. But it's kind of abandoned.
18:34:42
makomo
faheem: hah :-). have you found and tried any?
18:34:51
faheem
And I'm not sure how usable it is.
18:35:15
faheem
makomo: Haven't you come across it? It's the topic hit for -> data frame common lisp
18:35:34
makomo
yeah, i found it, but was wondering whether you tried it
18:35:38
makomo
this is another one https://github.com/blindglobe/rho
18:36:10
faheem
makomo: No, I haven't. It didn't exactly seem developed. Tamas apparently decided that Julia was a better better bet.
18:36:22
faheem
I'm not exactly a regular CL user, but I'd like to use it more.
18:37:13
faheem
makomo: Did you find anything that seemed at all usable. Something like Pandas seems hopeless, unfortunately.
18:38:13
makomo
at a glance both of these seem kind of usable, but both are underdocumented and not really finished, so, nope :(
18:38:24
faheem
Ah, blindglobe is Tony Rossini. The statistician.
18:38:42
faheem
I think he was trying to developed CL for statistics. Looking.
18:39:16
makomo
yup, just found that -- https://github.com/blindglobe/common-lisp-stat
18:40:19
makomo
there's also ann -- https://groups.google.com/forum/#!topic/lisp-stat/AN41IBIR59w
18:40:26
makomo
"ann" being the name of the dataframe lib
18:40:42
makomo
https://github.com/mirkov/data-table/
18:40:48
shka_
what you are looking for?
18:40:51
makomo
but it looks abandoned as well
18:40:54
makomo
shka_: a dataframe library
18:41:13
makomo
as in R's or Pandas' dataframe
18:41:17
faheem
See also -> https://github.com/blindglobe/common-lisp-stat/commit/eaddd8205c840dd0fe34f06c9bd03cbac698e611
18:41:20
makomo
does your suck as well? :)
18:41:29
shka_
makomo: kinda, it is fresh
18:41:34
faheem
"experimenting with Tamas' cl-data-frame"
18:42:13
shka_
and cl ecosystem has some mostly lackluster CSV libs which makes things harder
18:42:27
faheem
shka_: Oh dear. Everyone writing their own is kind of a guarantee that nothing with ever be complete enough to use.
18:42:46
shka_
https://github.com/sirherrbatka/cl-data-frames
18:42:56
shka_
but, as i said, nothing was really any good
18:43:14
shka_
this one solves few critical points at the very least
18:43:40
faheem
shka_: 384 commits is better than some. Of course, it depends on the commits. Do you have tests?
18:43:54
faheem
And have you used it for real work?
18:44:08
shka_
and yes, i use it quite a lot
18:44:17
makomo
shka_: does yours support stuff like joins/merges/group-bys?
18:44:21
faheem
shka_: Was the Tamas one no good?
18:44:38
shka_
faheem: could you please give me a link to it?
18:44:55
makomo
https://github.com/tpapp/cl-data-frame
18:45:18
shka_
makomo: no joins, no merges, it leans back on the cl-data-structures though so group by just works
18:45:49
faheem
shka_: What makomo said.
18:46:46
shka_
yeah, it is another lib, has some data structures (not relevant here) and operations that are supposed to work on abstract interfaces
18:47:28
faheem
shka_: Are you talking about tpapp's library?
18:47:42
shka_
no, i am talking about my own lib
18:47:50
shka_
give me a second, i will show you how this works
18:49:41
shka_
https://github.com/sirherrbatka/cl-data-structures/blob/ef1ad2064804759d65b0bd6f4520501fa40c4f5c/src/algorithms/meta/meta-tests.lisp#L48
18:50:11
shka_
ok, so there is this abstraction called ranges, and there is abstraction called aggregation functions
18:50:15
faheem
shka_: What kind of printing/display do you have?
18:50:33
faheem
I know that's kind of trivial, but I tend to be visual.
18:50:41
shka_
faheem: none, currently, i am planing of making viewer in CLIM
18:50:54
shka_
which seems to be the best option nowdays
18:51:06
faheem
shka_: I was just thinking of an ascii one. Like R or Pandas.
18:51:30
shka_
ascii is fine, but I strongly prefer having GUI
18:51:31
faheem
makomo: Are you still there?
18:52:05
faheem
shka_: If I might interrupt, how does yours compare to the tpapp one?
18:52:09
makomo
this indexing business is pretty neat. tpapp has a library related to that https://github.com/tpapp/cl-slice
18:52:13
shka_
anyway, idea is that aggregation functions work on ANY range so if you just manage to implement the right protocol it simply chew it trough fine
18:52:45
shka_
i have stuff like line-by-line range so you can work with text files this way
18:53:01
shka_
and defining aggregation functions is actually easy to it is extendable
18:53:01
aeth
For CL GUIs you can use Sketch, but you'd have to write your own way to render graphs/etc. to it. https://github.com/vydd/sketch
18:53:05
faheem
aggregation reminds me of SQL.
18:53:31
shka_
anyway, back to data frames
18:53:33
makomo
faheem: that's pretty much what i want to use dataframes for, and yeah, the two are very related
18:53:40
makomo
see https://pandas.pydata.org/pandas-docs/stable/getting_started/comparison/comparison_with_sql.html
18:53:51
faheem
Ideally one would be able to write to some kind of abstract canvas.
18:54:27
shka_
so my implementation is very basic at the moment, and main focus was on core ideas like mutability and the right API
18:54:39
shka_
but i find it personally useful
18:54:43
faheem
shka_: Ok. I hope you don't abandon yours as well.
18:54:53
faheem
That seems to be the fate of CL libraries, sadly.
18:54:53
shka_
no, i am still working on it
18:55:09
faheem
shka_: Good for you. So, have you looked at the tpapp library?
18:55:20
shka_
but before i can put it in the QL i will have to stabilize cl-data-structures
18:55:20
faheem
Don't mean to pester you...
18:56:14
shka_
faheem: yeah, i did look at it, does not support sparse columns which is deal breaker for me
18:56:15
faheem
is Quicklisp pretty open, in the sense that anyone can put anything in there?
18:56:41
faheem
shka_: Ok, but overall how does it compare to yours? Less developed or just different?
18:57:03
faheem
You should see if you could take features/ideas from there. All this replication is very inefficient.
18:57:14
shka_
it is simpler, i had very specific set of features i wanted to have
18:57:21
faheem
And I don't know what sparse columns are.
18:57:39
faheem
shka_: Ok. Based on what? R / Pandas / something else / nothing?
18:57:42
Xach
faheem: There's a process that I moderate and most stuff goes in but some stuff does not.
18:57:53
shka_
faheem: so the thing is that i have HUGE data sets filled mostly with nulls
18:58:08
faheem
shka_: Right, sparse data sets.
18:58:11
shka_
they literally won't fit in memory unless you are smart about it
18:58:28
faheem
shka_: What kind of data?
18:58:29
shka_
QL is pretty open but i don't want to put garbage in it
18:58:56
shka_
faheem: mostly medical stuff
18:59:06
Xach
I don't consider the garbage-ness of a project when looking at a request to add something.
18:59:07
faheem
Ah. Bioinformatics, or something else?
18:59:18
faheem
Xach: Do you require tests?
18:59:44
Xach
faheem: no. it must build on sbcl on linux, though.
18:59:53
Xach
that means it compiles without (full) warnings
19:00:03
shka_
Medical, claims. Some bright mind figured that ideal way to represent the diagnosis is to store them in 20 seperate columns
19:00:09
faheem
shka_: You said the kpapp library does slicing.
19:00:30
faheem
shka_: Ah, so patient data. Got it.
19:00:43
faheem
Probably not too large, thne.
19:00:55
shka_
faheem: very large actually
19:01:16
shka_
mostly filled with nulls, but still
19:01:26
faheem
that must be a lot of patients
19:01:41
shka_
so as you would imagine i love me some destructive operations
19:01:56
faheem
shka_: Does it do slicing too?
19:02:30
shka_
well, you can slice columns subset if you want to, you can filter rows as well
19:02:32
makomo
faheem: regarding duplication of work :) https://fare.livejournal.com/169346.html
19:02:49
faheem
makomo: Yes, i read that.
19:03:37
faheem
It's definitely a real problem with CL. A dozen libraries for the same thing. None of them usable.
19:04:05
shka_
as I said, nothing really cut it for me so i rolled my own
19:04:22
faheem
shka_: Do you have a mailing list or something? Can you shoot me an email when you project is on QL?
19:04:27
shka_
but if you are willing to try it, i will have more input to consider the future development
19:04:30
faheem
Or I could test it earlier, I suppose.
19:04:55
faheem
And perhaps makomo would test it too.
19:05:18
faheem
shka_: Sure, I could try it. Is it usable now, then?
19:05:24
makomo
maybe i'll give it a go, but i really wanted the ability to do joins
19:05:37
faheem
makomo: Would it be hard to add?
19:05:49
shka_
makomo: sorry, i will be working on it in the future
19:06:15
shka_
yeah, it is on my todo list
19:06:18
makomo
faheem: i'm not sure, maybe?. i'm not familiar with the code so i have no idea. :)
19:06:19
faheem
BTW, where are you guys? Bombay, India, here.
19:06:37
makomo
it's not your fault anyway :D
19:06:48
faheem
Data frames aren't actually complicated, as things go.
19:06:56
faheem
Of course, you have to take the time.
19:07:16
shka_
faheem: if you have enough of wiggle space to be inefficient
19:08:23
shka_
anyway, here is the frames library: https://github.com/sirherrbatka/cl-data-frames but you will need also this (which is way to big) https://github.com/sirherrbatka/cl-data-structures and this one (which is pretty small) https://github.com/sirherrbatka/lense
19:08:45
shka_
lense will probably be soon in the QL
19:09:40
shka_
cl-data-structures is kinda a mess because it has some stuff like statistical estimators that don't have any business being in there
19:09:59
makomo
yeah, i saw the standard-deviation thing :)
19:10:20
faheem
shka_: It's quite large.
19:10:29
faheem
Did you write the whole thing yourself?
19:10:39
shka_
i will probably have to modularize it
19:10:57
faheem
shka_: I see you forked it from something else.
19:11:24
shka_
long story, but this is pretty much everything written by me
19:11:40
faheem
shka_: So not a fork, really?
19:11:52
faheem
Again, where are you guys located? Not that it really matters.
19:12:52
shka_
anyway, i would be thrilled if you guys want to try my stuff out and maybe help me a little
19:13:12
faheem
shka_: I take it you're using this for contract work, or something?
19:13:30
faheem
Judging by the patient data thing.
19:13:46
faheem
shka_: Sure, I could give it a whirl. But my CL is beginnerish and rusty.
19:13:47
shka_
data-frames work although are rough around the edges, cl-data-structures are genuinely useful as fuck
19:13:52
faheem
Do you want my email address?
19:14:16
shka_
you can write issue on github if you have ANY question
19:14:28
shka_
i will do my best to answer
19:14:42
faheem
shka_: Ok, will do. Unfortunately, your library isn't high on Google.
19:14:53
faheem
Putting it on QL should help with visibility.
19:15:05
faheem
How many data frame libraries are there already?
19:15:18
shka_
makomo: sounds about right
19:15:21
faheem
shka_: It's not something you can control.
19:15:41
makomo
not sure if they're on QL
19:15:56
shka_
i really need to clean up cl-data-structures and modularize it before i put on github
19:16:03
makomo
https://github.com/blindglobe/rho, https://github.com/mirkov/data-table/, https://github.com/tpapp/cl-data-frame, https://github.com/sirherrbatka/cl-data-frames/
19:16:25
faheem
shka_: Isn't it already on github? Do you mean QL?
19:17:31
shka_
anyway, try aggregation functions from cl-data-structures as well
19:18:05
shka_
probably the best thing i ever made with CL
19:18:15
shka_
and the most useful as well
19:19:22
faheem
shka_: BTW, in terms of performance and reliability, does CL work well for you?
19:19:30
faheem
As in, are you able to do real work with it?
19:19:48
faheem
Not many people use it for numerical work, I think.
19:19:59
faheem
shka_: So it doesn't crash a lot, or anything like that?
19:20:08
faheem
And are you using SBCL, CCL, or something else?
19:20:38
shka_
it is pretty good, GC is the only weak point i ran into
19:21:20
shka_
i run this on ubuntu, arch linux and whatever admin decided to put on the server
19:21:29
faheem
shka_: maybe what? maybe Debian?
19:21:37
shka_
maybe debian, maybe red hat
19:21:59
faheem
So GC is not so great? Too much memory use?
19:22:22
faheem
Or it doesn't give back fast enough?
19:22:45
shka_
it works fine, but huge heaps sometimes bug it out
19:23:08
shka_
if you need just 8GBs of ram you are fine
19:23:17
shka_
sometimes, i need a lot more
19:23:19
faheem
shka_: Oh. That's not much.
19:23:56
shka_
well, it is not painful enough for me to stop and i seem to be the only person with such problems so maybe it is fine
19:24:18
faheem
shka_: I doubt you're the only person. Do you file bug reports?
19:26:45
shka_
faheem: don't let this discourage you from sbcl though
19:26:59
faheem
shka_: I've used it before, but not recently.
19:27:13
faheem
I remember having problems with GC. I did file bug reports, too.
19:27:19
faheem
At least one, that I remember.
19:27:42
shka_
i know i should, doing this in a productive way would require a lot of time, and i have way to many things on my hands already
19:27:51
faheem
For some reason I have https://bugs.launchpad.net/sbcl/+bug/1446962 in my browser.
19:27:56
faheem
Not sure where I got the link from.
19:30:31
faheem
shka_: Well, sleep time. Nice talking to you.
19:32:03
faheem
shka_: BTW, did you see this? http://within-parens.blogspot.com/2016/12/rcreational-common-lisp.html
19:32:46
shka_
faheem: nope, i like some aspects of R though
19:32:57
shka_
unlike the abominations that pandas is
19:35:01
slyrus_
is cffi supposed to find things in /usr/lib/x86_64-linux-gnu ?
19:38:00
slyrus_
the real problem being that I can't load sdl2-ttf
19:39:24
PuercoPop
shka_: Have you seen https://github.com/fiddlerwoaroof/data-lens? How does it compare to lense?
19:40:57
shka_
PuercoPop: it is not the same stuff, all i really wanted is to have direct equivalent of racket style lenses, this goes for full blown DSL build with monads
19:41:18
shka_
which i don't need, because cl-data-structures ranges and aggregation functions are way more useful
19:42:45
shka_
so obviously, my lens is simply a first class accessor
19:43:10
shka_
and here is something way more complex
19:54:00
shka_
well, good night everyone
20:27:34
pfdietz
faheem: I am interested in improved curation of the Quicklisp ecosystem. But time is finite.
22:00:05
makomo
huh interesting. if there's a bound global dynamic variable *VAR*, doing (defvar *var* (error "hello")) makes it unbound (using SBCL). is this expected?
22:07:34
zulu-inuoe
makomo: I'm not seeing that behaviour. Are you sure the variable wasn't already unbound?
22:07:53
makomo
zulu-inuoe: lemme try again in a fresh image
22:08:49
makomo
zulu-inuoe: do you get an error when executing the form?
22:09:44
zulu-inuoe
makomo: Not on the second defvar, since the variable was already created
22:09:59
makomo
must have been a package mixup then
22:14:37
makomo
zulu-inuoe: lol it was swank, gah.. there's a special case just for DEFPARAMETER when you're using slime's SLIME-EVAL-DEFUN
22:15:02
zulu-inuoe
makomo: Oh? Really? What does it do?
22:15:05
makomo
it instead calls SLIME-RE-EVALUATE-DEFVAR which then calls SWANK:RE-EVALUATE-DEFVAR on the CL side which uses MAKUNBOUND
22:15:45
makomo
only after unbinding it does it evaluate the form, which then errors, leaving the symbol unbound
22:16:01
zulu-inuoe
@_@. It does that for `defvar`?
22:16:13
makomo
so to conclude, it didn't have to do with packages after all
22:16:23
makomo
(string-match "^(defvar " form)
22:16:47
makomo
or just C-h f slime-eval-defun
22:17:22
makomo
erm, s/DEFPARAMETER/DEFVAR/ above
22:18:36
zulu-inuoe
Yeeep. Still exists in sly, too. Interesting. But I guess I've never called it (I always C-c C-c or C-c C-e)
22:19:57
makomo
heh, i frequently do slime-eval-defun (keybinding ,ef in spacemacs)
3:42:14
oni-on-ion
https://european-lisp-symposium.org/static/2019/rhodes.pdf
Saturday, 6th of July 2019, 4:14:37 UTC