freenode/#lisp - IRC Chatlog
Search
18:34:40
faheem
The only thing I'm aware of is a library that Tamas Papp wrote. But it's kind of abandoned.
18:35:15
faheem
makomo: Haven't you come across it? It's the topic hit for -> data frame common lisp
18:36:10
faheem
makomo: No, I haven't. It didn't exactly seem developed. Tamas apparently decided that Julia was a better better bet.
18:37:13
faheem
makomo: Did you find anything that seemed at all usable. Something like Pandas seems hopeless, unfortunately.
18:38:13
makomo
at a glance both of these seem kind of usable, but both are underdocumented and not really finished, so, nope :(
18:41:17
faheem
See also -> https://github.com/blindglobe/common-lisp-stat/commit/eaddd8205c840dd0fe34f06c9bd03cbac698e611
18:42:27
faheem
shka_: Oh dear. Everyone writing their own is kind of a guarantee that nothing with ever be complete enough to use.
18:43:40
faheem
shka_: 384 commits is better than some. Of course, it depends on the commits. Do you have tests?
18:45:18
shka_
makomo: no joins, no merges, it leans back on the cl-data-structures though so group by just works
18:46:46
shka_
yeah, it is another lib, has some data structures (not relevant here) and operations that are supposed to work on abstract interfaces
18:49:41
shka_
https://github.com/sirherrbatka/cl-data-structures/blob/ef1ad2064804759d65b0bd6f4520501fa40c4f5c/src/algorithms/meta/meta-tests.lisp#L48
18:50:11
shka_
ok, so there is this abstraction called ranges, and there is abstraction called aggregation functions
18:52:09
makomo
this indexing business is pretty neat. tpapp has a library related to that https://github.com/tpapp/cl-slice
18:52:13
shka_
anyway, idea is that aggregation functions work on ANY range so if you just manage to implement the right protocol it simply chew it trough fine
18:53:01
aeth
For CL GUIs you can use Sketch, but you'd have to write your own way to render graphs/etc. to it. https://github.com/vydd/sketch
18:53:33
makomo
faheem: that's pretty much what i want to use dataframes for, and yeah, the two are very related
18:53:40
makomo
see https://pandas.pydata.org/pandas-docs/stable/getting_started/comparison/comparison_with_sql.html
18:54:27
shka_
so my implementation is very basic at the moment, and main focus was on core ideas like mutability and the right API
18:56:14
shka_
faheem: yeah, i did look at it, does not support sparse columns which is deal breaker for me
18:56:41
faheem
shka_: Ok, but overall how does it compare to yours? Less developed or just different?
18:57:03
faheem
You should see if you could take features/ideas from there. All this replication is very inefficient.
18:57:42
Xach
faheem: There's a process that I moderate and most stuff goes in but some stuff does not.
18:59:06
Xach
I don't consider the garbage-ness of a project when looking at a request to add something.
19:00:03
shka_
Medical, claims. Some bright mind figured that ideal way to represent the diagnosis is to store them in 20 seperate columns
19:03:37
faheem
It's definitely a real problem with CL. A dozen libraries for the same thing. None of them usable.
19:04:22
faheem
shka_: Do you have a mailing list or something? Can you shoot me an email when you project is on QL?
19:04:27
shka_
but if you are willing to try it, i will have more input to consider the future development
19:08:23
shka_
anyway, here is the frames library: https://github.com/sirherrbatka/cl-data-frames but you will need also this (which is way to big) https://github.com/sirherrbatka/cl-data-structures and this one (which is pretty small) https://github.com/sirherrbatka/lense
19:09:40
shka_
cl-data-structures is kinda a mess because it has some stuff like statistical estimators that don't have any business being in there
19:12:52
shka_
anyway, i would be thrilled if you guys want to try my stuff out and maybe help me a little
19:13:47
shka_
data-frames work although are rough around the edges, cl-data-structures are genuinely useful as fuck
19:15:56
shka_
i really need to clean up cl-data-structures and modularize it before i put on github
19:16:03
makomo
https://github.com/blindglobe/rho, https://github.com/mirkov/data-table/, https://github.com/tpapp/cl-data-frame, https://github.com/sirherrbatka/cl-data-frames/
19:23:56
shka_
well, it is not painful enough for me to stop and i seem to be the only person with such problems so maybe it is fine
19:27:42
shka_
i know i should, doing this in a productive way would require a lot of time, and i have way to many things on my hands already
19:32:03
faheem
shka_: BTW, did you see this? http://within-parens.blogspot.com/2016/12/rcreational-common-lisp.html
19:39:24
PuercoPop
shka_: Have you seen https://github.com/fiddlerwoaroof/data-lens? How does it compare to lense?
19:40:57
shka_
PuercoPop: it is not the same stuff, all i really wanted is to have direct equivalent of racket style lenses, this goes for full blown DSL build with monads
19:41:18
shka_
which i don't need, because cl-data-structures ranges and aggregation functions are way more useful
20:27:34
pfdietz
faheem: I am interested in improved curation of the Quicklisp ecosystem. But time is finite.
22:00:05
makomo
huh interesting. if there's a bound global dynamic variable *VAR*, doing (defvar *var* (error "hello")) makes it unbound (using SBCL). is this expected?
22:07:34
zulu-inuoe
makomo: I'm not seeing that behaviour. Are you sure the variable wasn't already unbound?
22:14:37
makomo
zulu-inuoe: lol it was swank, gah.. there's a special case just for DEFPARAMETER when you're using slime's SLIME-EVAL-DEFUN
22:15:05
makomo
it instead calls SLIME-RE-EVALUATE-DEFVAR which then calls SWANK:RE-EVALUATE-DEFVAR on the CL side which uses MAKUNBOUND
22:15:45
makomo
only after unbinding it does it evaluate the form, which then errors, leaving the symbol unbound
22:18:36
zulu-inuoe
Yeeep. Still exists in sly, too. Interesting. But I guess I've never called it (I always C-c C-c or C-c C-e)
5:38:46
faheem
I remember when I previously tried using SBCL, most of my issues were with GC. And I bit to do with not being able to figure out how to implement/silence
5:39:39
faheem
Any significant changes in the GC or numerical calculations? I know this is vague, so feel free to ignore.
5:42:21
faheem
I don't think this is the one, but it's related -> https://bugs.launchpad.net/sbcl/+bug/936304
5:46:44
aeth
faheem: I know how to write numerical SBCL. Yes it takes years to master, although it always does.
5:51:39
aeth
This is probably why Lisp lost to C in the era when performance mattered a lot more. That and SBCL didn't exist yet. Then again, lots of people write stuff in Java or even JavaScript now where performance matters...
5:52:29
White_Flame
I think python is the most popular language in the "numerical performance scenario with slowest language" category
5:54:03
aeth
White_Flame: I just enjoy when I'm bottlenecked by one core running at 100% of some (probably pure) Python script. /s
5:56:17
aeth
White_Flame: This came way after Lisp lost, though, which suggests it might just have been too early
5:57:36
aeth
faheem: Learning all of the edge cases where you need to pay attention and where you don't
5:57:41
faheem
Slowless isn't a real issue with SBCL. It's fast enough. But GC weirdness can be an issue.
5:59:00
faheem
Has much changed with the GC situation since 2012/2013? I know there has been a lot of work done.
6:00:12
aeth
I'm not sure. #sbcl probably knows. I have a fairly non-idiomatic style for performance-critical numerical code.
6:01:09
faheem
aeth: Would it be fair to say that SBCL is the way to go for performance-critical numerical work?
6:01:47
aeth
I've gotten fairly good at controlling when allocations happen, which requires discipline that's more than normal (but less than C++). And I can't get that same level of insight on other implementations.
6:03:30
aeth
SBCL gives you structs of arrays (defstructs of specialized arrays). I don't think that the struct slot type is as optimized in other implementations
6:04:19
faheem
aeth: I see. You might be aware that Tomaz Papp wrote a blog post about how he was switching from CL to Julia, citing mostly performance reasons.
6:04:43
aeth
It's not quite manual (unless you involve the FFI, which I sometimes do), but it does take a lot of thinking
6:05:49
aeth
Numerical is basically the same thing but more double-float and less single-float (so harder to avoid boxing)
6:07:40
aeth
I would never use a JIT like Julia because I prefer what you get from an AOT. You do basically have to effectively write statically typed in SBCL, though.
6:08:25
aeth
I guess this means there is necessarily going to be a bit of overhead because now you're pairing dynamically typed (types with the values) with static type declarations (types with the variables) in one thing
6:09:44
aeth
e.g. https://gitlab.com/zombie-raptor/zombie-raptor/blob/master/CONTRIBUTING.md#consing-heap-allocations
6:10:33
aeth
e.g. https://gitlab.com/zombie-raptor/zombie-raptor/blob/061b90122b20dcca292064bca0e7d6d95c76daa3/util/util.lisp#L84
6:12:59
aeth
define-function is almost feature complete. The complementary LET-style macro needs to be finished at some point. That's a lot more elaborate, actually. It also needs its own indentation since FLET-style macros can't really be auto-indented by SLIME properly because it doesn't know enough about their structure
6:13:51
aeth
define-function isn't really special. There are probably 5+ other macros that do something similar, the most popular being the defstar library. What it is, though, is the fanciest.
6:14:04
faheem
aeth: Sounds like it would helpful to have in its own library. Provided the documentation was there too.
6:14:51
aeth
(And defstar is GPLv3, which sort of made me make define-function as elaborate as it is, out of frustration of being forced to reinvent the wheel for my MIT-licensed game engine. If you force me to reinvent the wheel, I will, but it's going to be a better wheel.)
6:22:58
aeth
With 32-bit single-floats would be a chore to work with and I wouldn't be able to do things like use (unsigned-byte 64)s which I can use in SBCL without boxing them if I'm really careful