freenode/#lisp - IRC Chatlog
Search
19:23:30
Nilby
p_l: as a programmer I used clsql as an ETL tool, but I had to write the transform part myself. ETL tools are mostly GUI to write what would be a function. Unfortunately it's hard to have non-programmers do your data cleaning for you, if you don't make them a GUI.
19:33:38
Nilby
Unfortunately, years later, the ETL got outsourced to a bank who did a crap version in SSIS or something :( , probably because nobody could understand WTF I was doing in Lisp.
19:34:52
p_l
that's one part of it, another is that the bank probably outsourced it to idiots who made crappiest SSIS possible
19:46:52
defunkydrummer
i have also seen the horrible SSIS packages unexperienced people create when given the tool
19:47:26
defunkydrummer
"ETL tools are mostly GUI to write what would be a function" --> Basically, this.
19:48:38
defunkydrummer
now you have tools like Alteryx, in which an inexperienced person, after receiving a full course in Alteryx, can be able to create in a full one or two hours of clicking and clicking through menus, the equivalent to a few dozen lines Python script.
19:49:59
p_l
defunkydrummer: for what its worth, that few hundred lines of python better not have cpu-bound points
19:50:35
defunkydrummer
minion: memo to iarebatman: you wanted a function to traverse nested property-lists (plists). I have one here: https://gist.github.com/defunkydrummer/011c66f40a52977c2b33a1671bd5af0c. Enjoy.
19:51:03
p_l
defunkydrummer: I had to babysit a script once which was technically a transform script from pcap to CSV
19:51:16
minion
iarebatman, memo from defunkydrummer: you wanted a function to traverse nested property-lists (plists). I have one here: https://gist.github.com/defunkydrummer/011c66f40a52977c2b33a1671bd5af0c. Enjoy.
19:53:55
defunkydrummer
iarebatman: "TELL ME NOW SLAVE BOT"---> rofl!! So, why do you need to create plists that are so nested?
19:54:10
stylewarning
defunkydrummer: let's write a new quicklisp library where everything is written in spanish
19:55:08
defunkydrummer
stylewarning: i'd better write some lyrics and you can add music to them in your brand new yamaha SK-7
19:55:11
iarebatman
And I’m new to CL, with a tiny bit of Clojure experience - so their get-in macro came to mind
19:55:35
defunkydrummer
stylewarning: or we can create in spanish, a ML dialect embedded in common lisp, and call it "TONELADA DE CARBON"
19:56:00
defunkydrummer
awww sorry!! SK is a line of yamaha synths, my brain played neural network jokes on me
19:56:31
defunkydrummer
spanish is my native language, i'm from <redacted out>, same city as PuercoPop
19:57:40
defunkydrummer
so, iarebatman , welcome to the wonderful world of Common Lisp, now with 100% less underlying Java
19:59:24
defunkydrummer
stylewarning plz contribute more memes to LispMemes, it's a moral imperative to satisfy the hunger of the Lambda Calculus gods. I admit the quality of my own memes have declined a bit.
19:59:53
iarebatman
That’s definitely a huge plus in my book - I was ecstatic when I discovered CL was actually still used by people and that I could build native executables with it!
20:00:20
defunkydrummer
iarebatman: if you check the (english) documentation of "getf*", you'll see it will also pick up lists of plists, which is useful.
20:01:03
stylewarning
defunkydrummer: i'm giving a lisp talk at a julia meetup saturday so i need to prep some
20:01:49
iarebatman
Thank you for the gist defunkydrummer, I will have to take a look later tonight when I get to work on my personal stuff
20:02:06
defunkydrummer
iarebatman : yes, however note that modern lisp implementations are going to compile your functions to machine language anyways, so you'll get no speedups when producing an executable. In other words, if you are looking for creating an executable only for a potential speedup, it isn't needed at all.
20:02:47
defunkydrummer
phoe: Xach doesn't know the benefits of our job compensation plan: 1. Free Space Cadet Keyboard
20:03:14
iarebatman
Naw, I simply enjoy being able to hand a stand-alone executable to someone that they can run without a bunch of dependencies
20:03:47
defunkydrummer
iarebatman: yes, but note that the dependencies will "be there" in the form of a gigantic executable size
20:04:35
iarebatman
I ran “strip” on my buildapp test and it was around 200KB. That seemed reasonable to me
20:04:41
defunkydrummer
because they will need to be loaded in the Lisp image for your system to work. When producing the executable, this effectively means embedding that image into the executable. Unless you use an implementation like Embedded Common Lisp (ECL) which is able to do all kinds of magic.
20:04:44
katco
LiamH: well, it appears as though as of asdf v3.2, `:defsystem-depends-on` does not do what we think it should do in this case. it can't find `CFFI-GROVEL` despite it being in `XDG_DATA_DIRS`: "READ error during LOAD: Package CFFI-GROVEL does not exist."
20:05:32
defunkydrummer
SBCL and you got 200KB? ummm are you from the future? tell us, did we solve global warming?
20:06:01
katco
i can correct my guix package by also specifying `:cffi-libffi` in the `depends-on` clause, but i remain unsure what the correct way to do something like this is...
20:06:42
defunkydrummer
although i havent "stripped" a binary before. I rather stay with my clothes on.
20:08:16
Bike
trying to understand the asdf manual... it might say that you can just have :grovel-file instead of cffi-grovel:grovel-file
20:08:29
stylewarning
defunkydrummer: I think I'm just going to sit there projecting emacs saying "this is how you debug problems in lisp, maybe u should consider this in julia"
20:10:19
iarebatman
yup - so great news - it DOES reduce my 38MB hello world CL program down to about 270KB
20:10:42
defunkydrummer
stylewarning: it would be nice if Julia was a pathway drug to Common Lisp... "if you like Julia, you'll love CL. It will make you feel like Rick James"
20:10:53
Bike
https://common-lisp.net/project/cffi/manual/html_node/Groveller-ASDF-Integration.html okay now THIS should be helpful, katco
20:11:16
katco
iarebatman: i recall having success with `:compress` options in sbcl. ~1-2MiB binaries i think? been awhile
20:11:53
stylewarning
defunkydrummer: unfortunately, given that it's Literally Impossible (TM) to interactively plot data in Lisp, and it's Literally Impossible (TM) to do any real world linear algebra, nobody in the Julia community will be converting to Lisp. :[
20:12:36
phoe
stylewarning: the only thing that's literally impossible in Common Lisp is fitting it on a microcontroller
20:12:37
iarebatman
sigh - well 38MB files aren't awesome for sure - I thought I had hit the jackpot here. you guys sorta bummed me out and now I'm a sad panda =/
20:12:44
defunkydrummer
iarebatman : if you are looking for small binaries, don't use SBCL. I mean, you can reduce the binary size by enabling compression when using save-lisp-and-die. If smaller binaries are required, CLISP produces smaller ones. If ultimate small binaries are required, ECL will compile your Lisp code to C and can produce reasonable sizes. Or if you hav
20:12:57
phoe
the rest is a matter of the Borg^W^W absorbing enough of the other paradigms to be able to emulate them
20:13:00
stylewarning
katco: Well the capitalization and TM were sort of jokes, but right now, Lisp isn't suitable for the kind of thing the usual scientific programmer wants to do.
20:13:05
dlowe
iarebatman: the nice thing is that if you make a huge app, it probably will only bump it up a few MB
20:13:34
MichaelRaskin
Also, there are some things that are cleaner in Julia from the language point of view
20:14:24
katco
stylewarning: i definitely got matplot histograms out on my first day of trying. i'm working with linear algebra and data science libs right now
20:14:33
stylewarning
defunkydrummer: even MAGICL, which was supposed to be good, just hasn't had the time to actually be good.
20:14:48
defunkydrummer
stylewarning the other hand... and i'm going to be a SmugLispWeenie here, if suddenly CL had the same support for "data analysis" and "data science" as Python, we would have a huge influx of people adding very low quality libraries to the ecosystem. Which is what is happening with Python now.
20:14:50
stylewarning
katco: well if you want to chat/want guidance/collaborate/etc., i'd be happy to
20:15:01
phoe
dlowe: brief googling tells me https://gist.github.com/burtonsamograd/f08f561264ff94391300
20:15:27
defunkydrummer
dlowe: please call the function "shake-shake-shake" like the LW guys did, or "shake-yer-booty". Or you can call the project: SHEIK YERBOUTI
20:15:38
katco
stylewarning: i would definitely be interested in chatting and or guidance. i'm using CL to learn data science and ML stuff
20:15:38
iarebatman
I mean that makes me feel just about as icky as bundling up a node project or a python project with virtualenv and a bunch of deps. I guess at least in this case there are possible solutions..
20:16:02
stylewarning
dlowe: phoe: We used a more "up to date" version of this before... See %ZAP-INFO: https://github.com/rigetti/quilc/blob/master/app/src/impl/sbcl.lisp
20:16:31
defunkydrummer
iarebatman the situation in CL doesn't get remotely close to the horrible mess that is the NPM module system.
20:17:38
defunkydrummer
stylewarning uses windows!! suddenly i don't feel alone!! We're two now, two!!!
20:17:58
stylewarning
defunkydrummer: no, i just have to build our sdk on windows, which is housed in some VM somewhere :[
20:19:19
stylewarning
katco: it provides relatively no-nonsense bindings to BLAS/LAPACK and another fortran lib called expokit for matrix logarithms and exponentiation
20:19:45
stylewarning
katco: but the high-level library part of it is really poor and lacking, and deserves to be redesigned. There's also a decent einsum macro.
20:20:06
defunkydrummer
dlowe: don't measure binary size by using "hello world" programs, that doesn't make too much sense. Considering that CL is able to generate code and compile it at runtime, for a complex program, or even a slightly complex/big program, binary sizes will be totally alright.
20:20:30
katco
stylewarning: cool. i'm not really in a position to comment on utility yet as i'm just starting out. my focus is ML stuff
20:21:02
katco
stylewarning: and my immediate focus is data exploration prior to building a ML model. cl-ana seemed like a good fit for that
20:21:30
iarebatman
yay my CL book came in: Common Lisp: A Gentle Introduction to Symbolic Computation
20:22:16
phoe
in case you have programmed, it might be a little bit too gentle - but let's see what you think after the first few chapters
20:22:21
defunkydrummer
dlowe: It doesn't make sense for practical purposes. Nobody uses a language like Lisp for "hello world" programs. If your goal is the ultimate small hello world, you just create a shell script that prints Hello World. Or use assembly language to put "Hello World" in a memory location and jump to the OS function that prints it out... In any case, y
20:22:21
defunkydrummer
ou CAN do a small hello world in Lisp by using Embeddable Common Lisp (ECL), which will compile it to C and then produce the binary.
20:22:53
phoe
by the time it comes to deployment we're already doing docker shit that weighs hundreds of megabytes in total
20:28:59
defunkydrummer
dlowe : fair. But IMO that test would only show the size of the runtime and the presence/absence of a tree shaking mechanism
20:50:11
stylewarning
dlowe: to be clear, there shouldn't be a magical tree shaker, there are tradeoffs to what gets shaken out
21:25:15
aeth
Am I being naive if I store an immutable graph like this? 1 <-> 2 <-> 3 and 1 <-> 4 would be #(2 4 1 3 2 1) and #2A((0 1) (2 3) (4 4) (5 5)) where the 1D array is the connections and the 2D array is essentially the key that says what range the (n+1)th element is in (since the graph here starts with 1)
21:29:41
aeth
I'm not using lists because of random access, e.g. I can look up an item (and there might be thousands) as (subseq connections (aref range (1- item) 0) (aref range (1- item) 1))
21:31:08
aeth
(Except subseq is probably unnecessary in every actual use because most sequence functions have a start and an end, which is why I'd be storing the start and end)
22:33:22
grewal
aeth: why do you need two arrays instead of one? You can just store the nodes each node is connected to. Using your example, something like #(#(2 4) #(1 3) #(2) #(1))
22:37:28
vms14
(defclass meh () ((oh :accessor oh))) this won't let me push on this slot unless I put :initform nil
23:06:13
aeth
grewal: You can't, rows have to be the same length. I could pad it with -1s (or 0s, if I keep them starting at 1) but then I wouldn't know where they end ahead of time
23:06:39
aeth
grewal: It would also be better to use fake 2D arrays rather than actual 2D arrays, so sequence operations could be used with start/end.
23:15:15
grewal
aeth: I didn't use #2a. It's a vector of vectors, not a multi-dimensional array. Ultimately, the two approaches are probably the same under-the-hood, but your approach just seems a bit oblique
23:17:14
Bike
defstruct does not initialize to nil. it's undefined, which is worse than defclass, because if you read an uninitialized structure slot anything could happen
23:20:10
aeth
grewal: I think my solution scales better because two specialized arrays are going to be more efficient than a T array of specialized arrays because the GC will afaik have to iterate through the T array, but not the two specialized arrays.
23:32:25
White_Flame
aeth: more efficient would be to ensure the GC isn't triggered often, and use the most appropriate data structure regardless of its latency effect on GC
23:59:02
vms14
so the way is try to control when the gc is triggered and minimize as possible the work of the gc
23:59:43
aeth
vms14: of course you can't actually not allocate, so it's more choosing where to allocate... so preallocate
0:01:27
aeth
vms14: (defun foobar () (let ((foo (make-foo)) (bar (make-bar))) (do-some-fancy-game-loop foo bar))
0:01:58
aeth
vms14: And the bindings in do-some-fancy-game-loop aren't allocating (well, probably aren't)
0:05:38
aeth
vms14: This is a very Java-oriented way of saying "reuse buffers" like White_Flame said earlier. https://en.wikipedia.org/wiki/Object_pool_pattern
0:29:42
aeth
Me? I've never heard of that before. The website link is dead. This appears to be the source. https://github.com/ilitirit/manardb
0:30:22
aeth
I know a lot of people are proud of being able to use old libraries in Common Lisp, but I'm not sure I'd use a database that hasn't been updated in 10 years.
0:31:56
aeth
There's a handful of things that are just inherently really hard to write: operating systems, web browsers, databases, office suites, IDEs (and most content editors in general that are fancier than a basic text editor), game engines, etc.
0:32:59
aeth
Databases are pretty important, too, because you don't want your data to be corrupted or lost.
0:56:24
White_Flame
my first thought was actually destructuring-bind, but you'd still need to name intermediate value holders
0:56:50
vms14
also I should use :initargs instead of that bunch of setfs and will be even better than use pop
1:01:47
rtypo
i also tried sqlite last week, what i do differently is run every query inside 'with-open-database'
1:03:57
rtypo
i think i'm just paranoid with managing connections, since i'm bad at using databases :D
1:06:10
vms14
but I start by making first a simple forum, the most difficult thing is mantain data organized and know what kind of data I need
1:14:02
Nilby
I'm a terrible programmer, but that hasn't stopped me from writing 100k lines of semi-working Lisp.
1:14:29
vms14
I guess the better is to make a prototype and then write a new program taking that prototype as a reference
1:26:34
vms14
the comic is about the programmers having bugs, and those bugs evolve and end slaving the humanity, only lisp can save us
1:30:06
Nilby
After watching that video, I now realize I've been writing "Balance weasels on a rake".
1:54:05
equwal
So is there a way to generally reorder arguments to a function at compile time with a macro, based on the type?
2:01:42
Bike
if you really want to, you can usually get at the declared type information through semiportable mechanisms
2:51:27
no-defun-allowed
You could use a generic function and do something like (defmethod foo ((bar type) baz) (foo baz bar)), but that isn't "compile-time" and may cause fun call loops if baz is also type.
2:55:33
aeth
specialization-store does type-based dispatch at compile time IF there are type declarations. https://github.com/markcox80/specialization-store/
4:32:11
beach
equwal: There is a general rule in programming that you should use the most specific construct that has the desired effect, so instead of (= 0 ,..) it is preferable to use (zerop ...).
4:34:09
beach
equwal: Normally, a compiler macro should return the original form if it can't alter it, so that the normal function is called instead.
4:38:36
equwal
I think it might not be possible to do this in general without strong typing. How can I return the original form by default if the original form has arguments in the wrong order? So when you send something like (group (list 1 2) (+ 1 1)), things break.
4:40:28
beach
Anyway, if you don't respect the compiler-macro protocol, you might as well put the logic in a function.
4:43:15
beach
That is why, if they detect a situation that they can't do anything about, they return the original form so that the function itself is called at run time.
4:44:57
beach
I am also puzzled by the fact that your compiler macro returns a form that calls GROUP-AUX.
4:45:33
beach
If you know your arguments at compile time, why don't you just compute the result at compile time rather than generating a form that will compute it at run time?
4:48:21
beach
equwal: Are you aware that you can not use your GROUP function/compile-macro when the arguments are variables, like you can't say (group x y)?
4:48:58
beach
oni-on-ion: That seems like a basic truth about programming to me. Every programmer should already know it.
4:50:04
equwal
I thing the compiler macros are meant for optimizing stuff at compile time when constraints are met, so should follow your profound advice.
4:50:11
beach
equwal: They work at compile time, just like macros do. So your arguments N and SOURCE are unevaluated parameters to the form.
4:50:40
beach
equwal: If you have a call such as (GROUP X Y) then they will both be symbols, so the compiler macro will error.
4:51:13
beach
equwal: Therefore, your compiler macro works only when the arguments are known at compile time.
4:51:33
beach
So you might as well compute the entire result then, rather than generating code that computes it at run time.
5:14:06
equwal
Well I have proven to myself that reordering arguments based on type data is way more difficult than I am willing to go for a such a frivilous thing.
5:15:01
beach
equwal: Strong typing means that there is no way an object of a particular type can mistakenly be taken for an object of a different type at run time. Weak typing is the contrary. Static typing means that some of the types are checked at compile time. Dynamic typing means that the types are checked at run time.
5:15:45
beach
equwal: Yes, it breaks with (+ 1 1) because that is not a number. It is a list of three elements, a symbol and two numbers.
5:16:46
beach
equwal: What you can do in your compiler macro is to return a form that will check the types at run time, and call group-aux differently in each case. But then, there is no point of having a compiler macro. You might as well put that logic in the GROUP function itself.
5:18:02
beach
equwal: I think you really need to contemplate what information is available at compile time and what information is only available at run time, and design your code accordingly. Macros and compiler macros deal with information available at compile time.
5:30:13
beach
equwal: The entire idea of reordering argument forms of a function at compile time is bogus. A programmer using your function will rightfully expect the evaluation order of arguments to be respected. Suppose you were able to do what you say, and then the programmer types (GROUP (1+ X) (MAKE-LIST X)).
5:30:46
beach
equwal: You definitely do not want to reorder those arguments, because that will alter the intentions of the programmer.
5:31:36
beach
So there is basically nothing you can do at compile time, other than for constant arguments.
5:34:46
beach
And that is what compiler macros are for. They can identify information available at compile time, and generate alternative code for the form. If no such information is available, then they should return the original form so that the associated function is called as usual at run time.
5:36:26
beach
p_l: The "programming" in "dynamic programming" is probably the same as the one in "linear programming", i.e. it has nothing to do with computer programming, but with establishing a "program" in the sense of a sequence of actions to be taken.
5:37:31
beach
p_l: But I agree that both those terms are confusing these days when the main use of the word "programming" is "writing computer programs".
6:01:42
equwal
It makes sense to me now. I can't reorder arbitrary arguments at compile time, not sure what I was thinking.
6:17:44
aeth
equwal: I think you could do this with the specialization-store library by having a specialization for ((a foo) (b bar) (c baz)) as well as ((a bar) (b foo) (c baz)) etc. generated by a macro
6:36:05
aeth
note that unless you have type declarations (so it's known at compile time) libraries that use these macros (or making them yourself manually with these macros) will be slow
7:08:57
asarch
What does #+(or) mean (line #63)?: http://paste.scsys.co.uk/584262?ln=on&submit=Format+it%21