libera/#commonlisp - IRC Chatlog
Search
7:46:01
White_Flame
Bike: it can't prove that the LOOP will always find a match, and so could return NIL
7:47:14
White_Flame
you'd have to declare that dat is a recursing list and thus would never terminate (if that's the intent)
7:48:04
empwilli
EdLangley[m]: I'm toying with this idea for compiled languages: Instead of having the class description in, let's say, c++ dictate how data is placed in memory, let is shuffle around (or do even fancier stuff, e.g. the things that databases do, in games terminology, this would be entity component systems)
7:49:45
White_Flame
empwilli: I've had plenty of ideas similar to that as well. My notion would be that data tracks where it was allocated from, and usage feeds back into how it should be initially allocated/generated
7:57:43
EdLangley[m]
So, what got me down this road was thinking about how, if the MMU were designed for the languages people like to program in (ones with lots of pointer chasing), it could use access patterns to transparently move data around in physical memory for better locality (sort of like how SSDs move data around on disk for wear leveling)
7:58:23
EdLangley[m]
Then, I realized that the GC already walks all the objects periodically, so if you had some sort of statistics about access patterns, a moving GC could take those into account when it compacts the heap
7:58:42
empwilli
I'm not too deep in this literature but I think it has been shown to be too expensive to bring a benefit
7:58:58
EdLangley[m]
The simple case here would be, if you have a list that's iterated over frequently, to move all the cells together
7:58:59
empwilli
the other issue is, that, generally speaking, the granularity of 4k pages is too big
7:59:16
EdLangley[m]
It might be too expensive to do all the time, but maybe there's some sort of profile-guided optimization you could do?
7:59:36
EdLangley[m]
e.g. run a bit of code with a profiler and store statistics about the actual observed access patterns
8:01:28
EdLangley[m]
So, a naive way to do this would be to attach a counter to every member of a class
8:01:50
EdLangley[m]
And increment it every time you follow the pointer from the instance -> the actual member
8:02:14
EdLangley[m]
Then, the gc picks a threshold and moves members close to the instance in memory, if the counter is above the threshold
8:02:56
EdLangley[m]
This probably has a bunch of disadvantages (turning reads into writes, etc.) but it'd improve the locality of instances and commonly used members.
8:04:19
empwilli
So what I can point you to is literature on NUMA systems where this is frequently a problem (at least from the perspective where to allocate memory to/from)
9:52:17
unixlisp
Alfr: "3.2.2.3 also says that, apart form the listed exceptions the new definition takes precedence" That is "about minimize the observable differences between compiled and interpreted programs".
10:00:19
beach
Alfr: Frequently unixlisp seems to make statements without any context, and sometimes without any stated purpose. I think you need to get used to it.
10:03:27
Alfr
unixlisp, but what are you trying to say with those quotes? Or what's the problem/question?
10:04:24
beach
I think unixlisp now realizes that the page that was cited before as being about redefining functions, in fact is not. But I already said that.
10:05:44
Alfr
beach, I cited that, and I still think it's relevant. As it defines what conforming programs can expect, thus conforming implementations must arrange for.
10:07:55
empwilli
EdLangley[m]: hm, given that GCs really are notorious sources for performance bottlenecs (at least my understanding, I mostly don't do GC programming, so please correct me): Moving around members in memory would really add a large amount of overheads
10:11:03
Alfr
beach, (okay for the other case where it simply happens during runtime, DEFUN's documentation already specifies the behavior as replacing.)
10:13:59
beach
empwilli: So if you are avoiding GC for performance reasons, you may be doing the wrong thing.
10:14:32
empwilli
no worries there :) I'm avoiding it just for the convenience of the languages I feel comfortable in
10:15:22
empwilli
I'm currently working on adding common lisp to my repertoire, but I'm far from fluent in it
10:24:56
beach
empwilli: It is too bad, though, that people seem to think that automatic memory management is a "notorious source for performance bottlenecks". They are then doomed to program in a way that makes it extremely hard to get both performance and modularity in their code.
10:27:51
empwilli
this sounds as if there were design patterns to program in favour of the GC. Is this the case? Can you point me to some resources?
10:30:17
beach
I had no design pattern in mind. I am just making a general observation that "liveness is a global property" [from Paul Wilson], so not a "modular" property. If your program is modular, you either need to copy all your objects, or use reference counters if you don't have automatic memory management.
10:30:39
seragold[m]
I think it was David Ungar back in early nights who showed that no sort of reference counting and its variants overall performed as well as state of art GC then. So why would you want such in your way of whatever the most efficient and elegant patterns are it is possible to come up with in a language that gives you all the power?
11:06:22
unixlisp
beach: early about (setf (symbol-function 'foo) newdef), you said "It could be defined behavior only if the function is currently undefined". but in clhs about symbol-function entry, "setf may be used with symbol-function to replace a global function definition"
11:44:18
beach
unixlisp: Yes, notice the word "could" in order to explain to masinter that the existence of that operator doesn't necessarily mean that the behavior is always defined. But as it turn out, it is.
11:50:02
loke[m]
beach: Take your average C programmer: malloc and free "feels" free in terms of performance impact. So when someone suggests that a GC will run automatically and search for all the memory to be released, that feels "obviously" slower, since now it has to do stuff when before it didn't have to.
11:50:59
unixlisp
beach: I noticed the word "could". In clhs defun entry: "defun can be used to ... redefine an already-defined function", that word is "can".
11:51:03
loke[m]
Any argument that malloc/free are actually slow falls of deaf ears since their microbenchmarks all use stack allocation, which indeed is free.
11:54:58
flip214
loke[m]: not in terms of cache misses... alloca()te a few KB and you're out of your L1
11:55:56
loke[m]
flip214: Well sure, but that applies to all allocations, regardless of whether they are heap or stack. Unless I misunderstand your point?
11:57:17
beach
loke[m]: Yeah, but it's disturbing to me that so many decisions in the software industry are based on "gut feeling" rather than research. And I can only imagine the amount of money wasted that way.
11:57:35
flip214
empwilli: one example favoring a GC: https://people.eecs.berkeley.edu/~fateman/papers/cachelisp.pdf
11:58:11
flip214
loke[m]: no, you're understanding me correctly. My point is that stack allocation is not completely free either.
12:01:24
flip214
empwilli: and ELS2018 (https://www.european-lisp-symposium.org/2018/#programme) had "Dynamic Optimizations for SBCL Garbage Collection" which would switch a load balancer around before doing GC, so the equivalent to free() would be free here
12:01:51
beach
unixlisp: Do I really have to spell this out to you? I mean to say "masinter, you are right that there is an operator (setf symbol-function), but you say that its existence is proof that redefining functions is defined behavior. However, it COULD very well be the case [but it isn't] that this operator is defined only when the symbol does not already have a function defined, in which case definition is OK, but REdefinition is not
12:01:56
beach
unixlisp: ... So the existence itself does not guarantee that redefining a function is defined behavior. However, redefining a function IS defined behavior, as the Common Lisp HyperSpec specifies".
12:05:48
unixlisp
beach: oh. "redefining a function IS defined behavior", acording to "defun can be used to ... redefine an already-defined function"?
16:27:47
phoe
In case of (loop for sym being the symbols of :bar collect sym) - is it possible that the list will contain duplicates?
16:33:28
Bike
phoe: do-symbols can hit symbols multiple times if they're inherited from multiple pa- right.
16:34:09
specbot
The for-as-package subclause: http://www.lispworks.com/reference/HyperSpec/Body/06_abag.htm
16:34:17
Bike
well, according to the description in 6.1.2.1.7, "the symbols" means it will iterate over symbols accessible in the package
16:49:42
mfiano
It has things like positive-fixnum, non-negative-fixnum, etc. Seems like it should have that
16:53:07
phoe
but only during compilation! and it usually happens in between tons of other packages too, so it's possible to miss it
16:53:29
EdLangley[m]
Also, if you do alex:dim-in-bounds<TAB> for complete, like I do, SLIME will put in the -2
16:55:48
dim
BTW has anyone tried shipping a software built with ECL: a C-core and some APIs exposed to a Lisp part of the application? -- I am thinking about doing that for pgloader, allowing me to use the existing C drivers for database sources etc
16:57:32
jackdaniel
dim: I've shipped a C application linked against ECL that allowed connecting via swank medling with a running C application (i.e recompiling functions), the api was available to the lisp world to test things out
16:59:25
dim
jackdaniel: sounds a good starting point ; in the case of pgloader I would do the basic boring stuff in C (copy data around) and the interesting bits in Lisp (merge catalogs, parse commands, all the advanced / user-friendly stuff) ; would you have an opinion?
17:00:07
dim
I mean I would like to both simplify the build process and also have an eye to better perfs for the inner loops when written with the official database drivers in C
17:00:46
jackdaniel
dim: core written in C and linked against ECL sounds like a good fit for that, yes
17:01:01
dim
more important than perfs is also full protocol support even with new versions, including auth and encryption
17:01:23
jackdaniel
you could also write your own lisp libraries and compile them to shared objects (you may inline C code in Lisp functions too)
17:01:55
dim
it's either that or porting to Clojure to benefit from all those JDBC drivers out-there
17:02:15
jackdaniel
the only limitation is that if you want to recompile functions that have inline c code, then a c compiler must be available on the system
17:02:24
dim
maintaining qmynd for MySQL and patching Postmodern for Postgres SCRAM authentication and all those things, I believe it's too hard an ask for just maintaining pgloader
17:03:00
dim
yeah well I would like to ship a copy of pgloader that's all static built in the ideal world
17:03:25
dim
I like the dynamic aspects, but I prefer the “just works” story for end-users that don't even know lisp is involved in pgloader, and when they do, don't understand why
17:03:27
jackdaniel
ecl may be linked in statically, but you need to keep in mind that the license of the implementation is lgpl-2.1+
17:04:08
dim
well I mean I can also build dynamically, my focus is easy-to-build, easy-to-ship, easy-to-use, don't need to know anything about lisp
17:04:23
jackdaniel
sure; the application I've mentioned earlier had ecl mostly for debugging, live patching and such, the end user was not concerned with any programming languages
17:04:36
dim
and to be fair the current pgloader story doesn't provide these when using either SBCL or CCL
17:05:48
dim
I have no problem getting a debian package included in debian and apt.postgresql.org for pg_auto_failover and the latest tool I wrote, pgcopydb, that are written 100% in C...
17:06:36
dim
I have tons of problems with pgloader where I bailed out and the current maintainers each time want to orphan it and find a way to update bits here and there and continue to maintain it but spend hours and hours on the package building and maintenance
17:10:00
etimmons
dim: ECL is probably great for this, but you can also link C code into the SBCL runtime. You can even deliver a fully static executable (I've got some patches floating around for that. I really need to work on getting them upstreamed)
17:11:16
dim
I have heard and keep hearing about that capability each time I bring the topic up, but I never got a clear documentation using maintained APIs to do that in pgloader myself, unfortunately
17:11:50
dim
rewriting the inner core of pgloader in C looks the easiest way around for me nowadays :/
17:18:37
mgl
jackdaniel: In ECL, (let ((*print-pretty* nil)) (princ '(function foo))) prints #'FOO. Is this compliant? Is there a way around this?
17:22:11
etimmons
I actually don't think CFFI's manual is great here. That static linking it talks about is really only for CFFI generated wrappers, not system libraries like OpenSSL
17:23:39
etimmons
I use a homebrewed system to build my static executables. But it's definitely not fit for others to use at the moment
17:25:13
etimmons
The current best (and I believe supported) way is to build SBCL with :sb-linkable-runtime option. Then link your libraries with sbcl.o to make a new runtime that has all your C libs already present
17:31:41
dim
remember that I need something that builds on machine I have never seen and will never see, it needs to be built in debian systems and by users when they want to, most of them using a Docker image of some sorts...
17:31:52
fitzsim
etimmons: with that sbcl.o approach, is it possible to get a REPL on the running binary?
17:32:44
dim
anyway a C application that uses ECL to implement some of the higher form logic (decision making, etc) in Common Lisp sounds a good approach for my delivery needs here
17:33:58
Shinmera
Not really, SBCL has quite specific requirements about the runtime (such as catching a bunch of signals)
17:34:41
etimmons
fitzsim: Shinmera's right. It sounds like you want the relatively recent additions to SBCL for producing a dynamic library
17:36:25
fitzsim
SBCL is "in charge" in the current iteration, but the idea is to create a linkable libsbcl.so so that the C-based runtime can be in charge
18:14:08
seragold[m]
<dim> "remember that I need something..." <- I am confused. If it is run from docker image then the OS environment is already controlled by that image regardless of environment of machine the image in run on, right?
18:15:44
seragold[m]
<jackdaniel> "right, eql5 is a very cool..." <- I don't trust the Qt company folks enough to invest much here. Starting to look at CLOG
18:26:24
seragold[m]
tyson2: too early in my journey to say so far as day job eating too much energy. more to say in about a month.
18:37:23
White_Flame
I had built a few flash & html based GUIs, but not a general library yet. I tend to believe that having a desktop metaphor _inside_ a webpage isn't necessarily the most useful
18:38:04
White_Flame
but having multiple tabs/windows be part of the same lisp application (or server) context is quite important
18:43:09
Shinmera
I'm *also* sad that I've so far been unable to procure any real contributors for Alloy
18:48:55
Shinmera
Ime most elements are positioned in a way that does not require feedback from the text renderer.
18:50:24
dim
seragold[m]: I mean I have seen too many recipes/how-tos with manual fiddling or worse interactive adjustments along the years, and finding the so files for cffi and more generally in common lisp has always been a problem when I had to
18:54:27
seragold[m]
<Shinmera> "I'm sad that Clog is completely..." <- So what makes it so? Might be good information for the fond dreams of many of us.