freenode/#lisp - IRC Chatlog
Search
5:13:19
|3b|
expense of GC depends on your data to some extent, lots of conses is more expensive to GC than same amount of storage for vectors specialized for things like fixnums or characters that can be known not to contain other objects
5:14:37
|3b|
when it is invoked depends on the implementation, usually in some way related to amount of allocations
5:14:37
aeth
|3b|: Does that mean one big 2D array is better than a vector of specialized vectors because the latter has a T vector that has to be walked through?
5:16:00
jmercouris
|3b|: What is the expensive part based upon the number of cons' is each cons cell it's own object that must be free'd?
5:16:11
|3b|
right, though small effect on GC time might be outweighed by increased costs in the actual code if the abstraction is worse
5:17:11
jmercouris
basically on a system level here, when we run a GC in lisp, are we actually freeing the memory, or is it just now available within our loaded lisp image as "free" memory?
5:17:31
|3b|
jmercouris: speaking very generally, GC tends to involve walking through all objects on the heap starting at for example a register or other known starting point to see what is still 'alive'
5:17:57
jmercouris
furthermore, is this memory every coalesced or just exists neubulously within the lisp image, and then has to be coalesced to reduce the size of the image?
5:18:26
|3b|
if you have a 1 million element list of numbers, it has to look at 1 million conses. and at each number to verify it is just a fixnum/single-float/etc that doesn't need further processing (and if not, walk whatever is there)
5:18:55
|3b|
if you replace that with an untyped array, it skips the million conses, but still has to look at the million values
5:19:34
|3b|
if you switch to a specialized array, it can say "this 1 array only holds fixnums, so it can't hold anything else i need to look at" and the whole array is 1 check
5:19:40
jmercouris
what is the garbage collection strategy in lisp? is it different across implementations?
5:20:04
jmercouris
So there exist some optimizations for the compiler to be more efficient, makes sense
5:20:30
jmercouris
|3b|: I guess one could use any of many strategies, but I'm asking in a general sense, what to expect
5:20:33
|3b|
and differs between implementations (or even options/architectures within an implementation)
5:21:06
jmercouris
Okay, so it is all over the place, and I shouldn't worry too much about it except for when I can use recycling or somehow let the compiler/system know to optimize something
5:22:06
|3b|
ACTION doesn't even usually bother with using the destructive functions (and then only on obvious things like a list i just created, like in the push + nreverse idiom)
5:22:47
jmercouris
do you know of any lisp implementations that use reference counting to facilitate gc in an efficient manner?
5:23:25
jmercouris
I remember back in the early days of dalvik, good times, GC would freeze the phone for several frames
5:24:02
jmercouris
and there was no easy way to 1. limit when it runs, 2. control what it garbage collects (technically you could, but it was an abuse of java)
5:26:06
aeth
I have a game loop that essentially doesn't cons at all (a tiny bit of cl-sdl2 it uses conses a little; I'll have to replace cl-sdl2 eventually)
5:26:40
aeth
You have to pre-allocate everything and write a ton of macros to make complicated things seem natural
5:27:48
aeth
jmercouris: No language is good for game programming, then. C++ engines literally never get completed because C++ is is hard to do right when you're writing a large application
5:28:29
jmercouris
I don't even like C#, but I can't pretend it isn't good for game programming, MS put a lot of work into it
5:28:53
|3b|
jmercouris: people made plenty of games in flash with bad GC, lots of games in JS with bad GC, failure to make games in CL can't be blamed on GC :/
5:29:23
aeth
jmercouris: Most C# game programming (by an overwhelming amount) uses an engine (Unity) written in C++
5:29:37
|3b|
and even if it were horrible, i'd still argue for prototyping in CL even if parts of final thing had to be reimplemented in C or whatever
5:30:19
jmercouris
Java also has terrible GC indeed, thats why I mentioned Dalvik, when I was doing some very very early android games, what a nightmare
5:30:36
jmercouris
there's perhaps just not a good platform for CL, so maybe that's why nobody writes games in CL
5:31:47
|3b|
(java actually has some good GCs from what i hear, no idea if the defaults on consumer java stuff are any good though)
5:33:03
loke
|3b|: Java probably has the best CG's available in any system. It's not focused as much on low/zero latency though, and tuning for near-realtime can be a hassle, which is what I suspect jmercouris was referring to.
5:33:23
|3b|
doesn't help that there are lots of ways to rate a GC, some tasks might think a high-throughput GC is great (batch jobs that run over night for example), while some things might care about latency (games, sort of), or maximum pauses (games)
5:33:35
aeth
jmercouris: Anyway, I would argue that CL is the only language suitable for game programming. No matter how badly you have to abuse the language (and literally every large enough game engine will run into limitations with the language unless the engine authors also wrote the language) you can cover it up with metaprogramming. And not the kind of painful metaprogramming you'd see in a language like C++.
5:34:19
aeth
jmercouris: Sure, maybe the CL implementations haven't really caught up yet, but optimizations follow use, they don't precede it.
5:34:44
aeth
You're wasting time as a language implementor if you optimize for things that no one is writing.
5:35:09
loke
|3b|: Java comes with several GC's that you can choose between. And on top of that you can tune them in about a million ways. The default GC is great for long-running server applications, which is kinda the opposite of games. So yeah, for games stuff you have some tuning to do.
5:36:03
jmercouris
loke: The implementation on Dalvik was a nigthmare, go ahead and spin up some old android vms and you'll see what I mean
5:36:08
aeth
loke: and Minecraft was poorly optimized Java, which gave Java gamedev a bad reputation
5:37:13
loke
Also, Android now has multiple implementations as well, so one has to be specific when talking about Dalvik, or ART, or the thing that ART has become in Android 8.
5:37:35
jmercouris
it is only recently in my mind that I've begun to distnguish between language and implementation
5:37:54
loke
As far as I can tell, the VM in Android 8 seems a lot better, but I haven't pushed it very hard. :-)
5:38:03
aeth
jmercouris: And my non-consing style is actually unnecessary, I roughly lowered 4%-5% CPU usage to 2%-4% CPU usage
5:38:39
aeth
(Although that reduction in CPU usage included replacing efficient SIMD linear algebra that consed with naive not-SIMD linear algebra that didn't cons)
5:38:47
jmercouris
loke: The VM in Android can get as good as it likes, it's still a VM and, bare metal is always best
5:39:10
jmercouris
as much as I like the java "code once, run everywhere", the experience is still a pipe dream, and it is 2018 now
5:39:27
aeth
jmercouris: I'm writing a game engine because it prioritizes CPU performance without having any real RAM limitations. That's a fun combination for me.
5:39:42
jmercouris
|3b|: I remember when NDK was so terribly documented, there wasn't even ANY documentation
5:39:54
loke
jmercouris: The question is how close to Java the VM is... The problem with the JVM is that a lot of its features are designed to make Java very efficient, but makes it hard to implement some features of other languages (efficiently).
5:39:55
jmercouris
I also remember having to edit special strings in eclipse to load your APK onto different phones
5:40:55
jmercouris
loke: I have no problem with the JVM speficically, I have problem with VMs in general
5:41:18
DevilsDulcimer
anyone have any experience working with outsourcing? preferably an American company working with Indians over Telecommuting solution such as skype...are they a viable lot when it comes to going forward with a tech company?
5:41:24
loke
jmercouris: THat is fortunately no the case now. OWEVER, it's worse in another way. Now the entire ecosystem is based on Gradle, which is probably the worst piece of shit build system I ever had the misfortune to be force to work with.
5:42:07
jmercouris
loke: Gradle is a war crime, I'm not sure why they pushed that and the Intellij train so hard
5:42:27
loke
jmercouris: IntelliJ is great, but what they did to it when thet made AS was a disaster
5:42:50
loke
jmercouris: I used to do ANdroid development in IDEA, and clicking on "run" compiled an, built, and deployed on the emulator in about 5 seconds.
5:42:59
aeth
If there's a quality outsourcing firm with Common Lisp experience, maybe we could crowdfund some patches to key CL projects.
5:43:20
loke
jmercouris: With Gradle and AS, that increased to about a minute. It's better now, if you're lucky enough that the damn thing doesn't roll over and die on you.
5:44:09
loke
Oh, and if you _DARE_ to try to load load an old project after 3 weeks of not using it, it won't beuild anymore because Gradle decides to upgrade half your dependencies and nothign will work anymore, and you are force to cargo-cult random changes in your Gradle project file to get it to run again... MAYBVE
5:44:56
jmercouris
loke: Lol I love those stack overflow "I changed x to y" in these lines for upgrades which are basically breaking against user will, and undocumented
5:44:58
loke
The funny thing is that I listen to the Android Developers podcast (by the Android team at Google themselves), and they'
5:45:27
jmercouris
What's wrong with maven? what's wrong with manually coyping some jars when you need them?
5:45:56
loke
jmercouris: I absolutely abhor maven, and it's quiote telling that I'd gladly commit to Maven 100% just to be able to get rid of Gradle.
5:47:15
loke
The worst par tis that Gradle is just a script, with a horrific syntax hack to make Groovy look like a DSL. That means that every single line is executable and then AS attempts to actually parse this thing to make sense of it.
5:48:06
jmercouris
I didn't even know that, but it doesn't exactly shock me, that's some next level stuff... in a bad way
5:48:14
loke
Damn it... Just thinking about that crap makes me angry. I need to go have lunch now and cool down.
5:48:40
beach
jmercouris: Reference counting is way more expensive than tracing collectors, and on top of that, it doesn't work for cycles.
5:49:05
loke
jmercouris: Try it next time, pluck some Groovy code in there (it's essentially Java with a more scripty syntax, but most Java constructs still work) and see the... erm... beauty
5:49:48
jmercouris
beach: of course reference counting will also fail for cycles, it's just made for lightweight applications, but I see your point
5:50:27
jmercouris
How so? reference counting is MOST DEFINITELY lighter, you simply maintain a hash table of counts, in fact, you could even maintain a sorted hash table
5:50:45
beach
jmercouris: By reference counting, we mean that when a variable is assigned to, the object previously contained in it has its reference count decremented and tested for 0.
5:51:31
beach
jmercouris: Yes, so every time you do an assignment, you have to go to memory and update a reference counter. On modern processors that kills performance.
5:52:13
loke
jmercouris: The issue with refcounting is that you have to access this refcount variable all the time, and in a multithreaded environment you also need some way of dealing with data races
5:52:17
jmercouris
Perhaps assignment will be slightly slower, eg. intialization code, but garbage collection will definitely be faster
5:52:31
beach
jmercouris: And C# uses garbage collection as well, so there is nothing related to garbage collection that makes C# better suited for anything than Common Lisp.
5:52:58
rme
reference counting spreads out the pain. with a tracing gc, you suffer the pain all at once, but you get a bulk discount.
5:54:32
jmercouris
not some form of automatic reference counting, not some form of marking relations, but literally incrementing referencing counts manually as a user
5:55:14
jmercouris
it's just BARELY different than manually alloc and freeing memory, but can mean a performance boost in some cases where you know you don't need to actually now mark something during a transient operation (for example)
5:56:23
loke
jmercouris: There is also the issue of bookeeping. In C, for example, both allocating and freeing memory requires the malloc implementation manage a tree of free memory structures which can be quite expensive. In a compating GC language, a malloc() is essentially a single instruction (an ADD, moving the free memory pointer), and a free() takes zero cycles, since you simply drop the pointer.
5:56:56
loke
Of counrse the GC intself takes some reasoucres, but it's not clear-cut which one wins.
5:57:01
beach
jmercouris: Earlier you asked about CONS cells being freed. Did you think that some version of free() was being called in order to free those cells?
5:58:33
jmercouris
I don't believe so, I literally just this past year studied GC, within the context of distributed systems even
6:00:23
jmercouris
loke: Yeah, that was more or less my question about what common lisp implementation is likely doing when freeing a cons cell
6:00:53
jmercouris
CL has surprised me in so many ways so I keep asking questions that I know the answer to in other languages, but I'm never sure
6:01:05
loke
jmercouris: I presume you are referring to the work neede by the GC to release the memory used by CONS cells that have been released?
6:01:44
beach
As loke says, a modern GC does not touch unreferenced objects, so it does not "free" any cells. And this is not specific to Common Lisp. It has to do with modern GC algorithms.
6:02:33
loke
jmercouris: By the time the GC frees the unused memory, that memory dones not consist of CONS cells any more. It's just a blob of memory that no one cares about :_)
6:02:48
aeth
I heard before in this channel that SBCL doesn't have a good GC and that CCL's GC is better.
6:03:16
loke
aeth: There are certainly problems with the SBCL GC. It could be a lot better. But it's not terrible.
6:04:27
loke
aeth: well, of example, there are cases where it needs to fill a structure with zeroes instead of just dropping the reference in order for the GC not to mistake random values with a pointer. It was disucced on the SBCL devel mailing list last week in fact.
6:09:01
jmercouris
beach: here's where I was asking that question: "basically on a system level here, when we run a GC in lisp, are we actually freeing the memory, or is it just now available within our loaded lisp image as "free" memory?"
6:10:01
beach
jmercouris: But most GC algorithms, just like malloc() and free() do not give memory back to the operating system.
6:15:00
fouric
"By the time the GC frees the unused memory, that memory dones not consist of CONS cells any more. It's just a blob of memory that no one cares about"
6:15:04
jmercouris
aeth: Presumably always one would know more than they do because knowledge always precedes action
6:15:34
fouric
...or do we not consider them "cons cells" because for our intents and purposes they aren't?
6:20:09
beach
fouric: Because all information about the exact contents of that memory has been lost. At some level, there are still Common Lisp objects there, of course, but as far as the system knows, it is just random garbage.
6:24:22
fouric
...and one more question since we're talking about GC anyway: can SBCL's garbage collector run concurrently with running programs?
6:24:42
fouric
i poked around in the manual, which says that it's a "generational" garbage collector
6:25:09
fouric
...but i haven't been able to determine if that says anything about its ability to run in parallel with the program it is managing
6:26:52
krwq
soes anyone know how to write strings to in-memry-stream with flexi-streams? I've tried with-output-to-sequence + setf flexi-stream-external-format and also wrapping that stream with another flexi stream with :external-format set
6:28:41
fouric
(...would you happen to have any suggested resources for reading up on garbage collection and memory management?)
6:29:51
krwq
actually bummer - it worked correctly but I misread the error (it was on file-length call)
6:30:14
rme
https://ccl.clozure.com/docs/ccl.html#understanding-and-configuring-the-garbage-collector might be worth a look, too.
6:32:15
aeth
Looks like I just learned that the magic words that a game would need are "real-time concurrent garbage collection"
6:34:36
rme
sometimes use the phrase "stop the world" to describe collectors which are sort of the opposite (and which are more tractable to implement and thus more common)
6:40:12
fouric
ACTION is not very good at search-engine bingo, which is why he asks for recommendations from humans whenever possible
7:52:11
jmercouris
what's the best way to create a config file that can be loaded by a standalone lisp program?
7:52:33
jmercouris
I've forked "site-generator" and I want to be able to pass it a series of commands to execute in the form of a config file
7:53:00
jmercouris
e.g. if I launch "site-generator" in a dir with a config file, it should pick up that config file
7:53:26
jmercouris
I guess the real question I am asking is, how can my program know from where it was launched in order to load the appropriate lisp file?
7:58:41
Shinmera
If it's a source file, the usual trick is #.(make-pathname :name NIL :type NIL :defaults (or *compile-file-pathname* *load-pathname* (error "Welp"))
8:31:09
jackdaniel
what's more, *default-pathname-defaults* and current working directory may be inconsistent (because one or another may be adjusted at runtime independently)
8:46:48
p_l
at least it's rare these days to write lisp for OS that doesn't have the concept of CWD ;)
10:20:04
hjudt
is there an easy way to read a whole text file into a string instead of reading it line by line and concatenating?
11:33:59
jackdaniel
you may want to read about the differences between ~s and ~a (readably vs unreadably)
12:20:42
genos
(sudo rsync --verbose -r --ignore-existing ~/Downloads/books/* /media/${whoami}/247B-947E/books)
12:20:42
genos
(sudo rsync --verbose -r --ignore-existing /media/${whoami}/247B-947E/books/* ~/Downloads/books))
12:21:10
Shinmera
This is not the channel for elisp or emacs. And it's even less the channel to just paste code into.
13:07:07
Xach
scymtym: xml.location references lisplab in one of its systems, but lisplab isn't in quicklisp - should i add lisplab? or ignore the xml.location system?
13:19:41
scymtym
Xach: it used to be an optional dependency but is no longer relevant. i can just remove the integration
13:20:30
scymtym
Xach: i have been meaning to do that for quite some time but forgot about it. thanks for bringing it up
13:22:12
Shinmera
That depends on your file system, but generally no. You can write out a new file and delete the old one though.
13:30:50
billitch
the url scheme is subject of much more design than disk layout which is left to the OS
13:34:53
billitch
well i think the most common data structure is a variable length string which is the filename
14:18:16
pjb
(setf (ldb byte place) val) = (setf place (dpb val byte place)) ; but without the place duplication.
14:20:49
jmercouris
there doesn't appear to be a straightforward way to copy a dir in CL, is there one I'm not seeing?
14:21:42
jmercouris
I see uiop:copy-file and I guess I could copy the folder file, but I'm not sure how that would work
14:21:46
_death
jmercouris: copying a directory (or even a file) is not a straightforward operation.. it depends on what you want to use it for
14:22:21
jmercouris
_death: I'm trying to copy a directory of static assets for exporting a site-generator page
14:22:30
Shinmera
jmercouris: Use DIRECTORY, a wild-inferiors pathname, and COPY-FILE, and ENSURE-DIRECTORIES-EXIST.
14:23:14
_death
jmercouris: here's an old snippet https://gist.github.com/death/2b7b1cfe6a38fc4b368ba91419c75e2e
14:26:07
_death
jmercouris: depending on your situation, it may even make sense to invoke cp -R or somesuch
14:26:08
jmercouris
Shinmera: What do yo umean by "wild-inferiors" pathname? a pathname that includes a wildcard matching all child files of the dir I wish to copy?
14:27:15
Shinmera
jmercouris: For instance: (make-pathname :directory '(:absolute :home :wild-inferiors))
14:28:39
jmercouris
Shinmera: So i'd match all the dirs, and which I would then have to walk, and then match all the files and then copy them?
14:29:40
Shinmera
(directory (make-pathname :name :wild :type :wild :directory '(:absolute :home :wild-inferiors))) will list all files reachable from your home directory.
14:30:46
jmercouris
Right, so I would still need somehow to know the tree structure of the dir I'm copying over
14:31:35
Shinmera
You truncate off the common path to make it relative, then make it absolute to the destination directory again. Then use ensure-directories-exist to make sure the folders are there, then copy.
14:31:42
jmercouris
That's true, but unless there's a copy-file flag that will construct all the necessary folders to get to a path, I'd still need to somehow do some processing no?
14:33:51
Xach
For example, getting a list of relative pathnames via DIRECTORY on a given base directory has a pitfall to carefully avoid.
14:34:18
Shinmera
Yeah, might do symlink resolution or truenaming, so your paths might not all have the same root.
14:34:22
Xach
if the base pathname isn't a truename, the results from DIRECTORY won't be amenable to enough-namestring.
14:34:49
_death
ACTION wonders why that snippet didn't give him a warning as it's missing a symbol in import-from
14:35:36
jmercouris
would be a nice utility for someone to write though, or nice to have in uiop like uiop:copy-dir
14:37:08
Xach
It sure would be nice to have a utility like that. I can understand why it's hard to get right, though.
15:42:59
rumbler31
not bad on the east coast of the US, waiting for it to make up its mind about the cold
15:46:04
JuanDaugherty
ACTION is only wanting to destroy all humongs by merciful means, does that count as a good day?
16:06:46
Xach
http://report.quicklisp.org/2018-01-26/failure-report/claw.html <-- argument count mismatch
16:38:02
pjb
beach: I have a suggestion for sicl (and really, for all CL implementations): annotate the function with the special variables used, and issue warnings or print notices about them (systematically and automatically). Notably, for implementation operators. Again this week somebody in #ccl had a problem that a ccl function to open a network stream failed, when *print-base* or *print-radix* were different from the default value; thi
16:38:02
pjb
function used princ to convert a port number to a string… This can be considered a bug, but in any case, it should be documented!
16:56:01
pjb
beach: by annotate here I mean in the compiler data structures. So that it may issue the warnings or write notices on its output.
16:57:27
rumbler31
paule32: in half adder you also setf a and b, but take in i1 and i2. I don't think these would explain your errors though, since a,b,c are globals, but maybe the fact that they are special is messing things up
16:58:44
makomo
pjb: hmm, why would you need to document it if it's only used internally by the function?
16:59:12
makomo
if we suppose that the function correctly binds *print-radix* and w/e it needs before PRINC-ing
17:02:07
megachombass
if i manage to compile fibo (means be able to compile constants, a defun and a IF) ill be good
17:04:47
pjb
beach: additionnal similar annotations could be about side effects: mutation of bindings, mutation of lisp objects, and mutation of external state (file system).
17:05:02
shrdlu68
megachombass: You'd be better of using "CL" instead. clisp means something different to most people here.
17:05:55
pjb
beach: the problem being that programmers have a mental image of what state is used or mutated by functions, but it is not always exact (eg. a lot of *print-…* variables are ignored for non-I/O functions…), and it's in general unknown for implementation specific functions.
17:07:11
pjb
makomo: yes, indeed if there is a local dynamic binding of a dynamic variables used by a function, then it can be removed from the list of dynamic variables usedby the caller.
17:10:05
makomo
megachombass: clisp happens to be the name of a particular cl implementation. that's why you should use "cl" instead when referring to common lisp as a language
17:11:57
pjb
megachombass: it's easy, Common Lisp comes from a package named "COMMON-LISP" and with a nickname "CL". So call it common-lisp or cl.
17:12:19
pjb
"LISP" was an old, pre-standard nickname. This is why #lisp is #lisp and not #cl or #common-lisp :-)