libera/#commonlisp - IRC Chatlog
Search
8:48:02
Duuqnd[m]1
I would do that, but the part that writes to the socket is in a library that I don't really want to modify
8:48:28
Duuqnd[m]1
This program only has to run on SBCL (because I'm the only user) so I can live with some uncertainty
8:49:20
Duuqnd[m]1
There's lots of functions that call the socket writing function and putting a lock around each one would be very inconvenient.
8:51:22
hayley
Well, my apologies then. But having your program break because someone decided to change undocumented behaviour, or worse, having it break completely randomly (cause that's threading for ya), is also very inconvenient.
8:52:54
Duuqnd[m]1
It is indeed. I have to choose between blindly trusting SBCL or making the program much worse to use and I don't like either option. Maybe I'll just add locks in the library after all.
8:54:36
Duuqnd[m]1
All this is because if the socket connection dies while it's listening I have to try to send something down the socket for it to notice that it's dead.
8:55:16
Duuqnd[m]1
A separate thread that pings the server every 10 seconds was the first solution I came up with (admittedly not a great one but the whole program is a soup of bad decisions)
8:56:05
Duuqnd[m]1
I can hardly believe it held out long enough to bridge IRC and Twitch for the two days that ELS needed it
9:01:44
Duuqnd[m]1
I guess I'll be using the worst possible solution: Assume it works until it breaks.
9:02:41
Duuqnd[m]1
I'll probably end up replacing the library with my own thing anyway since CL-IRC doesn't seem to support SASL which Libera usually wants.
9:11:31
pjb
Duuqnd[m]1: you can count on it, on the contrary: count that concurent write-sequence WILL send their data randomly mixed on the socket.
9:12:39
pjb
Duuqnd[m]1: basically, as soon as you have a loop, it's not atomic. Obviously, write-sequence contains loops.
12:07:18
Cymew
lotuseater: You were not alone in your reaction to that expression. I did a double take before I re-parsed it.
15:35:55
q3cpma
Hello, I've finished my wordcount program (http://0x0.st/-Y-W.lisp), but it somehow manages to blow SBCL's 1GB heap just during the word gathering part while processing a 40 MB file. Any idea why?
15:44:08
etimmons
I just finished the first release of my summer side-project on a pure CL library for reading and writing tar files: <https://common-lisp.net/project/cl-tar/>
15:44:56
etimmons
phoe: I was thinking of you when I wrote the extraction code (it uses a lot of conditions and restarts)
15:46:26
q3cpma
Even if it doesn't work, I would expect at most twice the memory usage, to be honest
15:48:35
q3cpma
By the way, related question: can I trust the output of (room) concerning memory usage?
15:49:05
etimmons
yitzi: I'm getting messages from IRC again! I think the culprit was Alex . When they left the room I started getting things again.
15:49:55
etimmons
Would still love to prevent it from happening again, but it requires a mod to tweak a setting.
15:50:49
yitzi
Yeah, I just switched to matrix and it is pretty nice so kind of irritating to have a hickup.
15:52:56
random-nick
I'm guessing the shared structure ppcre refers to is array displacements, which shouldn't be making any weak references
15:54:57
random-nick
q3cpma: references which the garbage collector doesn't follow (i.e. something can still be garbage if there's a weak reference to it) and automatically invalidates when the referenced object is collected
15:56:15
q3cpma
I see, so what does a displaced array becomes when the parent is reaped (or is it reaped)?
15:58:45
random-nick
as in, a "living" displaced array should keep its parent "living" or at least its storage
16:03:45
q3cpma
With a 4MB input file, I get 120MB of dynamic usage vs 40~45MB just for SBCL (using room)
16:12:22
q3cpma
Here's the "final" version: http://0x0.st/-Y-G.lisp, NOT using sharedp allows me to process the 40MB file, but I still get 490MB of memory use (323M in 8.5M simple-character-string and 143MB in 9M cons).
16:14:20
q3cpma
And there are 8M words in the input file, so the object count figures aren't absurd, only the mem use
16:23:19
mfiano
sb-ext:primitive-object-size and the other memory introspection functions may be of help
16:25:37
q3cpma
scymtym: what's fun is that room reports 55MB of array-headers IF I use sharep, but not otherwise
16:27:29
scymtym
q3cpma: you previously mentioned SIMPLE-CHARACTER-STRING. those have header data as well but probably not in the sense of that ROOM report
16:27:35
q3cpma
Breakthrough: enclosing the code in a dummy main functions completely solve the problem
16:28:29
q3cpma
Man, I left Tcl where only procs are bytecoded to find CL where I get something similar. What a strange world.
16:39:34
q3cpma
So, playing with SB-EXT:PRIMITIVE-OBJECT-SIZE showed me that strings of size 0, 1-4, 5-8 weigh respectively 16, 32 and 48 bytes.
16:42:02
q3cpma
So, is SBCL not suited at all for string processing, or am I supposed to do something?
16:50:58
mfiano
Using type declarations is undefined behavior anyway. Infact, relying on the internal representation of the string types to stay consistent is bad. Just program Common Lisp instead of an implementation.
16:53:48
mfiano
I don't agree with implementation specific assumptions. We have a standard for a reason
16:56:00
q3cpma
saturn2: it is a lot better, but the original problem was an OOM crash during processing
16:57:06
mfiano
SBCL's default heap size is incredibly small. Personally I set this to half of my physical ram, 32G. It won't all be allocated unless needed.
16:57:44
q3cpma
Maybe this isn't a Lispy way of thinking, but when I have to use 1G for no apparent reason, I'm a bit alarmed
16:59:38
saturn2
the default heap size is small, and also the default settings tend to let a lot of uncollected garbage build up
17:00:16
Bike
weird that it wouldn't run a collection rather than die from OOM, though. maybe something is actually kept alive? i don't see anything obvious...
17:01:28
saturn2
using displaced strings as hash keys seems like a bit of an out of the ordinary thing to do, maybe there's a bug?
17:02:11
q3cpma
saturn2: I switched to :sharedp nil, as it was causing a lot more memory usage (paradoxally)
17:03:45
mfiano
(sb-ext:primitive-object-size (make-array 15 :element-type 'simple-base-char)) ; => 144
17:05:56
saturn2
you can do (dotimes (n 7) (setf (sb-ext:generation-minimum-age-before-gc n) 0.1d0)) to make the garbage collector more aggressive
17:06:14
q3cpma
(sb-ext:primitive-object-size (coerce "abcdefghijklmno" 'simple-base-string)) does give me 32
17:07:11
q3cpma
Just for info, python does in 0.12s what the SBCL image with max speed declaim does in 3.94
17:11:05
mfiano
Since they have to do bounds checking on both arrays, and implementations don't really optimize displaced arrays as much as they could
17:12:39
CodeBitCookie[m]
Also I was thinking about McCLIM. I really love it. The big problem is that GUIs are made for people who can't use a terminal interface or it's just for an application like video editors which are better with a GUI. People today cannot stand to see something look even a tad bit outdated. I don't care about the looks but people who use my application will. Do you think McCLIM could be made pretty?
17:12:58
jeosol
I remember seeing a lisp paper that actually demonstrated it. I probably have it somewhere so it should be easily searchable on the web.
17:14:02
jeosol
my applications involving mostly number crunching, and I don't optimize things and use type hints for now since bottle neck lies elsewhere
17:14:18
q3cpma
And I really want to love it, because the language is beautiful (the stdlib, "a bit less")
17:14:47
jeosol
q3cpma: i agree, but with the gradual typing facilities and I am sure there will be ways to optimize things
17:18:11
jeosol
I think it depends on your application and problem. For me, I came from matlab and c/c++ (grad school), then I came across CL in a thesis, and I switch because of how easy to read the code was. And emacs+slime+paredit for me is a good IDE.
17:18:34
mfiano
It takes a lot to write efficient code. Understanding the standard in depth is a must, and the nuances of the implementation you are writing implementation-specific optimizations for
17:18:37
jeosol
I really think it depends on what you are trying to solve and what is expected from it
17:19:33
jeosol
once my application is extremely stable, then I plan to look into that, but I don't do any optimizations for now and things run really fast - and could be faster
17:19:59
jdz
I may be late, but was looking into SBCL's character reading from streams, and there's currently no way to read anything but strings with element type CHARACTER in SBCL. The only way is to COERCE after reading.
17:20:07
q3cpma
You can only appreciated such dynamic languages when you have tried horrible stuff with the C preprocessor
17:20:15
mfiano
You can do a lot at compile-time too, which is what cl-ppcre does, and why it is much faster than Perl, which has to interpret bytecodes
17:22:34
jeosol
the guys here are extremely and you'd get multiple perspectives on your questions here
17:23:02
q3cpma
Will never get over the inconsistency of the stdlib/naming, but I know already enough to persevere
17:27:54
pjb
q3cpma: the explaination is: READ parses "words" (symbols) and intern them in a case-insensitive way, in the current package (a temporary package). The hash table can be an EQL hash-table because the symbols are already interned. The rest is identical.
17:28:42
pjb
q3cpma: actually if the file contains special characters, READ will break because it'll try to parse lisp syntax (parentheses, numbers, etc).
17:28:51
q3cpma
Well, it doesn't seem to solve memory usage, to be honest, but that's a different and interesting solution
17:32:44
pjb
q3cpma: of course, it could be made faster, but it'd really be micro-optimization. Only if you had to work with google-size data would you need that. Are you doing a hiring test to apply at google?
17:34:25
saturn2
pjb: "Just for info, python does in 0.12s what the SBCL image with max speed declaim does in 3.94"
17:52:04
Josh_2
etimmons: Just seen your link for the tar project, very nice that will come in useful for me
17:54:41
q3cpma
etimmons: on that subject, since Tar has no real standard, what did you use as reference? GNU?
17:58:53
etimmons
q3cpma: it can read ustar, pax, gnu, and v7 tar files. It can write each of them as well, but I'm missing a couple of gnu extensions in each direction.
17:59:50
etimmons
Josh_2: great! Sorry to hear about 40ants doc, what errors were you getting? I had some mgl-pax code that I just ported over so I didn't exactly follow the examples to the letter
18:00:24
q3cpma
etimmons: GNU is the standard if you want sparse file and long names, these day, so nice
18:03:09
Josh_2
etimmons: the example on the github wont build. I run the (40ants-doc/builder:render-to-string @foo-random-manual :format :markdown) and get an error saying 'unable to figure out *package* for object :(
18:03:16
etimmons
q3cpma: PAX handles long names quite nicely as well. sparse files is one GNU extension I'm missing (in the high-level wrapper, at least). But it should be too difficult to add
18:04:53
q3cpma
etimmons: yeah, I'm mixing things up, the problem I had were with the only pax utility (mirBSD pax), and not the format itself
18:05:01
etimmons
Josh_2: Huh, that's weird. I'll try to work through the example and see if I can see what's up
18:05:37
yitzi
q3cpma: Did you try sbcl with --no-userinit to do that benchmark? For me it takes it from ~4s to about ~.5s
18:07:12
etimmons
Josh_2: I was using mgl-pax for a while and really wish the 40ants stuff wasn't forked and was instead merged into mgl-pax.
18:07:53
etimmons
Josh_2: But I think there was some strong differences of opinion and I really like being able to load my documentation tool *without* also loading ironclad, so 40ants it is
18:09:46
yitzi
It was just once with their simple.lisp ... in any case my point was that the startup code to load quicklisp/clpm can be significant.
18:20:31
q3cpma
yitzi: I gained <0.1s by using (asdf:load-system "cl-ppcre"), is it the way to do it?
18:21:10
Bike
i think yitzi means the time advantage of not loading quicklisp at all. using ql:quickload isn't really going to be much slower if you already have it downloaded, since it just does asdf:load-system anyway
18:22:08
yitzi
Correct. It might not make much of a difference since they are dumping an image and then executing that.
18:26:50
q3cpma
Thanks for helping me again, I'll come back another time to at least try to bridge the gap with Slowthon =)
19:26:06
etimmons
Josh_2: I just tried the example. Only problem I ran into was I needed to add the package to the section name in the call to `render-to-string` (so `foo-random::@foo-random-manual`).
19:31:52
etimmons
Josh_2: I did the equivalent of cloning <https://github.com/40ants/doc> and <https://github.com/40ants/commondoc-markdown> to local projects, and then `(ql:quickload :40ants-doc-full)`
19:34:15
Josh_2
Thats what I was quickloading, I'd then realize I obviously need builder and ignored-words
19:42:06
jmercouris
Josh_2 pointed it out, and macroexpanding it leads me to see that the documentation is very misleading
19:43:35
_death
it sets the bindings in parallel (rather than sequentially).. maybe you're conflating several meanings of the word
19:43:58
Bike
"More precisely, all subforms (in both the place and newvalue forms) that are to be evaluated are evaluated from left to right"
19:46:56
jmercouris
fair enough, but we are usually discussing in the context of the fourth dimension
19:47:00
Bike
as far as the programmer is concerned, they can take place at the same time. there's no way to determine what order they take place in, if any.
19:47:53
_death
with setf, it would be: (i) set a to the value of b (a=2,b=2) (ii) set b to the value of a (a=2,b=2)
19:48:37
Bike
for example, if your macroexpansion happened to expand to the two setqs in a different order, or starting two threads that do the setq and then joining them, you'd never know