freenode/#lisp - IRC Chatlog
Search
16:55:28
jasom
Anyone have a link to a blog article about memory usage on sbcl when running a service? Their solution was to manually invoke the GC periodically, and this caused the memory usage to stabilize sooner and at a much lower value
17:05:40
jasom
minion: memo for emaczen: you don't need to know the size of anything for searching a B-Tree; you only need to know the position of the child nodes.
17:11:11
gendl
Hi, using net.html.parser:parse-html on some utf-16 input - external-formats seem to differ in each Lisp I am trying to use:
17:11:55
gendl
and with SBCL, it is retaining the BOM from the beginning of the file (not so on the others).
17:12:48
gendl
Should I expect to have to conditionalize for SBCL to throw away that first character? Or is there a way to have it auto-detect the BOM and throw it away as the other Lisps apparently are doing? or should I ask this in #sbcl?
17:19:41
jasom
gendl: though if net.html.parser isn't set up to use babel than you may need to modify it or use something else
17:21:32
gendl
jasom: yeah, without overhauling what I'm doing, I think all I need is a portable mapping for :utf-16 external-format - i.e. a parameter *utf-16-ef* which maps to the correct keyword depending on the host Lisp -
17:22:00
gendl
my understanding is that Babel is both overkill and not applicable for that simple purpose... is that right?
17:23:03
gendl
... but i would have to retro-fit net.html.parser to call through babel rather than directly reading stuff, right?
17:23:36
jasom
gendl: correct, and it's been a while, but babel does have some code for handling BOM
17:26:55
jasom
gendl: I just checked and babel will decode any string specified as :utf-16 and starting with a BOM correctly
17:29:10
gendl
well then... looking at my code, maybe I can use babel in conjunction with net.html.parser...
17:29:35
gendl
the external-format is specified on the (with-open-file (in file-name) :external-format ... ) ... )
17:30:49
gendl
so i'm not sure if babel can be used in lieu of the standard with-open-file, or somehow in between the with-open-file and the parsing...?
17:30:51
jasom
gendl: so you can use babel-streams then, assuming your HTML fits reasonably within memory
17:32:25
jasom
gendl: so yeah, if you slurp the file into an octet vector, you can use babel's with-input-from-sequence
17:34:25
gendl
it looks like with-input-from-sequence is a flexi-streams thing rather than a babel thing.
17:37:44
jasom
ACTION needs to finish up his portable unicode stream reading library. UTF-8 is within 20% of the speed of a bare stream on sbcl (sbcl wins only because it generates a conditional move where I generate a branch; I haven't figured out how to massage the compiler there).
18:12:05
ecraven
does slime have a command to copy the output of the last command? I can clear it, but there seems to be nothing to copy it to the kill ring?
18:15:54
pjb
ecraven: you could write a command that would move the point to the previous presentation and use slime-copy-presentation-at-point-to-kill-ring.
18:26:25
rpg
When I copy stuff into the REPL in SLIME sometimes I get errors where a list is mistakenly treated as a funcall...
19:06:04
Xach
Hmm, is there a sly repl command to search for previous input in history? I use M-r for that a lot in slime.
19:29:25
didi
In SLIME, can I limit the number of elements of a sequence it will print? If my sequence has many elements (> 1 M) my slime lose the connection to the inferior process. Emacs also consumes lots of RAM in the process.
19:37:24
Bike
i usually just set it to 20 or 50 or something. not like i'm gonna get much out of longer lists or strings
19:37:38
pjb
The thing is that you may want a different *print-length* in the normal REPL, the slime REPL or C-u C-x C-e in buffers.
19:41:29
mtl_
anyone here remember how to define progn as a macro? Just trying to explain it to a friend, but I'm too rusty(haven't touched any code in months) and tipsy to remember
19:42:49
Xach
mtl_: there are many ways, but (defmacro .progn (&rest forms) `((lambda () ,@forms))) is one option
19:44:04
mtl_
again not for serious programming just to illustrate a point to a friend who's learning lisp
19:44:26
Bike
you can do pjb's, or you can do like (defmacro progn (&body body) (cond ((null body) nil) ((null (rest body)) (first body)) (t `((lambda (x) (declare (ignore x)) (progn ,@(rest body))) (first body)))))
19:46:38
pjb
(defmacro progn (&body body) (cond ((null body) 'nil) ((null (rest body)) `((lambda () ,(first body)))) (t ((lambda (x) (declare (ignore x)) (progn ,@(rest body))) ,(first body)))))
19:47:32
pjb
Here we have a recursive call to the macro body, so the total macro expand is like (lambda (x) (lambda (x) … (lambda (x) last-body-form) …) first-body-form).
19:48:55
Bike
what this relies on is that lisp is call by value, which is kind of interesting, but kind of far away from progn
19:49:02
pjb
Bike: Sorry, I'm sleepy. I thought you were defining .progn, not progn, and your recursive call was cl:progn instead.
19:55:20
mtl_
was just a bit rough to jump straight into recursive macros after not touching lisp for months(or any programming in general)
20:01:57
Bike
(defmacro progn (&rest forms) (let ((s (loop for f in forms collect (gensym)))) `((lambda (,@s) (declare (ignore ,@(butlast s))) ,(first (last s))) ,@forms)))
20:02:14
Bike
saying that progn can be a macro basically amounts to saying other parts of the language have sequencing
20:05:16
mtl_
well I'm not getting into the gritty details, just trying to illustrate the power of macros to my friend
20:07:48
pjb
You probably need to insert this nil in your lambdas too to disable declarations in the body.
20:11:29
phoe
you could dig ppcre and trivia. they both generate complicated code at macroexpansion time to save up on cycles during runtime
20:13:25
pjb
Sure. You can implement everything in lambda calculus, so you can expand everything to lambdas.
20:13:34
Bike
in scheme let is a macro in terms of lambda. it's more difficult in lisp due to declarations and speciald and stuff.
20:14:16
mtl_
yeah you can turn it into a lambda function call with all the names being arguments to the lambda
20:31:53
minion
emaczen, memo from jasom: you don't need to know the size of anything for searching a B-Tree; you only need to know the position of the child nodes.
20:35:23
phoe
emaczen: for very implementation-specific stuff, implementations have their own channels.
20:38:45
Bike
sizeof is implementation specific in C (which is why it exists), and C has a lot more restrictions on memory than lisp does
20:40:31
emaczen
Bike: Is it completely unreasonable for language designers to enforce memory restrictions in their standards?
20:42:57
Bike
it's not completely unreasonable, but imposing perfect restrictions of "this object takes up N bytes" wouldn't be very reasonable either, for almost all applications.
20:44:28
Bike
first off it's usually pointless, and secondly the language would be obsolete as soon as a new system with differently aligned loads or something appears.
20:45:08
Shinmera
Plus forcing memory constraints eliminates a good class of potential optimisations an implementation could do.
20:46:04
Bike
like what, "a word is some implementation defined number of bytes, and a cons is two words"?
20:46:55
emaczen
Bike: nvm, I'm just going to experiment, call (room) use :space 3 and see how much memory I am using up
20:48:00
Bike
reasons to need an exact memory layout include understanding resource usage of a program and communicating with systems that are that exact, and... that's all i got off the top of my head
21:07:18
jasom
emaczen: also for a database like b-tree usually you store your index separately from your data, so your values are just offsets into the data file. In any event, you should create your own fixed-sized layout for your tree nodes, since the whole point of a B-Tree is to use disk-friendly node sizes
21:08:19
jasom
emaczen: if the underlying file system uses fixed logical block sizes, that makes sense, since logical blocks will be contiguous in LBA space (which usually means fast reads)
21:16:43
pjb
emaczen: In general the O(.) in space of each object is not implementation specific, but the constant factors are.
22:22:03
pillton
tjb Bike: I'm not sure I follow. Jasom's interpretation of that part the CLHS makes it possible for a compiler to inline a function or use a different entry point to the function.
22:25:11
pillton
I'm fine with performing optimizations on one function given an analysis of the operators it calls.
22:26:31
jasom
pillton: it's a compromise betweeb efficiency and dynamicsm; by default require people to recompile a single file (i.e. C-c C-k in slime), which is pretty quick when making changes while developing, but give people a way to make functions freely redfinable (via the declare)
22:28:07
jasom
recompiling a single file has been a typical way of making changes going back to the 70s at least, so they don't want to break that, but they do want to leave room for the compiler to optimize.
22:29:35
Bike
pillton: you can't do any analysis of the operators it calls if they can be redefined arbitrarily
22:36:20
jasom
pillton: the compiler will also do dead code elimination without being requested to... in general most optimizations are not explicitly requested by the user
22:39:33
pjb
jasom: I think it's more linked to the way separate compilation generates independently linkable modules. So internal references are resolved by the compiler (by direct jumps or inlining), while external references go thru the linker symbolic tables, the equivalent of lisp symbols.
22:40:16
jasom
pjb: I don't think so since traditional lisp systems didn't work that way, and function definitions can be moved around in memory
22:40:22
pjb
Bike: in any case, analysis requires global knowledge of the program: it could as well redefine all the functions in a file!
22:41:12
pjb
jasom: were there files on lisp machines? just kidding, but I mean unix may have strongly influenced this part of the design of the language.
22:42:05
pjb
As soon as you have a linker, you have this distinction between internal references compiled in directly, and the references across modules, resolved by the linker.
22:42:47
Bike
if it redefines all the functions in the compilation unit, any optimizations in the previous versions are gone with the functions, so no big
22:42:51
jasom
pjb: I maintain that it is unrelated to unix style linking, but am willing to be proven wrong. There should be notes somwhere on why this was decided upon.
22:45:36
pillton
I assumed you could only eliminate code in that example according to an ftype declaration present in the environment.
22:47:34
Bike
i guess an explicit "this form is wild" annotation could be nice, but in practice it seems okay
22:51:43
Bike
i think that might technically allow dead code elimination of the kind you imply. dunno
22:59:26
jasom
I just always use C-c C-k, but you can use notinline as well. Also no sane implementation would do these optimizations when you set (debug 3) (speed 0), so that's another option
3:06:22
didi
You know... I miss having a COND clause that I could use the result of `test-form', e.g. (cond ((test-form) => (use test-form-result)))
3:19:34
stylewarning
At which people were amazed that a few carefully chosen optimizations made code 50x faster without sacrificing code quality.
3:20:34
stylewarning
Mostly techniques in Common Lisp for identifying areas ripe for optimization, and how to do that
3:28:32
stylewarning
The answer is "probably yes". Working out some details now since I came back from vacation
3:34:30
stylewarning
Yes of course. I'd argue things are a little more nuanced than what he presented (in my opinion, IIRC)
3:38:58
stylewarning
For instance, consider extending his pixel problem to be an array of structs of RGB values
3:40:39
stylewarning
Idiomatic Lisp would be a simple array of structs. You could write special accessors into a specialized array of floats that emulates a flat array of structs. You could even construct a struct when you pull the array out.
3:41:16
stylewarning
You could even remember a pointer to the array the struct came out of so you could setf the struct to transparently setf the array
3:42:34
stylewarning
I would guess that even with all of that involvement, none of it will likely beat some SIMD craziness if a flat array of structs in C.
3:43:26
stylewarning
(One of my long term goals is to think about CL extensions to give you some of that additional power.)
3:45:33
stylewarning
(Franz did some stuff like this and has a YouTube presentation about it. https://youtu.be/S7nEZ3TuFpA Not quite what I want. I want primitives for pinning objects and getting pointers basically.)
3:59:51
mfiano
I would not say that aeth. We've told you time and again, most of a game is not even on the CPU. SIMD is nice, but unless you are doing some crazy physics simulations, a typical game will be using a very small percentage of your processing power.
4:04:42
aeth
mfiano: And I've said repeatedly that that common wisdom only applies to graphics-heavy, AAA games that no one here is making and that simulation-heavy CPU games (not even necessarily physics simulations) can still definitely live on the CPU.
4:05:51
mfiano
Which puts you in the domain of your statement, not the lispgames community. You are the only one I know of that spends years optimizing low level game math rather than making anything remotely like a game.
4:06:12
aeth
RTSes and building games (especially the subgenre of building games with stockpiles) can still fill the CPU.
4:08:13
aeth
Bike: the problem with building games is you can have hundreds or thousands of units doing things at once, on the CPU
4:10:48
Bike
stylewarning: anyway, i'm probably going to try to add arrays of structs to clasp at some point. we need it for chemistry stuff.
4:11:57
stylewarning
The semantics of that reference, it's out-of-line modification, storage, identification on the heap
4:13:37
aeth
stylewarning: What I've been doing is retrieving and setting multiple values in 2D arrays, and covering up that messiness with macros.
4:16:23
stylewarning
Aeth, the most reasonable way I've found to hack it is to just internally allocate as a struct of arrays
4:23:39
stylewarning
Just a macro DEFSTRUCT-ARRAY that's like DEFSTRUCT but the accessor takes an integer N
4:26:05
Bike
yeah, maybe something i should prioritize... dealing with gc might be annoying, but so it goes
4:34:46
vtomole
So anytime a variable is created, that symbol is stored in the environment along with it's value? Are function names also "variables"?