libera/#commonlisp - IRC Chatlog
Search
13:36:53
jeosol
just came here to thank all the sbcl devs, upgraded from 2.2.6 to 2.2.7 without any issues
13:38:15
jeosol
I work with Python (no language wars) but I very much appreciate this about working with CL/SBCL, most upgrades give no issues except small ones (like compression lib change with 2.2.6*)
14:18:58
rendar
i guess that getting '(' while reading a symbol name is an error, all must be space separated, right?
14:20:26
NotThatRPG
ASDF is ready for a 3.3.6 bugfix release, with a bunch of fixes. Per semantic versioning, this should be entirely backwards compatible. I encourage people to download a copy of 3.3.5.10 and try it out with their systems. Bugs can be reported at https://gitlab.common-lisp.net/asdf/asdf/-/issues Thanks!
14:26:44
phantomics
Morning everyone, does anyone know the fastest way to keep a count of active workers in an lparallel kernel? I want to make the count frequently so I can efficiently assign tasks to inactive workers and frequently calling (task-categories-running) seems like it will have substantial overhead
14:26:47
White_Flame
rendar: and the main term involved here is that #\( is a terminating macro character, eg it will terminate any token being built and start a new one
14:27:28
NotThatRPG
Get your pre-release ASDF here: https://gitlab.common-lisp.net/asdf/asdf/-/wikis/Pre-release-ASDF
14:29:36
jackdaniel
last time we had libraries versions in the topic the length was quite big, and there was no clear criteria what should be listed there
14:38:33
jeosol
does wrapping a call in handler-case result in performance hit? I am assuming it's neglible. Context: I am trying to make a task run as fast as possible
14:39:53
jackdaniel
(i.e when there is no non-local jump, then there shouldn't be any performance hit, otherwise - I guess that it depends)
14:40:58
scymtym
many implementations cons a little bit and bind (as well as later undoing the binding) a dynamic variable
14:41:37
beach
jackdaniel: The handler code, yes, but the dynamic environment must be modified somehow.
14:42:30
jackdaniel
yes, but binding a special variable is not overly expensive (unless someone is really desperate, but at that point perhaps common lisp is not the right choice)
14:44:34
White_Flame
here's the disassembly for a single hander-case around a single function call: https://plaster.tymoon.eu/view/3337#3337
14:44:36
scymtym
consing a handler binding, manipulating a dynamic variable (including undoing the binding for any kind of exit) can far exceed the execution cost of some operations
14:47:07
jeosol
right now, I can get my calls, I do a 1000 such calls, to run around 1-1.5 seconds. sbcl in docker takes 3.6 seconds
14:47:08
Bike
i mean, i said it before you even mentioned safety 1, so obviously i wasn't referring to you
14:47:54
Bike
if you're on sbcl, it has pretty good profiling tools for figuring out what particular is taking up time
14:48:06
aeth
If you have (iirc) CASE or some arithmetic, (speed 3) is basically free speed. The best kind of optimization. Might hurt compilation times or inspecting with the debugger or something, otherwise idk why it's even configurable instead of the default
14:48:53
jeosol
Bike: Good point. I will look into that. There is an external call to a Fortran exe (can't optimize this at all). Beside that call, I want everything on the SBCL side to be tight as possible
14:49:13
jackdaniel
aeth: you've said it - might hurt inspecting; examining what's wrong is quite important
14:49:43
Bike
cos like if you want to catch errors you need to have it, but if you're doing something more specific there may be alternate solutions
14:50:07
jeosol
Bike: sometimes I may get input that makes a call go wrong. I have hunchentoot in front, so I just want to fail silently
14:50:53
Bike
and your failure is in the form of an error you don't control? because in that case you're not going to be able to lose the handler-case
14:51:09
White_Flame
jeosol: so there's no loop involved, just the performance of a single chain of events?
14:51:50
White_Flame
Bike: prechecking might speed it up, because then type inference knows what's up
14:51:54
Bike
jeosol: the sbcl manual explains exactly what it does with types in 4.2 "Handling of Types", including defclass slots
14:51:59
jeosol
White_Flame: I am doing a series of calls to hunchentoot server that runs SBCL task. The ratio of server request time and sbcl task is 0.99 so far
14:53:12
jeosol
Bike: thank you. I try to specify the types at least for documentation. With recent updates, I recall SBCL complained when I have a type as float but left :initform nil (from old code)
14:54:49
NotThatRPG_away
Thanks jackdaniel -- it would be good to get feedback before the release. Especially from users of implementations I find it hard to test locally.
14:55:14
scymtym
jeosol: yes, the :INITFORM vs. :TYPE check is a relatively recent improvement. people are talking about (defclass foo () ((%bar :type integer :writer (setf bar)))) (setf (bar (make-instance 'foo)) #\c) signaling a TYPE-ERROR at runtime which only happens with elevated safety
14:59:57
Bike
on sbcl, one of the nicer things about speed 3 is that it will make the compiler give you an overwhelming amount of advice on how to make your code faster
15:02:53
jeosol
so what's a generally strategy in using speed and safety. You go up to safety 3 to ensure everything works ok, and then do safety 0 and speed 3?
15:04:00
jackdaniel
0 is for local code that you are absolutely certain that it is correct under any circumstances
15:04:45
pjb
If you don't care what the result will be, then why not use safety 0 or 1. But then also, why run the program at all???
15:05:20
jeosol
just looked at the sbcl manual. there is a comment about corrupting the heap with "no type checks" when (= safety 0)
15:06:19
aeth
maaaaaaybe there's a bounds check that you can eliminate inside of a LOCALLY where you're absolutely sure it's impossible to go out of bounds with
15:06:23
jeosol
I have never configured these before and just used default setting. Now everything runs well, I want to optimize a bit and remove unnecessary code paths
15:06:50
White_Flame
yeah, hit speed 3 and (try to) follow SBCL's notes as far as you care to take it
15:08:31
jeosol
aeth: no optimization context per se. This is following from a previous convo with you on SBCL in bare metal repl and SBCL in docker. The former is my benchmark and the latter something I am improving
15:09:08
jeosol
so far brought the docker version from 14 secons to 3.7 seconds but these are changes on the docker side of things, not SBCL related. The bare metal repl is 0.8-1.5 seconds
15:13:53
jeosol
For task B, I read in CLOS instanced serialized to disk (it's the exact same each time), and then I changed some variables based on the call, then do call another function that takes that CLOS instance
15:18:34
jeosol
For the first call, it will read the object. The issue is that this test case is smaller compared to some other cases. The slots of the instance hold 3-dimensional arrays. My small test case is 100x1x1).
15:21:19
jeosol
I am also dealing with memory issues if I hold those arrays in memory which is what I do currently.
15:21:50
jeosol
Thanks everyone for chiming it, I appreciate the help. I have the cases running now and will share some of the benchmarks soon.
15:28:33
jeosol
for the performance and tuning hints to sbcl, I need to run (declaim (optimize (safety 1) (speed 3))) at toplevel?
15:31:55
jeosol
I just want to know so I save the first results, then run with higher speed and seel what performance boost I get.
15:32:26
aeth
speed changes the verbosity, though. Lots of notes about how you could make things faster, by changing what you're doing entirely, even if you... can't.
15:33:08
Bike
i think the messages i was talking about come up on speed 3 regardless of the other settings.
15:33:10
aeth
I have to do this in a few places: (locally (declare #+sbcl (sb-ext:muffle-conditions sb-ext:compiler-note)) ...)
15:33:47
Bike
and, like aeth alluded to, these will basically all be microoptimization hints. sbcl isn't smart enough to tell you a more efficient organization of your code
15:33:50
jeosol
Bike: the hints have suggests to improve code -- haha, hopefully it's not to change things radically
15:34:57
jeosol
Bike: that was my point. I have used suggestions from here over time, so organization is ok, I guess.
15:35:43
jeosol
As long SBCL duration is only marginally higher than the external exe, everything is fine
15:46:36
NotThatRPG
I've always wondered if there's some way to selectively mute those notes, so I can go through a big swodge of them, fix the ones I can fix, and muffle the ones I can't or don't want to
15:48:32
Bike
you can use muffle-conditions like aeth said, but i think getting more specific than "all notes" is not generally possible
15:54:42
Bike
my understanding is that you just put in the declare, and then sbcl won't note things about code within the lexical scope of the declaration
15:56:35
WBarends
The sicp book uses environment diagrams to explain the environment model and lexical, dynamic scope of variables and also state. The language in use in the book is scheme a lisp-1 language. Does ther exits a similar diagramatic way for common lisp a lisp-2 language?
15:58:04
jackdaniel
WBarends: it is basically the same, but with an extra namespace for functions, that is when the symbol is the first atom on the list that is evaluated
16:17:58
WBarends
jackdaniel: Basically you call it and that's propably right. But functions (defun) are only defined at the toplevel. So running along the environments from low level to toplevel searching for variables and procedures during an invocation of a procedure should be a more complicated than in scheme.
16:18:08
rendar
(+(+a b)b) is also valid, yes i know, its that i'm writing a little lisp interpreter in python for learning
16:20:31
beach
WBarends: Not really. And that's done only at compile time, and it is a minor part of the compilation process.
16:21:14
beach
WBarends: I mean, yes, it is slightly more complicated, because the compiler needs to search a particular namespace, rather than just the single one in Scheme.
16:21:19
WBarends
beach: Do you mean that LABELS and FLET add the procedures to the functions name space?
16:26:10
WBarends
beach: would you say that as a mental model i can use environment diagrams as used in sicp?
16:28:42
Bike
WBarends: the difference is pretty trivial. theoretically speaking, it's like if all functions were assigned names in a lisp-1 space that are distinct from all variable names. like for example if variable names had "p" on the front and function names "f"
16:34:01
WBarends
Bike: Oke, but sometimes you need funcall, apply and at othertimes it is the car of of list that invokes the procedure. So "v" and "f" a-lone is not sufficient.
16:38:13
aeth
but CL is a Lisp-3 because the third main namespace (of many, but nobody cares about tagbodies) is class/type
16:43:52
_death
Bike: if you consider only the difference of interpretation of the car, perhaps.. but then maybe there's also evaluation order.. what I'm saying is that SICP offers a view of a region of basic models, and the model of CL is more complicated in a variety of ways
16:49:01
Bike
i mean in context i'm pretty much only talking about variable bindings and environments, yeah
17:24:50
jeosol
update: I ran the analyses I was talking about 1000 serial calls: backend_server (Python) to sbcl/hunchentoot. The task duration is call on sbcl side. No difference with (speed 1) vs (speed 3) perhaps task duration is too small. Also, majority of the task is external exe bottle neck. Here is the full table (see explanation header on top)
17:24:51
jeosol
https://drive.google.com/file/d/1J6kusYDfkNDsdaJImoSK7GLCOubIV_3t/view and here is box plot summarizing results for (speed 1) https://drive.google.com/file/d/14VHbeIhcLYHQ90yA7NOEVft11JbNRj2R/view
17:35:39
jeosol
Thank you to everyone that chimed in regarding safety, speed, and performance optimizations with SBCL. I think I have done as much as possible on SBCL side. Here is a plot of ratio the durations. The one that pertains to SBCL (blue line) is steady at ~ 0.99: https://drive.google.com/file/d/1DMF8t8xLDLp0TfUCE6JJDrXPa4JY48dI/view
18:29:17
hexology
does (declaim (optimize ...)) apply to all compilation units that don't specifically have their own declaration or declamation attached? or only to the current compilation unit?
18:39:24
hexology
hmm.. so if i have several separate list files in my program (with one package per file), i might need/want to declaim my desired optimization settings in each one?
18:48:54
jeosol
Bike: really, then I don't think the optimization was applied in my case. I just ran the (declaim ...) part at the top level and ql loaded the required system
18:49:58
jeosol
However in my case with the default speed 1, I think there probably isn't much to be had
19:40:15
pjb
hexology: AFAIK, declaimations are global. Proclaim Establishes the declaration specified by declaration-specifier in the global environment.
20:06:24
NotThatRPG
Pretty sure DECLAIM is under-specified. It's certainly been a problem in figuring out how ASDF should deal with optimizations
20:13:33
pjb
NotThatRPG: basically: (defmacro declaim (&rest decls) `(eval-when (:compile-toplevel :load-toplevel :execute) ,@(mapcar (lambda (decl) `(proclaim ',decl)) decls)))
20:22:00
Bike
yes, but that doesn't actually help for the question of whether the effects last past the file or not
20:26:15
pjb
I consider that libraries should not contain optimization proclaimations. It should be the user who sets them depending on his needs. Therefore if they're limited to the file or the compilation-unit doesn't matter for me.
20:28:22
NotThatRPG
Right: as far as we could tell it's up to the implementation to decide the scope of DECLAIM
20:29:58
White_Flame
for one of my large projects, I have a (optimize-file) macro invocation at the top of each file, defined early in config, to punt learning how this might differ per implementation & environment :-P
20:42:19
White_Flame
yeah, on speed sensitive code, I have (fast-body . <body>) and (slow-body . <body>), which also interact with the global optimize-file config
20:43:48
aeth
in case anyone wants to do that, the code probably looks like this: (defmacro stfu-sbcl (&body body) `(locally (declare #+sbcl (sb-ext:muffle-conditions sb-ext:compiler-note)) ,@body))
20:45:29
White_Flame
actually I did a deftype optimization-note where I put the #+sbcl, for some reason
20:47:02
_death
for compiler notes, I learned to accept them.. they remind me that my data structures are probably suboptimal
20:53:05
_death
in some places, I do need to use potential bignums in code that needs to be performant, so muffling related notes could make sense
20:53:29
aeth
it'll be like (typecase x (double-float ...) (single-float ...) (integer ...) (t ...))
20:55:39
pjb
aeth: do you realize that (typecase x (double-float ...) (single-float ...) (integer ...) (t ...)) is not conforming? In some implementation, you may get a warning about duplicate clauses.
20:57:28
aeth
because there's no way to solve the general case of making sure that the second type in a typecase isn't entirely covered by the first, but ofc you sometimes can
20:57:40
aeth
(in the case where there's only one float type, in case someone doesn't get the issue)
20:57:54
aeth
(you can get this on almost all implementations if you do e.g. long-float and short-float)
20:58:12
pjb
aeth: strange, I solved the general case in com.informatimago.common-lisp.cesarum.utility:float-typecase.
20:58:46
aeth
I mean the general general case, not just for floats, which would require handling SATISFIES
21:03:42
_death
it's also being nice to macro writers, or when some type is changed and now overlaps another
21:14:07
Bike
duplicate typecase clauses would be a style warning at best, since the behavior of the typecase is totally defined, it's just that one clause will never happen
21:20:17
resttime
How much does SBCL use SIMD intrinsics when compiling? I'm wondering whether the compiler works fine or it's necessary to learn how to use sb-simd to squeeze out max performance for numerical operations
21:25:02
White_Flame
I don't believe it uses them by default, but there is a simd compatibility library and some asm/vop examples for explicitly using them
21:26:47
kakuhen
and I'd strongly recommend against setting safety to 0 for performance; there's often a good reason the compiler will introduce a type or bounds check at runtime, and its usually because you couldn't convince the compiler you'd consistently get X type or some array Y with exactly N elements
21:31:48
resttime
White_Flame: Hmmm, guess I'll look more into sb-simd, came across a paper: https://zenodo.org/record/6335627
21:34:53
resttime
kakuhen: Safety is good, I'm willing to sacrifice for performance since there's nothing critical. I'm trying to push the limits by writing with a raytracer and learn stuff along the way
21:38:59
kakuhen
on my machine, deep-copy-1 on a 4096 element array will cost me about 200,000 processor cycles whereas deep-copy-2 would be about 13,000; the number drops to about 5,000 if you remove bounds checks, but I'd rather pay the cost of bounds checks anyway.
21:41:00
kakuhen
Notice how with a sufficiently "good" type declaration I was able to emit much more concise code and not have to compromise on safety
21:41:29
kakuhen
this is why imo (declare (optimize (speed 3) (safety 0))) is one of the last things you should try doing for extracting performance out of a function
21:42:01
kakuhen
note: if you know the length of your simple-array ahead of time, the disassembly becomes even shorter (and probably faster)
21:51:42
resttime
Found the loopus library doesn't seem to be in there and I'm in the sb-simd package when trying to run the code (was trying to disassemble the deep copyfunctions), https://github.com/marcoheisig/Loopus/blob/main/code/packages.lisp
21:54:12
Shinmera
While I'm here and not asleep, here's an update on Kandria that may interest the particular set of individuals in this channel: https://twitter.com/Shinmera/status/1554585025789706241
21:54:50
resttime
kakuhen: haha well that's alright then I assume it's some kind of looping macro, might have been superseded by loopus
21:55:30
kakuhen
yes, the intent of do-vectorized was that you used types exported by sb-simd and the compiler would attempt using the best available simd instruction set on your computer (in my case, avx2)
22:00:17
resttime
Ohhh, in the sb-simd paper mentions a INSTRUCTION-SET-CASE macro for selecting best availble code at run time, might've been related/used to implement do-vectorized
22:01:00
Shinmera
dunno if it's smart enough to eliminate the dispatch by rewiring at startup, though.
22:06:51
resttime
Wait found a DO-VECTORIZED used here https://github.com/marcoheisig/Loopus/blob/bd84132eb5d0e94b1fadcb9be734a3ff8b0c1aff/code/ir/sb-simd.lisp#L46
22:07:04
kakuhen
resttime: well, assuming you can get loopus to compile, you may be able to replace sb-simd:do-vectorized with loopus:for
22:07:15
kakuhen
Unfortunately it doesn't seem to compile on my machine due to linker errors with libisl.so
22:09:21
kakuhen
resttime: i assume that's because the vectorizers get generated for various simd instruction sets on your cpu, then you're supposed to invoke loopus:for and let the backend handle the instruction set
22:09:26
Kingsy
curious, with clip, isnt it possible to use templates that inherit others? and if so how come the depp tutorial for radiance doesnt do it that way?
22:09:41
kakuhen
fwiw do-vectorizer doesn't explicitly exist in sb-simd codebase either, but it was provided to be when I was testing the example I sent you on SBCL 2.2.2
22:09:47
phantomics
From the sb-simd docs, I find it a bit unclear how the instructions map to the sb-vm primitive functions
22:10:41
Shinmera
As to why, idunno. It's been way too long since I wrote it. Probably thought it unnecessary to complicate matters for the two or what pages that it actually needs.
22:10:57
Kingsy
Shinmera: well with other frameworks, you get a base.twig or a base.whatever, which contains the <head> and all that other stuff, then you have a view.whatever which always uses the base.
22:11:44
Kingsy
no I mean, it makes sense if you wanted to include something in the head (which is common) you would only need to modify one file.
22:12:14
phantomics
https://sb-simd.common-lisp.dev/supported_sse.shtml lists the different add instructions that are supported, am I to assume that sb-vm::%sse-add/simple-array-single-float-1 implements ADDSS?
22:13:00
Shinmera
It's not clear to me whether it's better to have a master file that includes a subfile, or a subfile that includes a scaffold. You can do either, or none.
22:13:36
Kingsy
usually I like the subfile that includes the scaffold. btu yeah. nice! good to hear.
22:14:53
resttime
kakuhen: https://github.com/marcoheisig/sb-simd/commit/67ff8cb36962a02e36f1eeba71a50c0c6d073ced there we go, mystery solved
22:15:32
kakuhen
I was wondering why cl-isl was really insistent on picking libisl.so rather than libisl.dylib on my system
22:16:38
kakuhen
resttime: yes, if you look at the paper, heisig seems to use loopus:for much like do-vectorized was once used
22:16:55
kakuhen
so right now I'm on a journey to eventually get cl-isl compiling on my mac, to get loopus.sb-simd compiling, and test loopus:for
22:17:43
resttime
Issue that I see now is that loopus tries to use this missing symbol&package still so I it's prob outdated a bit https://github.com/marcoheisig/Loopus/blob/bd84132eb5d0e94b1fadcb9be734a3ff8b0c1aff/code/ir/sb-simd.lisp#L46
22:21:30
kakuhen
for instance, sb-simd-avx2:f64.4 will not exist on computers building sb-simd on intel processors without avx2 instructions
22:26:42
resttime
kakuhen: Want me to open a PR to add (:darwin "libisl.dynlib") to the prologue.lisp? That'd solve for the future for osx systems
22:28:50
kakuhen
I'm also having to make sure my copy of libisl is ABI compatible, since it looks like cl-isl needs version 22, but my system has version 23.
22:29:03
Kingsy
Shinmera: could I trouble you with a radiance question? I have asked in clschool, but as you are here I am wondering if I could be cheeky
22:31:16
kakuhen
looks like my SBCL is hanging in the middle of building cl-isl, so there may be ABI issues... I have no way of definitively telling, and I'm too lazy to install an outdated isl just to see if I can build this tbh
22:32:28
kakuhen
running in terminal now to see if I get dropped to LDB; I suspect that may be happening when SBCL hangs.
22:34:12
kakuhen
"READ error during COMPILE-FILE: Package SB-SIMD-VECTORIZER does not exist. Line: 3, Column: 71, File-Position: 96." in code/ir/sb-simd.lisp
22:37:28
kakuhen
With that said, I also am getting at the very bottom of the debugger output: "Bogus form-number: the source file has probably changed too much to cope with"
22:41:05
resttime
Went ahead and created PR, should solve minor annoyance with picking the write lib on OSX, I've written crossplatform CFFi bindings before and just specify the lib name explicitly depending on the platform
22:46:35
Kingsy
more generic question then if anyone might know, whatis the best way of recompiling an .asd lisp module and I am working on?
22:49:16
kakuhen
I'm guessing it's because :UNIX is in *FEATURES* as well, but I'm not sure since I've never used CFFI.
22:50:32
kakuhen
in any case, I proceeded building the package despite the warnings in loopus.sb-simd, and codegen became absolutely borked, so I think the presence of sb-simd-vectorizer is important :<
22:52:28
resttime
kakuhen: Yeah, the way DEFINE-FOREIGN-LIBRARY works is that it'll check the load clauses in order, so the one for :DARWIN will have to be before the :UNIX one
22:56:28
resttime
Kingsy: Dunno best way, but I keep a symlink to the project repo in quicklisp/local-projects and quickload the system when needed
22:57:42
Kingsy
resttime: I think I did it within emacs using SPC-m-c -> f, no restart of radiance needed. after that I just hit the radiance server and the new endpoint was there.. pertty sweet.
22:58:38
Kingsy
resttime: ah that makes sense. my project is actually in there right now, so just a quickload would have worked too I suppsoe, thats probably what emacs is doing under the hood.
23:00:30
resttime
np, and it kinda depends. My context is if I'm working on a project that depends-on another, I'd make a local copy of the other repo, make changes, add symlink to the quicklisp/local-projects
23:01:10
resttime
Then every subsequent quickload will automatically prioritise the one in local-projects instead of the one in the quicklisp repo
23:04:22
resttime
If it's just my own project then I'd just take advantage of hotloading with the SLY/SLIME keybindings
23:09:39
Kingsy
hehe thats kinda over my head. but I am sure i will figure it out at some point. I have a depends on and the sly keybinding worked. but we will see.