freenode/#lisp - IRC Chatlog
Search
15:06:15
pfdietz
There's infrastructure in the CL standard for solving this problem, since it has to be solved for various builtin macros on places.
15:10:23
pfdietz
See the example I gave: the place expression can have side effecting subexpressions.
15:13:26
pfdietz
One also has to make sure the subexpressions of the place continue to be executed in left-to-right order. ansi-tests checked for all this.
15:15:37
shrdlu68
pfdietz: Hmm, shiftf suffers from the same shortcoming. Is there a way of avoiding that?
15:18:11
pfdietz
You can use GET-SETF-EXPANSION to explode places into pieces that can be assembled in the macro expansion. http://www.lispworks.com/documentation/lw70/CLHS/Body/f_get_se.htm
15:22:08
pfdietz
(LET* ((#:Y1080 Y) (#:G1081 (INCF I)) (#:OUT1082 X) (#:NEW1 (AREF #:Y1080 #:G1081)) (#:NEW1 Z))
15:48:15
beach
I believe I have a first version of MACROEXPAND-ALL working. It takes a cleavir environment that it operates in.
15:52:11
makomo
pfdietz: do the subforms of a place *have* to be evaluated left-to-right? i thought that was just a nice property/convention, but that it wasn't required for user-defined places
15:54:37
makomo
pfdietz: for example, i wrote an IFF place, which conditionally writes/reads one of the two places you provide to it as arguments, along with a condition form
15:56:22
Bike
' The evaluation ordering of subforms within a place is determined by the order specified by the second value returned by get-setf-expansion. For all places defined by this specification (e.g., getf, ldb, ...), this order of evaluation is left-to-right. '
15:57:43
makomo
i was going to write condf and other conditional places by basing them on iff, but i have to make iff take an optional second place first
16:26:50
slyrus1
right. I was going to say "presumably the AST stuff makes it possible to write such a thing nicely"?
16:27:28
beach
Sure. I did it the easy way. I basically translated the code from generate-ast (by hand) to cover all the cases.
17:00:40
Demosthenex
i'm just trying to confirm some edge cases in what i'm working on, and honestly i could just repeat things in the repl, but thought i should document it a bit better
17:04:37
beach
makomo: When possible, I write "random tests", i.e. I generate huge numbers of test cases to my code. when applicable, I write two version of my code, one "production" version, and one "trivial" version.
17:04:58
beach
makomo: The are supposed to behave the same way, but the "trivial" one is too slow for the final version.
17:06:00
makomo
beach: oh i see. how do you generate the random tests without basically solving the problem though? how do you bootstrap, i guess? :-)
17:06:34
Colleen
Clhs: function funcall http://www.lispworks.com/documentation/HyperSpec/Body/f_funcal.htm
17:07:59
beach
makomo: Then I check the tests against the code cover. If there are places that have not been executed, I try to modify my random-test generator so that those are included.
17:12:57
makomo
beach: hmm, i see how one could easily generate sequences of those operations, but how do you generate the results to test against, without already implementing the API? maybe i'm thinking about it the wrong way or something
17:13:34
makomo
or perhaps you rely on an already existing implementation of the same API (or a similar one)
17:15:12
beach
There could be bugs in the trivial one as well of course, but the probability that it would be "the same" bug in both versions is infinitesimal.
17:15:15
makomo
so you're relying on the fact that the trivial version is as trivial as can be, and that you will probably easily catch the errors since the code is simple
17:16:44
beach
But yeah, the "trivial" version is usually so simple that you can just look at it and be convinced that it is correct.
17:18:29
beach
The "trivial" version stays the same unless the API changes, which should not be the case (at least not very often).
17:33:44
pfdietz
The first is generative: have a grammar (explicit or implicit) and attach probabilities to different production rules.
17:34:22
pfdietz
The second is mutational: given a corpus of interesting inputs, generate new inputs by changing or combining them in various ways.
17:34:52
pfdietz
One can also add constraints to the inputs to try to bias them toward more interesting executions.
17:35:33
pfdietz
And if one can instrument the code under test then the inputs can be tweaked based on whether in the code the executions go ("gray box fuzzing").
17:35:39
beach
pfdietz: I often need my generated operations to follow a Markov process. Unless I have a certain probability of generating long sequences of "the same" operation, my coverage won't be adequate.
17:37:17
pfdietz
Yes. There are games you can play with that, like "swarm testing", where you randomly prune down the set of production rules before each run of the test generator. Empirically this tends to find more bugs. I used this technique in the CL random test generator.
17:39:08
pfdietz
The experience with random testing is, I think, that different test generators cover different parts of bug space, so for a really large system you want diversity.
17:42:53
makomo
pfdietz: interesting, but you do arrive at the same "problem" of having to already have a working version of what you want to test, right? i.e. once you generate the inputs (sequence of operations, etc.) you need to somehow generate the outputs (the actual solution to those inputs)
17:43:30
pfdietz
You identify properties the code should have, even if they may not fully specify what the code should do.
17:43:59
beach
makomo: I can assure you that the "working version" is trivial to write in many cases.
17:44:24
pfdietz
For the Lisp testing I was doing, the property was that (for conforming CL) the code should do the same thing if compiled w. various different declarations, or if eval-ed.
17:46:38
makomo
beach: mhm, but it's interesting how there's a fundamental circularity/bootstrapping problem in there. either you have an existing implementation or you use your brain (which is just another implementation) and write out a finite number of tests yourself
17:47:30
pfdietz
If your code is loaded with assertions, each of them is an opportunity for testing. No assertions should fail.
17:49:01
makomo
pfdietz: true. how did you test that the two versions of code did the same thing? i suppose the stuff that was randomly generated were the declarations, not the code itself? the code was predetermined and you knew what to test for?
17:49:56
makomo
beach: yeah, i understand that. i get that it's a non-problem in practice, but fundamentally the ""problem"" is there
17:50:06
pfdietz
I randomly generated code and randomly generated inputs, and then checked that no errors were thrown (this was conforming code that would not throw errors) and that it had the same result on the same inputs.
17:50:07
beach
It takes less time to write the implementation of the trivial version than it takes to write the tests.
17:51:36
beach
makomo: I.e., that I think it is extremely hard to find abstractions for a testing framework.
17:51:49
makomo
beach: not quite. i just wanted to comment on the need of having to have an existing implementation in order to begin testing, no matter how trivial it is
17:59:11
beach
makomo: My main point here is that I don't believe in "testing frameworks" because I don't see how such a thing could capture even the most useful testing techniques that I know of.
18:18:58
shka_
but i like how i can use asdf extension to specify :test-file and i can simply use is and ok assertion
18:19:10
j`ey
this is my testing framework: (defun test (val) (cond ((eq val nil) (princ "E")) (t (princ "."))))
18:19:52
pfdietz
The question you want to ask is not "which testing framework?" but rather "what specifically do you want from a testing framework?"
18:27:33
beach
I am totally convince that this is one abstraction that everyone is looking for but that isn't possible in general.
18:28:49
beach
But the fact that there are so many and that people are not satisfied with what exists, tells me that there is a problem.
18:31:30
pfdietz
There are ten copies of RT in Quicklisp, last I checked. Not only are they easy to write, they're easy to copy. :)
18:39:42
jackdaniel
testing is not an easy task so many people have different ideas how the tool simplifying that should look like
18:43:29
jasom
beach: there are so many testing frameworks for the same reason I accidentally wrote a routing library for clack
18:44:06
jasom
beach: at first I didn't really need a routing library, so I didn't use one. Then I added a couple of features as I needed them. Then I realized that if I just moved it to a separate package, I had a complete routing library...
18:45:29
jasom
just like a testing framework it's easy to grow organically as you need more features *and* not much code when it's done: https://github.com/jasom/cl-fccs/blob/master/src/route.lisp
18:47:37
pjb
jackdaniel: right. IME, testing depends on the kind of software you are testing. Basically, most testing libraries are just unsuitable for a lot of software I had to test.
18:50:35
pjb
perhaps it would be more worthwhile to write down specifications, eg. of testing framework interfaces to testing report systems. eg. https://testanything.org
18:52:07
pjb
If asdf:test-op was specified to produce on *standard-output* a TAP stream, quicklisp could parse the results and produce nice reports…
18:53:14
pjb
You could use any testing framework, even multiple frameworks in a big system, as long as they would all follow a unique protocol, collected and interpreted by the surrounding tools and UI, it would be nice, integrated, and useful.
19:13:49
vsync
before I wanted to use it just because it was out of fashion and I'm a weirdo even among the weirdos
19:14:18
vsync
never had an actual use though... but lately I have found one where you want to have an inner named closure
19:16:39
vsync
though mine is for healthcare provider coordination data exchange between health systems and EMRs
19:17:07
vsync
for that purpose but generalizable! just like embedded in many other systems I'm sure, heh
19:18:33
pjb
vsync: the real use case for &aux, is when you write a macro and you need to add variables around a &body docstring-declarations-and-body <- NOT merely BODY!!!
19:19:14
pjb
I don't mind indentation. It's done automatically, and I've got a 5000+ pixel wide screen :-)
19:22:53
vsync
yeah I stumbled across it the other day when I didn't have CLOC installed and until my clean kernel compile happens the system crashes if I access a reiserfs and then a jfs
19:25:09
vsync
need to leave 3 extra characters for the minimum log graph view with diff, plus 1 for some terms wrap on the last column uselessly somewhy
19:25:37
stylewarning
vsync: sometimes a character in the last column is reserved for an overflow character
19:26:03
vsync
stylewarning: makes sense for consistency... like do you indent context lines in diff 1 char? I say yes
19:30:50
pjb
Well the max line length is not good. You'd have to look at the typical line length. My files contain outliers in comments or in data. Sometimes in code, but rarely.
20:12:48
makomo
ugh, i don't like it when i have to use composite keys for hash tables, it feels kinda clunky
21:15:32
aeth
Does anyone run CL on a processor with many cores? AMD has some 16 core / 32 thread and even (for servers) 32 core / 64 thread.
21:18:58
no-defun-allowed
I usually use lparallel and bordeaux-threads for stuff so I'm not paying much attention, sorry.
21:19:58
jasom
aeth: threads predate parallelism by a lot on PCs. In the 90s a lot of code was written multithreaded to avoid io blocking.
21:20:58
no-defun-allowed
cl-decentralise spawns a thread per connection since they actually do things in SBCL instead of splitting up one interpreter's time (cough CPython cough).
21:23:12
no-defun-allowed
cl-vep uses one generator and a lock for the generator. When a worker needs a new frame it holds the lock and funcalls the generator. This method (work "source" and lock) hasn't given me issues in the past.
21:23:52
no-defun-allowed
I've done high thread counts to avoid http blocking too. That method scaled pretty well up to 64 threads.
21:24:29
no-defun-allowed
However, the ffmpeg interface for cl-vep is shit slow and only puts out 4fps and only makes a load of 1.5.
21:24:41
jasom
aeth: I just spawn more worker threads. I haven't had any issue at 16 worker threads on my 8/16 ryzen
21:25:07
aeth
jasom: anyway, what I mean is things that are architectured so that they'd benefit from e.g. if AMD released a 64 core CPU tomorrow.
21:28:32
makomo
why is the syntax grammar for DEFSTRUCT so weird? i don't get what the keywords/singleton lists are supposed to do http://www.lispworks.com/documentation/lw70/CLHS/Body/m_defstr.htm
21:29:40
makomo
why would you want just :conc-name? is that supposed to construct (within the context of the grammar) a non-list version of (:conc-name conc-name)?
21:30:27
Bike
"A defstruct option can be either a keyword or a list of a keyword and arguments for that keyword; specifying the keyword by itself is equivalent to specifying a list consisting of the keyword and no arguments."
21:31:23
makomo
Bike: guess i should have read further, but it confused me right away so i didn't bother :c. thanks
21:33:42
jasom
aeth: lparallel has bounded queues implemented with vectors; I haven't checked the code, but it ought-not be consing if implemented right.
21:34:28
aeth
jasom: I'm guessing you'd queue or pass messages with things that are unboxed (like fixnums) or with keywords/symbols
21:35:27
jasom
aeth: setf on an array is going to be non-consing. Creating the value to setf in that locatin *might* be consing
21:36:28
jasom
aeth: I'm pretty sure an unspecialized array will not cons if you do something like (setf (aref foo) 3) since 3 is a fixnum
21:36:28
aeth
Would you sync with timers or would you have a thread sleep when done until it receives a start-working-again call?
21:44:49
aeth
Oh, there *is* a Threadripper 32 core (64 thread) CPU, the 2990WX. Wikipedia just put it in a separate chart much further down because it's Zen+.
21:46:55
aeth
So there is a chance that a (rich) Lisper is already running an application with performance on architecture close to manycore in mind (if you count the 64 threads via SMT)
21:47:37
jasom
aeth: Talos is available with 128T (or maybe even more) configurations since POWER8 is 4T per core
21:49:52
jasom
2x 22-core POWER9 is the highest configuration which translates to 196 threads I think?
21:49:54
aeth
Is PPC compatible with POWER8? I'm not sure if any CL runs on POWER8 if not. SBCL seems to have PPC and PPC64.
21:50:52
jasom
POWER9 is PPC64, but I think the talos is designed to run in little-endian mode, and I don't know if SBCL does that.
21:51:31
aeth
PPC64 is marked as in progress on the SBCL download page. I wouldn't be surprised if threading doesn't work
21:52:57
aeth
I wonder if there's a 2 socket AMD Threadripper mobo. That would be 2x 2x 32 at a max (right now... expect it to double next gen), so 128.
21:52:59
jasom
and the IBM E980 can support up to 16 12 core SMT8 processors for more cores than I care to do the math on
21:55:00
aeth
jasom: But yes, IBM's POWER is definitely worth mentioning in this kind of discussion. Last I checked, IBM seems to be keeping up with Intel, which is 10ish cores behind AMD.
21:56:39
aeth
A quick search has a reddit thread where the answer is "Threadripper is single socket only, for dual you'll need to go with Epyc." https://www.reddit.com/r/Amd/comments/6tqg88/dual_socket_threadripper_max_ram/dlmt8fn/
21:56:50
aeth
Of course, random reddit comments are even less reliable than random StackOverflow answers
21:57:48
aeth
(That's probably one of the ways they get people to pay 2x to 4x as much as the consumer line.)
22:17:08
Demosthenex
is there a way to start sbcl with swank using a port number from the command line? i'm trying to work on a few different things on a remote host, and i can connect with ssh/port forwarding. just starting sbcl with swank on separate ports is awkward
22:48:29
aeth
Wikipedia says POWER9 goes up to 24-core (96 threads via SMT4?) https://en.wikipedia.org/wiki/POWER9
22:53:12
aeth
It looks like both AMD and IBM are going to release 48-core CPUs very soon, though. It looks like Qualcomm already has a 48-core ARM server.
23:00:00
Bike
programs relying on particular details of the machine they're running on is something to be avoided. surely this is a basic principle
23:03:27
aeth
Bike: The architecture can also run on an 4 core machine, yes, but there's less of a motivation for it there.
23:05:29
aeth
Bike: But consider e.g. a game or similar application that can be divided into several systems. In today's world, if it's parallel at all, each major system would probably get its own thread. That'll only scale up to the number of major systems, which is probably far less than 90.
23:07:22
Bike
in that case whether you have five or ninety processes isn't important for basic architecture.
23:08:18
aeth
Except for the most part, afaik what you'd typically see when the end user has quad core is maybe putting one task on each a thread to be "parallel", not also subdividing those tasks.
23:10:40
Bike
if you already have an idea for how to structure a program for multiprocessing, what are you even asking?
23:13:32
aeth
Bike: I'm wondering if anyone has successfully deployed a CL application that takes advantage of one of these CPUs with a "very high" core count. ("very high" being relative to the normal 2-4 cores that you would have seen for this kind of hardware until recently)
23:14:23
Bike
the machine i use to build clasp has thirty something cores. during build we have each core do a compile-file so we can compile a bunch of files at once.
23:15:04
whartung
I mainge someone one has. At those core counts, you’re getting into “super computing” kinds of designs and applications.
23:15:53
whartung
Funny Bike, back in the day, when I had the CPU meter on my desktop, I’d just kick off builds until it reach 90%. … by hand. Whenever it dipped down, I’d kick off another one.
23:18:37
whartung
My friend laments how horrible Swift builds are on the Mac — it’s apparently very CPU intensive.
23:22:37
aeth
Bike: I would personally probably just manually implement the required algorithms and data structures when needed in my case rather than using a library. Micromanaging this sort of thing is kind of a key point of the engine (when I get there), not a secondary feature.
23:24:21
aeth
It's mainly due to the requirements for 100 FPS logic and no consing at the library level in the game loop (I have consed in all of my substantial tests, of course... too hard not to... but that's in the user's code)
23:27:44
aeth
no-defun-allowed: A 3D game engine. I switched my original design (nothing is stopping it from being general purpose, of course) from first person to third person strategy (i.e. far away camera) because of fewer physics requirements (otherwise the project would become more about writing a physics engine than a game engine)
23:31:06
aeth
I just need to do mouse selection of an entity and a destination and then I have arbitrary movement on a 2D plane within the 3D world (e.g. ships on water). After that (soundless) games would technically be possible, although still fairly difficult because the text rendering isn't complete yet.
23:33:59
aeth
I'm guessing the input and the final update of the authoritative game state would have to stay on a core thread.
23:36:38
whartung
as I understand it, it’s prety difficult to multitask the (a?) rendering pipeline.
23:37:14
whartung
you can easily have a rendering thread, a state thread, sevearl AI threads. But the busy thread is typically the rendering thread.
23:37:26
Bike
you don't really have the opportunity anyway since the graphics library wants to do rendering in mysterious ways
23:37:48
jasom
Bike: there are some tradeoffs you can make; e.g. make the serial performance much worse in trade for more parallelism; the number of cores you have can decide if you will make that tradeoff
23:38:10
Bike
mostly because rendering is so hard we've farmed it off to repurposed bitcoin-mining chips
23:38:39
whartung
well, they happen to be pretty good at the job. If they weren’t we wouldn’t be handing them off to it.
23:40:26
whartung
as I understand it, the games job is to get the models oriented properly, send them to the GPU, and make sure the proper textures are loaded at the same time. The GPU does the actual rendering stuff.
23:40:45
jasom
aeth: my most recent webapp will probably scale well past 8 cores; all the state is in the database and each worker thread has its own DB connection. It's when you get past 32 cores that things can get sticky.
23:44:37
whartung
but the gpu guys have a way of getting it to work on multiple GPUs (2 GPUs, alternatiing scan lines or something like that)
23:46:35
aeth
whartung: It's generally not worth it to have multiple GPUs. The game (or application) has to specifically support it (and probably support it differently for nvidia and AMD) and it's usually better just to get the next tier up in graphics cards so it probably only applies to you if you're using Titans.
23:48:32
aeth
For nvidia it's called SLI and the general advice on forums has been to not do it, at least for the past few years.
23:49:25
White_Flame
whartung: different generations had different limitations. I know there were 3x linked card setups,a nd probably 4 as well
23:49:56
White_Flame
I think some also just send along the PCIe bus instead of having their own inter-card connectors
23:50:06
aeth
See https://en.wikipedia.org/wiki/Scalable_Link_Interface and https://en.wikipedia.org/wiki/AMD_CrossFireX
23:50:26
aeth
but apparently nvidia just abandoned SLI on the next generation (that isn't out yet) in favor of "NVLink Bridge"
23:52:52
whartung
I’ve spent enough treasure on computer equipment, I’m glad I was never really in the GPU chase :)
23:53:15
White_Flame
nvidia is going off the deep end with prices anyway, best to ignore it for now, too
23:53:28
aeth
But when I was saying that rendering is heavy, I mean heavy on the CPU. The GPU is doing most of the work, but #'draw still seems to be the most CPU intensive function at the moment for me. I think Vulkan would allow this to be split up, though.
23:53:59
White_Flame
no-defun-allowed: most lisps are fully reflective, if you use their implementation internals
23:54:21
aeth
White_Flame: You see that all over the place. CPUs, GPUs, smartphones, etc. The high end is improving, but also getting more expensive.
23:54:53
White_Flame
aeth: RAM prices are settling down, though, finally returning to where they were 3 years ago or wherever it was before that spike
23:54:59
whartung
depends on how high powerful the polisher you’re using no-defun-allowed, and what kind of compound you’re using
23:56:46
whartung
no, I understand aeth. Just getting the frames set up. A lot of that is the modeling side (modeling the car(s) for example), and things like the physics engine. Is it worth while to farm our modeling the suspension dymanics of 40 cars in a nascar sim to different CPUs per frame? I dunno. it may not even be possible if you’re modeling drafting and what not.
23:58:42
aeth
whartung: but on the other hand I think a large part of strategy games is embarassingly parallel, or close enough.
23:59:09
whartung
yea it can be, quesiton is whether it’s cpu worthy enough to bother breaking it out.
23:59:56
aeth
I've definitely played some building strategy games (games with workers doing tasks and carrying resources around a village) where it uses all the CPU
0:00:05
whartung
The only time I upgraded a machine for a better game performance (among other things) was way back when I went from 133 to 400 for Total Annihilation. So it coul dhandle more units. But even that would be difficult to paralleize because of the dependencies on each other (notably in traffic jams)
0:00:56
aeth
Even without htop up, I can tell when my desktop's fully using the CPU in a multithreaded way because I can hear the CPU fan when that happens.
0:01:44
whartung
my friends laptop turned in to an air hockey table when compiling Swift. You’d think it would float off the desk...
0:02:28
aeth
In general, simulation(ish) strategy games seem to max out the CPU and recent AAA games tend to max out the GPU
0:02:55
aeth
(I don't think both would be possible, unless you wrote specifically to the hardware combination and optimized heavily.)
0:05:17
jasom
aeth: do you sometimes render inconsistent state when you do that (i.e. some objects 10ms behind others)?
0:06:13
aeth
Oh and 100 fps for logic because its inverse is 10 ms and what you want is (integerp (* 1000 (/ n))) so both measurements are integers imo.
0:07:28
whartung
unrelated, are they any good examples of network servers in CL floating around on the net? I’ve seen the http server (Hunchenroot? Is that right?) Just curious if there are others of that ilk I could look at.
0:08:08
aeth
In case anyone's wondering about the other numbers... (loop for i from 1 to 2000 when (integerp (* 1000 (/ i))) collect i) => (1 2 4 5 8 10 20 25 40 50 100 125 200 250 500 1000)
0:10:24
aeth
Looks like it's 1 and numbers whose only prime factors are 5 and 2. (I ran my factoring function on that list to verify)
0:14:05
aeth
(limit in the sense that you can't get better than 1000, not in the calculus sense of the word "limit")
0:20:05
aeth
Bike: I didn't expect the fundamental theorem of arithmetic but I guess I should have expected it as soon as INTEGERP entered
0:20:56
aeth
Here I was working with periods and frequencies in a real world problem and then suddenly PRIMES
0:24:14
Bike
people are all like one plus one is two, well in my opinion it makes more sense to define two as the octave
0:27:07
aeth
Bike: But this is sort of the same problem. With the logic running at 100 ticks a second, some parts might actually only need to run at 50 or 25 or even 1. And this can be determined with a simple MOD
2:04:10
stacksmith
Is there a foreign pointer validation function in CFFI? To check if dereferencing would crash...
2:07:43
stacksmith
I was hoping for a less-than-perfect validation - like check if it's in-range for any areas allocated with CFFI. But yeah, that was wishful thinking.