freenode/#lisp - IRC Chatlog
Search
17:59:32
dbotton
Does anyone know what the built in packages "gray" and "walker" are in ecl? The manual is blank for them?
18:05:48
Alfr
dbotton, Gray Streams is a rejected proposal for the standard, a number of implementations have it tough. http://www.nhplace.com/kent/CL/Issues/stream-definition-by-user.html
18:06:52
White_Flame
usually the walker goes through a source code form, distinguishing what's code vs data, what's a lexical variable name, etc
18:17:40
phoe
http://portability.cl/ lists ABCL, ACL, CCL, Clasp, CLISP, ECL, LW, Mezzano, MOCL, and SBCL under t-g-s
20:08:14
pillton
I have released a new system called data-flow (https://github.com/markcox80/data-flow). It is intended for applications which are composed of small independent components/units which interact with each other via ports or events.
20:12:37
White_Flame
pillton: interesting. We have a pipeline scheme, which is pretty verbose. This might do something similar
20:13:28
White_Flame
and of course, every library has its own threadpooling system. It'd be nice for there to be a central threadpool for everything to compose into
20:16:32
pillton
The system provides a scheduler which executes tasks if their resource requirements are satisfied. This should be fine to use with other systems.
20:18:26
White_Flame
I mean, the entire point of a threadpool is to better gracefully handle the case when there's too much in flight
20:20:11
pillton
You could use the sequential scheduler for the application and use the threadpool of the other systems for the individual computations.
20:21:11
White_Flame
well, that means manual allocation of resources between your libraries & other threadpool domains
20:22:09
White_Flame
(similarly, if all p2p bandwidth controls were centralized into a single system-wide config, as opposed to per application; that's a similar thing that's found in various domains)
20:45:09
_death
pillton: looks interesting.. usually I'd use zmq for that kind of thing, or mailboxes or just queues.. or coroutines and such if single-threaded.. data-flow seems framework-y and lets the user specify a scheduler so I'm wondering what are the use cases..
20:45:20
phantomics
Interesting, I've got some projects coming up that will require dataflow, I'll give this a try
21:01:15
pillton
_death: The factors which "tipped me over the edge" was a project at work and Paul Khuong's article titled "The unscalable, deadlock-prone, thread pool".
21:05:16
White_Flame
yeah, re the article, anything that can block should never be in a fixed-size threadpool
21:06:05
pillton
_death: I hate writing multithreaded software for the most part. What I wanted was something which executed loosely connected computations in parallel without me having to do anything i.e. locally sequential, globally parallel.
21:06:26
aeth
White_Flame: Speaking as a CL game engine author, you're simply not going to get a thread pool that satisfies every use case. You think that having one central thread pool means that libraries can interoperate with each other so more can use thread pools safely, but not really. I'll still have to avoid everything that uses a thread pool.
21:09:12
aeth
The thing is, a game engine is going to have to have its own threading system that handles things to its own requirements which are soft real time rather than batch-oriented. At best, you could just provide a common API and allow different implementations with different trade offs, but even the API is probably not going to really be the same with game engine vs. normal app stuff
21:10:07
aeth
Worse, some things need to synchronize every frame (probably physics, if "frame" means logic frame rather than rendering frame) and some things don't (e.g. AI).
21:10:12
pillton
_death: The project at work involved iterating a map-and-reduce task where each item in the map required performing I/O and then a hefty compute task. A lot of time was wasted waiting for I/O to finish because it wasn't executing in parallel with the compute.
21:10:22
_death
pillton: ok.. in the readme you mentioned non-blocking I/O.. how would you go about doing that with data-flow?
21:11:14
White_Flame
pillton: yeah, one the of the reason to have a grand central threadpool is that the priority, balancing, resource consumption, can ideally be well expressed in one place
21:11:28
pillton
_death: You can ask a scheduler if blocking is allowed using data-flow:blocking-allowed-p.
21:12:11
pillton
_death: All of the operations in the data flow system are non blocking. That predicate is meant for I/O.
21:12:15
aeth
(as an aside: confusingly, both game frameworks and game engines are software frameworks)
21:13:33
aeth
White_Flame: right... in particular, it creates the inversion of control where the flow of control is the responsibility of the framework, rather than the caller, at least by Wikipedia's definition. https://en.wikipedia.org/wiki/Software_framework
21:14:10
White_Flame
if you need external scheduling, because you don't know at the call site if/when/where something should run ... it still seems like a library to me
21:14:20
aeth
A language that does this directly (or that has some de facto standard library that everyone uses to do this, which might be what you're proposing) is one that gives less freedom to the software frameworks that should be micromanaging this sort of thing
21:14:49
aeth
Unless, of course, you go one level more meta and just provide an interface that anyone can implement (which is sort of a C++ or Java way of thinking)
21:16:23
pillton
White_Flame: In my mind there are two types of multi-threaded software 1) same task different data (usually requires barriers for synchronisation) and 2) multiple arbitary tasks. The barrier algorithms I have seen require knowing the number of threads before starting the computation.
21:17:40
aeth
A game engine is kind of the worst case for this type of multithreading (which might be why so many games are still bottlenecked by a main thread even though we've had mainstream quad cores for over a decade now).
21:19:41
aeth
You have to parallelize stuff that's of pretty much every kind of data (even network data, if multiplayer) on the GPU and on the CPU. Some's async (doesn't matter when it finishes, e.g. loading something in the background) and some's with very tight timing requirements.
21:20:23
White_Flame
so it's really "realtime" as you mentioned that's the confounding factor for many throughput systems
21:20:46
aeth
HPC is mostly just linear algebra simulations, which is a subset of what games do, but games just require things to visually look plausible rather than to be actually accurate (with a bit of a caveat that with multiplayer you can't do things too badly or you might desync)
21:22:03
aeth
You can do 100% utilization on a game, if you wanted the simulation to get more realistic the more power you have, but generally games only care about 100% GPU utilization, i.e. just the visual simulation. Multiplayer games really can't let players with better CPUs get a different logic simulation, anyway.
21:23:11
aeth
There are some games where you'll get a harder game if you have a better CPU, though. Most notably, timed chess. (The AI, called the "chess engine", will probably use 100% of everything until time's up, or some other constraint is up.)
21:23:34
White_Flame
I guess the same could be true on the CPU end, if everything's on a frame barrier
21:23:46
aeth
(Not like a slightly harder chess AI in 2020 would make an observable difference to a human...)
21:24:07
White_Flame
but really, those who are doing threadpool stuff are generally not doing hard realtime requirements
21:24:41
White_Flame
while trying to express things reasonably without requiring that level of hand-tuning
21:24:57
aeth
Yes, but... if you get everyone to standardize on a thread pool that isn't designed for hard real time requirements, you're basically forcing users with those requirements to not be able to use any library that uses that thread pool.
21:25:15
aeth
Maybe fine for a scripting language, but I don't think CL should restrict domains like that.
21:26:14
White_Flame
having multiple very customized ones in some situations might, but again I would take "I'm using threadpools" as equivalent to "I'm not doing hard realtime"
21:27:13
aeth
afaik, and I haven't looked into this in detail yet, you can just treat a game engine's thread situation like: "these n threads can return within a reasonable amount of time while these m threads have to synchronize every k frames"
21:27:57
aeth
i.e. the former are basically async but you need an answer of some sort "eventually" (which could be a few seconds, which is like an eternity to the game engine)
21:28:15
aeth
so even if a thread pool doesn't work for real time, well, you can still sort of thread pool the rest
21:29:09
aeth
I guess some of those things should be time limited (e.g. AI) and some of those things shouldn't (e.g. loading the next chunk of the world map in a Minecraft clone)
21:30:46
aeth
(and a game engine isn't hard real time, anyway... missing a frame or two isn't the end of the world)
21:32:10
White_Flame
right, but there's a framerate target, which means budgeting milliseconds for various tasks, as well as polygon/texture budgets, etc. It's still designed as realtime even if it doesn't kill 300 people if it fails its targets
21:42:49
aeth
White_Flame: that just means that the threads must synchronize by a certain amount of time. But if you have, say, 6 physics threads, then the physics work can still be distributed.
21:43:36
White_Flame
measured against the load that the game needs, and affects level design, weapon firing rates, etc
21:44:15
White_Flame
but still, this falls more into the SIMD-ish style processing that pillton brought up
21:53:48
aeth
White_Flame: AMD has been selling 12c24t and 16c32t CPUs to the consumers/mainstream/gamers segment for well over a year now. And anything started now will probably take years to ship.
21:55:02
aeth
Some systems will be "bucket of tasks" systems, such as perhaps the AI system. And when you have 32 real hardware threads to work with, they might be allocated a decent number
21:55:41
White_Flame
or I should better say, the game industry is heavily affected by that, while the rest isn't as much
21:56:33
aeth
The thing about games is that they can do basically everything. And when it does "basically everything", the timing requirements differ, too.
21:57:03
aeth
To the point where some of them are probably best expressed as not really having a requirement. As long as the rest of the world can move on so the game doesn't hang.
21:58:30
aeth
console-first AAA games will probably spend all of the thread budget on rendering threads, but basically just those will, and none of them will be using Common Lisp
22:52:33
pillton_
White_Flame: My primary goal was to enforce "component providers" to not require a particular scheduler in order to function. This should be chosen by the user.
23:19:55
aeth
In case someone's wondering why I brought up game engines... I think any "serious" language should consider a game engine, a web browser, and a database. Big, complicated applications that need some degree of performance and an unbounded number of features limited only by ambition and time.
23:20:18
aeth
Fairly non-overlapping, too, so just those three should span most application development concerns.
23:21:28
aeth
Maybe add something that heavily involves integers, but maybe that's already covered by the cryptography that's needed by networking...
0:13:38
matta
Can anyone tell me what the #+dev in https://github.com/vseloved/cl-agraph/blob/master/cl-agraph.asd is doing? I know that this conditionally includes the subsequent form, but how does this "dev" thing get set, and do conditional :depends-on tricks like this play nice with quicklisp?
0:24:13
pillton
matta: It would be better to define a cl-agraph/test system rather than use the conditional reader.
0:24:32
lotuseater
if the keyword is in *features*it will put the following atom/sexp. otherwise it will comment it out
0:28:23
pillton
matta: Have a look at secion 7.1.1 Predefined operations of ASDF in the ASDF manual https://common-lisp.net/project/asdf/asdf.html
0:34:38
matta
pillton: thanks, I had seen separate foo/test systems defined elsewhere. This #+dev approach was different, so it cought my eye.
0:43:32
fiddlerwoaroof
https://github.com/fiddlerwoaroof/daydreamer/blob/8b29101b298da546bc1555c5c2abc44622ffa980/daydreamer.asd#L35
0:44:36
fiddlerwoaroof
It's better than conditionals in your .asd because ASD can track the features
0:45:52
fiddlerwoaroof
I mostly use it to filter out files that contain code that doesn't work in Lispworks
4:07:24
phoe
well, my old personal statement still holds true; I can only properly wish you good morning if I don't sleep during the night
4:10:18
phoe
been busy earning the achievement of being proclaimed the biggest threat to the Common Lisp community right now
4:11:45
phoe
I guess that I annoyed him a wee bit too much with my posts at https://github.com/cl-library-docs/common-lisp-libraries/issues/3
4:14:07
phoe
beach: no worries. I am glad that I was able to spend some time and compile some publicly available data on him that will hopefully help keep the Lisp community a better place in the future.
4:15:01
phoe
and the best news is that neither Xach nor lispm hold the position of the #1 Lisp baddie anymore!