libera/#commonlisp - IRC Chatlog
Search
9:45:04
lisp123_
flip214: then that would be the answer to the earlier query on how to reverse proxy in CL
9:45:50
flip214
unless there is some specific rewriting to do, I'd go for a standard product, though (nginx, apache, haproxy, etc.). even casual text changes (via regexes) can be done in apache, for example.
9:46:19
lisp123_
yeah nginx is a very easy to use reverse proxy imo (well since I know how to use it, hence its easy to use :D)
9:51:05
flip214
lisp123_: depending on the answer time of the http services "behind" the proxy you might have quite a lot of simultaneous active http connections.
9:51:43
flip214
HT by default creates threads; quux-ht uses a thread pool; perhaps you should take a look at tpd2 or so, to have a bounded number of threads for an (more or less) unbounded number of connections.
9:52:59
lisp123_
flip214: Good to know :) I was just answering the earlier question on the chat, I've personally moved away from web dev for the foreseeable future
9:53:16
lisp123_
perhaps I will come back in the future, after all everything commercial is a SaaS these days
15:53:51
nij-
Can someone recommand a noob friendly CL implementation that comes with an editor, works with windows and macos, and that allows the user to start playing with the repl after a quick download?
16:01:30
nij-
I remember there's one that satisfies all of these requirements.. but I couldn't find its name..
16:11:09
etimmons
nij-: People like Portacle, but it's not an implementation. Just a bundling of an implementation and other things.
16:12:26
Shinmera
Unfortunately macOS' release of Portacle is pretty busted, and I don't have the time... or, rather, the patience and energy to play catch-up with Apples BS.
16:13:36
Shinmera
If there's any other poor souls willing to sacrifice themselves to do the work though, by all means, PRs are open.
16:24:55
Shinmera
beach: In my case it's because other people that do use Apple devices heckle me about it
17:12:14
jeosol
* debugging an application that works correctly in serial mode but gives inconsistent results in parallel mode (race problems, etc) is a different kind of hell
17:14:49
nij-
Shinmera: if Portacle is just a bunch of implementations, why wouldn't it do well on macos?
17:15:52
Shinmera
First fo all, that's being very dismissive of all the work that had to be done, and second the reason is because apple wants you to distribute stuff solely through the app store and makes everything else a huge pain in the ass to force your hand.
17:17:38
shka
OSX used to be pretty nice platform for the software distribution, shame that it ends this way
17:18:16
Shinmera
Turns out when companies are doing well they start fucking everyone over. Who knew???
17:20:27
nij-
Sigh* Do you think doing that favors the those who will never learn any bit of programming?
17:21:00
Shinmera
Don't worry, I'm currently bleeding through all of my savings and probably will end up bankrupt within the next year, so me not doing well is virtually guaranteed :)
17:37:16
jackdaniel
I think that it is a matter of incentives; when you respond before a board assigned by shareholders, then you aim for growth and revenue; when you respond to your family business then you seek stability; when you respond to a non-profit then you have "its" goal in mind
17:41:11
jeosol
I am sure we'd be able to rally round. when my lot changes soon, i should be able to make some contributions to your efforts
17:41:45
Shinmera
shka: You can! https://gum.co/yukari2022 you could also support me on github if you want: https://github.com/sponsors/shinmera
17:45:48
jeosol
calling out threading experts, my code exhibiting weird behavior and I don't throwing locks here and there is the way to do it. I am using bordeaux threads for simd-like tasks. I have 3 tasks A, B, C that I would like to run in parallel: A writes files required for step B, B does a system call using the file from step A and 3rd party exe, and C
17:46:52
jeosol
For step A, I occassionally, get the inputs jumbled when I look at the files for different cases, e.g., a file for a different task ends up being included for a different task.
17:47:26
jeosol
I would like to chain all three steps A, B, C, but perhaps doing A serially, B parallel, C serially is more tractable
17:48:22
Josh_2
Instead of a lock you could use semaphores to signal when B when A is done, and C when B is done
17:50:32
jeosol
Bike: It is a population-based optimization I am doing and I need to evaluate F(X) for several iterations and populations. Step B is the most expensive and can run for hours, I have created a small case that finishes in 2 seconds for my testing. steps A and C are very fast and not the bottle neck
17:51:33
jeosol
issue is when writing the files, some of the indices are jumbled. For instance in a given iteration, say iteration 1, I can have 20 solutions. so I append the iteration index and population index to the ends of files in each folder. e.g., input_iter1_sol2.dat
17:51:49
shka
Shinmera: i grabbed two for 15 bucks each, it also looks like something i could appreciate on aesthetically
17:52:21
Josh_2
jeosol: http://alumni.cs.ucr.edu/~kishore/papers/semaphores.pdf maybe you would find the pattern in section 3.5 useful
17:52:34
jeosol
All relevant files in the same folder should have same iteration and solution indices: e.g., <prefix file>_iter1_sol2.dat
17:53:26
jeosol
So far, I have used locks, but it stinks and works in some cases and other it doesn't. However in serial mode, everything is kosher
17:53:36
Bike
i'm sorry, you're giving a very high level description of your system but it's hard to understand the particular problem. is it something like, a file produced by an A task is being processed by the wrong kind of B task?
17:56:19
Josh_2
does sound to me like you need semaphores, you can use them to pause and synchronize the execution of the tasks. if B depends on the completion of A, then B can sit on its semaphore until A signals it, then B will execute, same for B and C
17:56:34
jeosol
Bike: I didn't want to get into too much details. I wanted to understand any CL aspects in case I am missing something. In the above only step A is critical, as if the wrong files ended up in the wrong folder, the application will file as it can't get the folder so I say objective function is some high value 1e10 (for minimization task)
17:57:06
jeosol
I want to be able to do a call, e.g., like (call-with-thread-lock (function-A) ...) or something similar so the file writing steps is correct
18:00:14
Bike
when you exit the with-lock-held form, the lock is released. your release-lock call is redundant and probably an error or something.
18:01:30
Bike
it will evaluate the cleanup form once unconditionally, and then again if it returns true
18:01:58
Bike
you probably meant to check whether there's a cleanup form at macroexpansion time rather than runtime
18:02:54
Guest82
phantomics a simple social network like app with a reactjs front end (or angular) and managing docs, videos, commenting, and the like
18:03:47
Bike
just to be very clear about with-lock-held - the lock will be released if the form is exited in any way. it essentially does an unwind-protect to ensure it's released.
18:04:30
jeosol
so my unwind-protect on top is reduntant since the with-lock-held would have released the lock on exit
18:05:38
jeosol
I guess I can simplify things, I rarely have a cleanup form, or just chain all tasks together
18:06:52
Bike
you could dispense with this macro, and instead just do (unwind-protect (with-lock-held ...) cleanup) when you need to
18:06:57
yitzi
Does anybody know if it is a hard requirement that logical pathname words get upcased? CLHS 19.3.1.1.7 seems to imply that to me, but CCL appears to preserve case words.
18:07:00
jeosol
Bike: Thanks for helping brain stop this. But in your view, if I do something like this, it should help with the race issues?
18:07:20
Bike
jeosol: like i said, i don't understand your race issues. i just know that you're doubly releasing locks and that can't be good
18:08:17
Bike
and Josh_2 is right that from your description you might want a semaphore structure rather than locks per se
18:08:47
_death
it's not clear to me what you expect to gain by parallelization, if processes A and C are much faster than B
18:09:40
Guest82
phantomics oh, didn't know that... have seen many people using react, yet also see many people and companies using angular...
18:14:54
phantomics
React is worth learning for sure, you can also use the Panic library to generate React in tandem with CL-WHO
18:15:02
jeosol
_death: I apologize since I it's not very clear. I am doing optimization and for each F(X) evaluation, I run three steps A, B, and C. B is a numerical simulation step (fluid modeling on a 3d grid) using a 3rd party application. Writing the input files and reading the output are easy and fast they are both textfiles
18:15:47
jeosol
For task B, the size of the grid affects the run time, so I created a smaller grid 40x40x1 for my test case. Realistic ones can have 500K cells and takes longer to run
18:16:18
phantomics
Using CL-WHO and Panic, you can meld React components, JS and HTML together in a seamless way
18:17:10
_death
jeosol: so are you running a single B process, or multiple ones doing different inputs?
18:17:20
jeosol
death: so in a population-based optimizer, in each iteration you get 20 solutions, there are 20 F(X)'s to compute, if each F(X) takes an hour, if I can do all F(X) in parallel, they would take 1 hour ceteris paribus, instead of 20 hours - that is where the parallelization comes in
18:19:25
jeosol
one process, but I create as many threads as need and call the function that does steps A, B, and C. But I think steps A where I write the input files, results get jumbled. Onces the files are written correctly, Step B is deterministic and Step C is just read a file in the correct folder
18:19:27
_death
jeosol: right.. so basically it sounds like https://zguide.zeromq.org/docs/chapter1/#Divide-and-Conquer
18:24:31
shka
so when you call lparallel:pmapcar, you will get list of results, just like with mapcar
18:26:52
jeosol
yeah its similar to what I am doing, but I can't say I am using lparallel. My issue is the shear size of parameters - unfortunately, I have functions all over the place. So I was wondering if I could somehow tie them together and use the lock, everything will be kosher
18:27:45
jeosol
it seems it will do that I want especially with the blocking part but still running in parallel
18:30:07
jeosol
I was only using it for those Task A writing files that don't take much time, but for B, no locking, until I get to C again to read out. The task B - more expensive part, has no locking
18:30:33
jeosol
But a cleaner solution that allows me to just chain everything and not worry, if at all, is better.
18:31:35
Bike
Josh_2: only a really dumb lock implementation actually just spins. they try to yield the processor.
18:31:52
jeosol
several months ago, some I chatted with on linked advised to look at lparallel but since bt-threads was working, I never saw the need. I guess I have to look into it now
18:32:46
_death
jeosol: maybe you can arrange it so that you have a function (defun my-task (number) (read-outputs (magic (write-inputs number)))) .. then pmapcar it as shka suggested
18:32:50
shka
i use lparallel almost everyday, it saved me countless hours of programming and debugging
18:33:10
jeosol
ok, I was low level, writing my joins, etc. May be I need to stay higher-level since there are aspects I may not understand very well
18:34:28
jeosol
_death: I see. That's would be much cleaner - so at the level of calling my-task function, the locking (?) and blocking is taken care of?
18:35:12
Bike
well if each indexed task can be handled independently you shouldn't really need to lock anything
18:35:29
jeosol
shka: I think I should do same, move to lparallel. I debugged yesterday and was able to isolate the issue to the parallelization as serial model gives the correct results and I noticed the jumbled input files
18:36:05
jeosol
Bike: There are some shared resources (CLOS objects) - this may be where the issues are
18:36:50
_death
jeosol: there's no need for locking if the inputs and outputs are written in separate files, and magic only depends on the inputs.. a naive approach would be to have start a thread for each task and just join them all.. lparallel allows you to limit the number of tasks running at once
18:37:15
jeosol
shka: good point, unfortunately, not all the way through, for instance, I am taking some parameters from a higher level object and saving them in a lower object (iteration and solution indices) - i noticed an issue there
18:38:50
jeosol
_death: ok I see, I am doing something similar but have a loop with bt-threads functionality to track the join, specify a batch-job parameter and all. But I should probably just be using a library if it will save me all the headache
18:39:33
jeosol
ultimately, my goal is to do distributed computing, but I have only used swank-crew to run on another box, for one of computation and it's ok so far
18:41:29
_death
there is also an lparallel for multiple machines, called lfarm.. I've no experience with it though
18:43:19
jeosol
oh really. that'd be interesting - I checked aws but they were expensive. Someone here referred me to some European option. The plan will be to have some machines with the executable for task B installed, and SBCL running, route the jobs, and get the results back
18:46:54
jeosol
_death, Bike, shka, Josh_2, others: Thanks guys, I appreciate the help. I don't want to take over the channel, but I have gotten good pointers to follow up on - should probably stay way from managing the threads creation joining etc and just use lparallel API's
18:49:19
shka
jeosol: sometimes you have to get your hands dirty with BT directly, plus at least sometimes you need to use locks, but yeah, most frequently problems can be solved in lparallel in a few lines of code
18:50:17
jeosol
shka: haha, I think they are dirty enough, lol, I used to use pthreads with C++ code - not an easy experience
18:51:13
jeosol
skha: I handle the thread batching via loops and joining, but I should probably just offload that task and not have to worry much. Yeah, the locks have helped but I should probably redesign to avoid setting variables in thread calls
18:51:48
jeosol
shka: My C/C++ days was back in graduate schools, I have not coded C/C++ since leaving, like 2010
18:52:30
jeosol
shka: thanks. I have a deadline for a conference, and need to have this parallelization part to work to be able to submit the runs
18:54:35
shka
the fun part with lparallel is that you can write (apply (if parallelp #'lparallel:pmap #'cl:map) nil function input-sequence)
18:55:28
shka
which allows you to use the exact same source code for both parallel and serial execution
19:00:33
jeosol
skha: good point: The ability to use the same code for serial and parallel is definitely import. For now, I have a use-parallel-p variable that I use to test both parts. Serial part is trival because I have a loop that calls a function withich chains all the steps I need
19:01:33
jeosol
shka: I definitely need to make my life easier when i comes to executing parallel jobs, so I should consider alternatives
19:03:39
jeosol
shka: by "removing synchronization points ..." you mean avoid writing to shared resource?
19:04:33
jeosol
shka: I agree. I think I will spend some time to redesign so as not to worry about this again
19:05:11
jeosol
I have a part were are save some variables in an object they get jumbled some pointing to previous indices.
19:06:01
shka
and if that does not work, you can use lparallel:future and chain which allows you to link execution steps
19:09:16
jeosol
shka: the call to F(X) is from an upstream higher level algorithm (optimizer) with F(X_i) doing F(A(X_i)), F(B(X_i)), and F(C(X_i)) - so having the main F(X_i) execute without the issues is better. The higher-level algorithm doesn't know about the CLOS object used to compute F(X_i)
19:09:17
shka
jeosol: real world example https://github.com/sirherrbatka/clusters/blob/a0c565a95d66ba025277fca41ec3b3e4c05b1226/source/k-means/utils.lisp#L67
19:09:54
shka
clusters.utils:pmap calls either cl:map or lparallel:pmap depending on the first argument
19:11:47
shka
https://github.com/sirherrbatka/clusters/blob/a0c565a95d66ba025277fca41ec3b3e4c05b1226/source/common/methods.lisp#L85 perhaps something more interesting
19:12:22
jeosol
noticed compiler type instructures "the" and other declare statements -- what is the speed up with and without, I know it probably depends
19:13:57
shka
well, i guess perhaps for arrays as well, since maybe compiler will be able to inline memory access
19:14:26
jeosol
my matrix-vector operations are not optimized and used loops, etc. it was just to check my understanding
19:16:46
jeosol
I normally do computational tasks but for my task B (numerical modeling part) I just use a 3rd party executable. It will take lots of efforts to write a 3D numerical solver and then worry about matrix conditions, optimize computations, etc. The 3rd party exe was written in Fortran
19:21:46
shka
jeosol: check the following (disassemble (lambda (a b) (declare (type double-float a b) (optimize (speed 3) (safety 0))) (* a b)))
19:23:12
Inline
so what does it get you to write the last one as (* (the double-float a) (the double-float b)) ?
19:23:32
shka
jeosol: check the following (disassemble (lambda (a b) (declare (type single-float a b) (optimize (speed 3) (safety 0))) (* a b)))
19:28:03
jeosol
Some neural network training on my desk top didn't finish after several days, other guy said it ran cuda arrays on gpu, in 24 hours - that's a massive speed up
19:31:54
jeosol
I occasionally work with python for the DL libraries, but it can be pain in notebooks having to go up and down to reevaluate some cells come back down, only to realize one other computation is stale, etc
19:32:21
Inline
i see no threading for single-floats but when using double floats there is some threading going on
19:34:34
Bike
a double float needs 64 bits to represent, so there's no room for a type tag. so they're boxed if they need to be used in a type sensitive context. ergo, consing.
19:34:52
Bike
a single float is only 32 bits so it fits with a type tag into a 64 bit word just fine.
19:35:08
shka
yet i vividly remember when i stumbled in something float related in SBCL that worked faster with double-floats then a single floats
19:36:05
Inline
welp, the fastness i also skimmed from books recommending to use bigger types, the machine architectures are then faster because of bus design or so....
19:36:25
jeosol
Bike: I was recently following a talk on Julia why the guy was saying it faster than python and many other languages (don't recall he mentioned C). But one thing he kept saying is that if you have to box and unbox, he said you are "dead"
19:38:25
shka
well, you may work fine with double-floats in sbcl, but you will gonna need type declarations for arrays and use inline quite a bit
19:40:24
jeosol
Btw, on my threading issues mention earlier, is passing data from upstream to downstream object a bad design or code smell - this is where I tell the lower object what their index is that is used to create a directory later on
19:42:16
jeosol
as long as I can map the X_i to some folder and just pick the results, I probably don't need to pass an object (and I do a setf on a slot) this is where things are getting messed up
19:44:34
shka
jeosol: as for index, if that's just iota, well, you can simply do something like (lparallel:pmap nil (lambda (argument index) ...) arguments (alexandria:iota (length arguments)))
19:45:46
Shinmera
shka: Thanks a lot! My finances will stay troublesome as long as I have to fund a team with no income, heh.
19:46:31
shka
Shinmera: btw, did you painted this calendar using your drawning app that name i can't remember right now?
19:47:11
jeosol
skha: technically, the index corresponds to rows of a larger matrix (num_solutions, num_dimensions), so I slice of each row (X_i) to a separate thread
19:49:27
shka
you may prefer to return vector which gets stacked into matrix as an after step but at this point this is really just stylistic choice
19:51:24
jeosol
shka: thanks buddy. I really appreciate the pointers and suggestions. I will try them
20:14:16
jmercouris
Shinmera: let's say I invoke it on the document root, and I want to avoid getting script tags