libera/#commonlisp - IRC Chatlog
Search
17:12:14
jeosol
* debugging an application that works correctly in serial mode but gives inconsistent results in parallel mode (race problems, etc) is a different kind of hell
17:14:49
nij-
Shinmera: if Portacle is just a bunch of implementations, why wouldn't it do well on macos?
17:15:52
Shinmera
First fo all, that's being very dismissive of all the work that had to be done, and second the reason is because apple wants you to distribute stuff solely through the app store and makes everything else a huge pain in the ass to force your hand.
17:17:38
shka
OSX used to be pretty nice platform for the software distribution, shame that it ends this way
17:18:16
Shinmera
Turns out when companies are doing well they start fucking everyone over. Who knew???
17:20:27
nij-
Sigh* Do you think doing that favors the those who will never learn any bit of programming?
17:21:00
Shinmera
Don't worry, I'm currently bleeding through all of my savings and probably will end up bankrupt within the next year, so me not doing well is virtually guaranteed :)
17:37:16
jackdaniel
I think that it is a matter of incentives; when you respond before a board assigned by shareholders, then you aim for growth and revenue; when you respond to your family business then you seek stability; when you respond to a non-profit then you have "its" goal in mind
17:41:11
jeosol
I am sure we'd be able to rally round. when my lot changes soon, i should be able to make some contributions to your efforts
17:41:45
Shinmera
shka: You can! https://gum.co/yukari2022 you could also support me on github if you want: https://github.com/sponsors/shinmera
17:45:48
jeosol
calling out threading experts, my code exhibiting weird behavior and I don't throwing locks here and there is the way to do it. I am using bordeaux threads for simd-like tasks. I have 3 tasks A, B, C that I would like to run in parallel: A writes files required for step B, B does a system call using the file from step A and 3rd party exe, and C
17:46:52
jeosol
For step A, I occassionally, get the inputs jumbled when I look at the files for different cases, e.g., a file for a different task ends up being included for a different task.
17:47:26
jeosol
I would like to chain all three steps A, B, C, but perhaps doing A serially, B parallel, C serially is more tractable
17:48:22
Josh_2
Instead of a lock you could use semaphores to signal when B when A is done, and C when B is done
17:50:32
jeosol
Bike: It is a population-based optimization I am doing and I need to evaluate F(X) for several iterations and populations. Step B is the most expensive and can run for hours, I have created a small case that finishes in 2 seconds for my testing. steps A and C are very fast and not the bottle neck
17:51:33
jeosol
issue is when writing the files, some of the indices are jumbled. For instance in a given iteration, say iteration 1, I can have 20 solutions. so I append the iteration index and population index to the ends of files in each folder. e.g., input_iter1_sol2.dat
17:51:49
shka
Shinmera: i grabbed two for 15 bucks each, it also looks like something i could appreciate on aesthetically
17:52:21
Josh_2
jeosol: http://alumni.cs.ucr.edu/~kishore/papers/semaphores.pdf maybe you would find the pattern in section 3.5 useful
17:52:34
jeosol
All relevant files in the same folder should have same iteration and solution indices: e.g., <prefix file>_iter1_sol2.dat
17:53:26
jeosol
So far, I have used locks, but it stinks and works in some cases and other it doesn't. However in serial mode, everything is kosher
17:53:36
Bike
i'm sorry, you're giving a very high level description of your system but it's hard to understand the particular problem. is it something like, a file produced by an A task is being processed by the wrong kind of B task?
17:56:19
Josh_2
does sound to me like you need semaphores, you can use them to pause and synchronize the execution of the tasks. if B depends on the completion of A, then B can sit on its semaphore until A signals it, then B will execute, same for B and C
17:56:34
jeosol
Bike: I didn't want to get into too much details. I wanted to understand any CL aspects in case I am missing something. In the above only step A is critical, as if the wrong files ended up in the wrong folder, the application will file as it can't get the folder so I say objective function is some high value 1e10 (for minimization task)
17:57:06
jeosol
I want to be able to do a call, e.g., like (call-with-thread-lock (function-A) ...) or something similar so the file writing steps is correct
18:00:14
Bike
when you exit the with-lock-held form, the lock is released. your release-lock call is redundant and probably an error or something.
18:01:30
Bike
it will evaluate the cleanup form once unconditionally, and then again if it returns true
18:01:58
Bike
you probably meant to check whether there's a cleanup form at macroexpansion time rather than runtime
18:02:54
Guest82
phantomics a simple social network like app with a reactjs front end (or angular) and managing docs, videos, commenting, and the like
18:03:47
Bike
just to be very clear about with-lock-held - the lock will be released if the form is exited in any way. it essentially does an unwind-protect to ensure it's released.
18:04:30
jeosol
so my unwind-protect on top is reduntant since the with-lock-held would have released the lock on exit
18:05:38
jeosol
I guess I can simplify things, I rarely have a cleanup form, or just chain all tasks together
18:06:52
Bike
you could dispense with this macro, and instead just do (unwind-protect (with-lock-held ...) cleanup) when you need to
18:06:57
yitzi
Does anybody know if it is a hard requirement that logical pathname words get upcased? CLHS 19.3.1.1.7 seems to imply that to me, but CCL appears to preserve case words.
18:07:00
jeosol
Bike: Thanks for helping brain stop this. But in your view, if I do something like this, it should help with the race issues?
18:07:20
Bike
jeosol: like i said, i don't understand your race issues. i just know that you're doubly releasing locks and that can't be good
18:08:17
Bike
and Josh_2 is right that from your description you might want a semaphore structure rather than locks per se
18:08:47
_death
it's not clear to me what you expect to gain by parallelization, if processes A and C are much faster than B
18:09:40
Guest82
phantomics oh, didn't know that... have seen many people using react, yet also see many people and companies using angular...
18:14:54
phantomics
React is worth learning for sure, you can also use the Panic library to generate React in tandem with CL-WHO
18:15:02
jeosol
_death: I apologize since I it's not very clear. I am doing optimization and for each F(X) evaluation, I run three steps A, B, and C. B is a numerical simulation step (fluid modeling on a 3d grid) using a 3rd party application. Writing the input files and reading the output are easy and fast they are both textfiles
18:15:47
jeosol
For task B, the size of the grid affects the run time, so I created a smaller grid 40x40x1 for my test case. Realistic ones can have 500K cells and takes longer to run
18:16:18
phantomics
Using CL-WHO and Panic, you can meld React components, JS and HTML together in a seamless way
18:17:10
_death
jeosol: so are you running a single B process, or multiple ones doing different inputs?
18:17:20
jeosol
death: so in a population-based optimizer, in each iteration you get 20 solutions, there are 20 F(X)'s to compute, if each F(X) takes an hour, if I can do all F(X) in parallel, they would take 1 hour ceteris paribus, instead of 20 hours - that is where the parallelization comes in
18:19:25
jeosol
one process, but I create as many threads as need and call the function that does steps A, B, and C. But I think steps A where I write the input files, results get jumbled. Onces the files are written correctly, Step B is deterministic and Step C is just read a file in the correct folder
18:19:27
_death
jeosol: right.. so basically it sounds like https://zguide.zeromq.org/docs/chapter1/#Divide-and-Conquer
18:24:31
shka
so when you call lparallel:pmapcar, you will get list of results, just like with mapcar
18:26:52
jeosol
yeah its similar to what I am doing, but I can't say I am using lparallel. My issue is the shear size of parameters - unfortunately, I have functions all over the place. So I was wondering if I could somehow tie them together and use the lock, everything will be kosher
18:27:45
jeosol
it seems it will do that I want especially with the blocking part but still running in parallel
18:30:07
jeosol
I was only using it for those Task A writing files that don't take much time, but for B, no locking, until I get to C again to read out. The task B - more expensive part, has no locking
18:30:33
jeosol
But a cleaner solution that allows me to just chain everything and not worry, if at all, is better.
18:31:35
Bike
Josh_2: only a really dumb lock implementation actually just spins. they try to yield the processor.
18:31:52
jeosol
several months ago, some I chatted with on linked advised to look at lparallel but since bt-threads was working, I never saw the need. I guess I have to look into it now
18:32:46
_death
jeosol: maybe you can arrange it so that you have a function (defun my-task (number) (read-outputs (magic (write-inputs number)))) .. then pmapcar it as shka suggested
18:32:50
shka
i use lparallel almost everyday, it saved me countless hours of programming and debugging
18:33:10
jeosol
ok, I was low level, writing my joins, etc. May be I need to stay higher-level since there are aspects I may not understand very well
18:34:28
jeosol
_death: I see. That's would be much cleaner - so at the level of calling my-task function, the locking (?) and blocking is taken care of?
18:35:12
Bike
well if each indexed task can be handled independently you shouldn't really need to lock anything
18:35:29
jeosol
shka: I think I should do same, move to lparallel. I debugged yesterday and was able to isolate the issue to the parallelization as serial model gives the correct results and I noticed the jumbled input files
18:36:05
jeosol
Bike: There are some shared resources (CLOS objects) - this may be where the issues are
18:36:50
_death
jeosol: there's no need for locking if the inputs and outputs are written in separate files, and magic only depends on the inputs.. a naive approach would be to have start a thread for each task and just join them all.. lparallel allows you to limit the number of tasks running at once
18:37:15
jeosol
shka: good point, unfortunately, not all the way through, for instance, I am taking some parameters from a higher level object and saving them in a lower object (iteration and solution indices) - i noticed an issue there
18:38:50
jeosol
_death: ok I see, I am doing something similar but have a loop with bt-threads functionality to track the join, specify a batch-job parameter and all. But I should probably just be using a library if it will save me all the headache
18:39:33
jeosol
ultimately, my goal is to do distributed computing, but I have only used swank-crew to run on another box, for one of computation and it's ok so far
18:41:29
_death
there is also an lparallel for multiple machines, called lfarm.. I've no experience with it though
18:43:19
jeosol
oh really. that'd be interesting - I checked aws but they were expensive. Someone here referred me to some European option. The plan will be to have some machines with the executable for task B installed, and SBCL running, route the jobs, and get the results back
18:46:54
jeosol
_death, Bike, shka, Josh_2, others: Thanks guys, I appreciate the help. I don't want to take over the channel, but I have gotten good pointers to follow up on - should probably stay way from managing the threads creation joining etc and just use lparallel API's
18:49:19
shka
jeosol: sometimes you have to get your hands dirty with BT directly, plus at least sometimes you need to use locks, but yeah, most frequently problems can be solved in lparallel in a few lines of code
18:50:17
jeosol
shka: haha, I think they are dirty enough, lol, I used to use pthreads with C++ code - not an easy experience
18:51:13
jeosol
skha: I handle the thread batching via loops and joining, but I should probably just offload that task and not have to worry much. Yeah, the locks have helped but I should probably redesign to avoid setting variables in thread calls
18:51:48
jeosol
shka: My C/C++ days was back in graduate schools, I have not coded C/C++ since leaving, like 2010
18:52:30
jeosol
shka: thanks. I have a deadline for a conference, and need to have this parallelization part to work to be able to submit the runs
18:54:35
shka
the fun part with lparallel is that you can write (apply (if parallelp #'lparallel:pmap #'cl:map) nil function input-sequence)
18:55:28
shka
which allows you to use the exact same source code for both parallel and serial execution
19:00:33
jeosol
skha: good point: The ability to use the same code for serial and parallel is definitely import. For now, I have a use-parallel-p variable that I use to test both parts. Serial part is trival because I have a loop that calls a function withich chains all the steps I need
19:01:33
jeosol
shka: I definitely need to make my life easier when i comes to executing parallel jobs, so I should consider alternatives
19:03:39
jeosol
shka: by "removing synchronization points ..." you mean avoid writing to shared resource?
19:04:33
jeosol
shka: I agree. I think I will spend some time to redesign so as not to worry about this again
19:05:11
jeosol
I have a part were are save some variables in an object they get jumbled some pointing to previous indices.
19:06:01
shka
and if that does not work, you can use lparallel:future and chain which allows you to link execution steps
19:09:16
jeosol
shka: the call to F(X) is from an upstream higher level algorithm (optimizer) with F(X_i) doing F(A(X_i)), F(B(X_i)), and F(C(X_i)) - so having the main F(X_i) execute without the issues is better. The higher-level algorithm doesn't know about the CLOS object used to compute F(X_i)
19:09:17
shka
jeosol: real world example https://github.com/sirherrbatka/clusters/blob/a0c565a95d66ba025277fca41ec3b3e4c05b1226/source/k-means/utils.lisp#L67
19:09:54
shka
clusters.utils:pmap calls either cl:map or lparallel:pmap depending on the first argument
19:11:47
shka
https://github.com/sirherrbatka/clusters/blob/a0c565a95d66ba025277fca41ec3b3e4c05b1226/source/common/methods.lisp#L85 perhaps something more interesting
19:12:22
jeosol
noticed compiler type instructures "the" and other declare statements -- what is the speed up with and without, I know it probably depends
19:13:57
shka
well, i guess perhaps for arrays as well, since maybe compiler will be able to inline memory access
19:14:26
jeosol
my matrix-vector operations are not optimized and used loops, etc. it was just to check my understanding
19:16:46
jeosol
I normally do computational tasks but for my task B (numerical modeling part) I just use a 3rd party executable. It will take lots of efforts to write a 3D numerical solver and then worry about matrix conditions, optimize computations, etc. The 3rd party exe was written in Fortran
19:21:46
shka
jeosol: check the following (disassemble (lambda (a b) (declare (type double-float a b) (optimize (speed 3) (safety 0))) (* a b)))
19:23:12
Inline
so what does it get you to write the last one as (* (the double-float a) (the double-float b)) ?
19:23:32
shka
jeosol: check the following (disassemble (lambda (a b) (declare (type single-float a b) (optimize (speed 3) (safety 0))) (* a b)))
19:28:03
jeosol
Some neural network training on my desk top didn't finish after several days, other guy said it ran cuda arrays on gpu, in 24 hours - that's a massive speed up
19:31:54
jeosol
I occasionally work with python for the DL libraries, but it can be pain in notebooks having to go up and down to reevaluate some cells come back down, only to realize one other computation is stale, etc
19:32:21
Inline
i see no threading for single-floats but when using double floats there is some threading going on
19:34:34
Bike
a double float needs 64 bits to represent, so there's no room for a type tag. so they're boxed if they need to be used in a type sensitive context. ergo, consing.
19:34:52
Bike
a single float is only 32 bits so it fits with a type tag into a 64 bit word just fine.
19:35:08
shka
yet i vividly remember when i stumbled in something float related in SBCL that worked faster with double-floats then a single floats
19:36:05
Inline
welp, the fastness i also skimmed from books recommending to use bigger types, the machine architectures are then faster because of bus design or so....
19:36:25
jeosol
Bike: I was recently following a talk on Julia why the guy was saying it faster than python and many other languages (don't recall he mentioned C). But one thing he kept saying is that if you have to box and unbox, he said you are "dead"
19:38:25
shka
well, you may work fine with double-floats in sbcl, but you will gonna need type declarations for arrays and use inline quite a bit
19:40:24
jeosol
Btw, on my threading issues mention earlier, is passing data from upstream to downstream object a bad design or code smell - this is where I tell the lower object what their index is that is used to create a directory later on
19:42:16
jeosol
as long as I can map the X_i to some folder and just pick the results, I probably don't need to pass an object (and I do a setf on a slot) this is where things are getting messed up
19:44:34
shka
jeosol: as for index, if that's just iota, well, you can simply do something like (lparallel:pmap nil (lambda (argument index) ...) arguments (alexandria:iota (length arguments)))
19:45:46
Shinmera
shka: Thanks a lot! My finances will stay troublesome as long as I have to fund a team with no income, heh.
19:46:31
shka
Shinmera: btw, did you painted this calendar using your drawning app that name i can't remember right now?
19:47:11
jeosol
skha: technically, the index corresponds to rows of a larger matrix (num_solutions, num_dimensions), so I slice of each row (X_i) to a separate thread
19:49:27
shka
you may prefer to return vector which gets stacked into matrix as an after step but at this point this is really just stylistic choice
19:51:24
jeosol
shka: thanks buddy. I really appreciate the pointers and suggestions. I will try them
20:14:16
jmercouris
Shinmera: let's say I invoke it on the document root, and I want to avoid getting script tags
20:43:57
Guest82
does anyone know how to print to the repl while using a controller (read: web.lisp) in caveman2? phantomics
20:45:29
Guest82
I am too much of a noob to even understand what inferior lisp means... I'm using sublime text with a package called sly
20:49:17
Guest82
jmercouris I'm getting errors, package doesn't exist, tried doing ql:quickload "swank" but then still didn't work
20:51:04
Guest82
oh, meaning doing something like this (format SWANK::*CURRENT-STANDARD-OUTPUT* "hello")
20:53:34
Guest82
pve not sure what that means..... I started the server in the repl... but the repl is probably running as a bash process and being called from the editor sublime text
20:55:49
Guest82
@pve Is there a way to see all the variables in the current script? I tried (inspect *readtable*) but didn't understand the output so much
20:56:49
Guest82
I feel like there's a bit of a jump between starting to program in lisp and understanding how to deal with all these things...
20:57:01
pve
Guest82: Sorry I'm not familiar with caveman2, but you could at least print to a file if you can't see standard-output
20:58:54
pve
Guest82: (with-open-file (*out* "debug.log" :direction :output :if-exists :append) (print *out* "Hello!"))
21:02:31
lisp123_
Guest82: "I feel like there's a bit of a jump between starting to program in lisp and understanding how to deal with all these things..." --> I won't lie, it will take a bit of time, but the benefits will be great down the track
21:04:45
pve
Guest82: it may be that running the thing from within sublime is complicating things too much if you are unable to see any output
21:05:14
Guest82
lisp123_ I find macros are the solution to my problems fighting languages to be able to abstract things, but I feel like there should be a smaller barrier to entry than learning lisp and emacs at the same time
21:05:44
pve
Guest82: You could try starting it manually from the shell instead. Then you should see the debugging output at least.
21:05:56
Guest82
pve I hear, I understand everything is easier with emacs, but I don't know how to use emacs and I feel it's too much to learn both in one shot
21:07:11
pve
Guest82: remember to familiarize yourself with the various command line parameters like --eval and --load to get maximum convenience
21:07:29
lisp123_
Guest82: Try Portacle. I don't disagree with you, the combo of learning lisp and emacs makes it a bit harder, but both tools have a lot of benefits too - so you will get a lot of success down the track :)
21:08:07
lisp123_
Guest82: Sorry to beat a daed horse, but I _highly_ recommend emacs for any sort of lisp programming
21:08:22
Guest82
lisp123_ yeah, from what've I've read it seems it's a great combination. I tried portacle but they didn't support latest mac os
21:08:54
pve
Guest82: and if working from the shell, do define convenience functions or symbol-macros to reload your project quickly
21:11:35
pve
Guest82: silly example: (define-symbol-macro rr (asdf:load-system "myapp")) will make "rr" reload your stuff after you've edited it in sublime
21:24:25
Guest82
lisp123_ what's the easiest tutorial about dealing with lisp on emacs? I feel like the tutrial here is just text editting, and then to do lisp modes and other things is another huge jump
21:25:35
pjb
Guest82: Bug for CL, it's http://cliki.net/Getting+Started and http://cliki.net/Online+Tutorial
21:43:21
pve
Guest82: I like Marco Baringer's SLIME video (https://www.youtube.com/watch?v=NUpAvqa5hQw), you can skip to around the 10:00 mark.
21:46:24
Guest82
Warning (initialization): An error occurred while loading ‘/Users/danielnussenbaum/.emacs.d/init.el’:
21:46:25
Guest82
File is missing: Opening input file, No such file or directory, /Users/danielnussenbaum/zsh:1: command not found: rosconfig
21:50:04
pve
Guest82: I'm rewatching the slime video now, and it's incredible how useful it still is, despite being a little old
22:03:27
kakuhen
the nice part about quicklisp is not only that it's widely used nowadays, but also lets you quickly test your projects on other cl implementations, assuming you added it to your implementation's init file (i.e. your .sbclrc, .eclrc, and so on)