freenode/#lisp - IRC Chatlog
Search
1:40:31
no-defun-allowed
we're doing a presentation on pidgins and creoles in eng lang so we have different places green screened
1:44:36
AeroNotix
I found an internal advertisement I was made to do ten years ago while I worked at HP on an old hard drive the other day. Other dimensions could feel the cringe.
3:00:40
Pixel_Outlaw
Is there a way to print a literal escape sequence? Thinking about toying with some ReGIS in xterm.
3:00:40
Colleen
Pixel_Outlaw: jackdaniel said at 2018.09.12 05:06:21: it seems that for some reason trivial-garbage's make-weak-hash-table doesn't remf this keyword argument before passing to make-hash-table in sbcl
3:00:40
Colleen
Pixel_Outlaw: jackdaniel said at 2018.09.12 05:06:53: what is weird, because I have newest trivial-garbage and all works just fine for me. do you have some hacked version in your local-projects?
5:37:28
gendl
I'm just going to bed (told my wife I would be there 2 hours ago but got stuck finishing a few things on our new web-based object browser)
5:40:49
gendl
can't rightly complain much, although I still do... been through some challenges lately,
5:41:46
gendl
you know about one of them. That looks like (knock wood) coming to a conclusion here real soon.
5:44:01
gendl
ALU has been in limbo, as far as I'm personally concerned, it's a bit far down on my list.
5:44:35
gendl
RS is technically still president and things are kind of in a holding pattern until he gets a new one
5:45:38
gendl
we have $5000 in matching funds and they're finally in the proper bank account and Stripe is hooked up to it
5:45:57
gendl
... so i just have to get the website spun up and re-skinned (the one from Quicklisp from almost two years ago)
5:47:50
beach
By the way, McCLIM development is very active these days, in case you are looking for a fundraising target in the future.
5:49:28
gendl
well, I think if we can demonstrate some modicum of organization, there are deep-pocketed donors out there who can help us out
5:50:21
gendl
i think it'll be great to have a balance between a good number of grass-roots donors of smaller amounts, and a few deep-pocketed well-wishers who can see that their larger donations are going to good use
5:52:12
beach
Seeing no-defun-allowed, I was reminded about the importance of coming to ELS so that one can recognize the people here.
5:53:01
gendl
I still have to make the XML file to index the ELS proceedings with ACM... I didn't see a hard deadline for that so I'm hoping they'll still accept.
5:53:52
beach
no-defun-allowed: I have told you this before, but I'll say it again. I think you should do what I recommended to froggey, namely put up a crowd funding site to pay your way to ELS 2019.
5:57:03
no-defun-allowed
anyways i do need to do school work for a while, i've got to hack up a video for tomorrow
5:57:55
no-defun-allowed
- make the parallel-mapvideo buffer guess the frame count that can fit in memory instead of a fixed frame count
6:00:27
no-defun-allowed
you're expected to have a medical certificate for days off, and anything below 90% attendance or some shit is failing too
6:01:23
Shinmera
talk to your department lead if you can get the week off to attend an important (!) conference that will further your studies (!)
6:07:36
beach
OK then, first check whether it is possible, hinting that you can get a letter from me.
6:11:54
beach
Go to the head of the school. Say something like "I have become involved in the Common Lisp community, and I have received an invitation to come to ELS 2019... Would there be a possibility for me to get that week off from school?"
6:12:28
beach
no-defun-allowed: It is a common mistake to believe that the talks are the important part of a conference like this.
6:12:49
beach
The important part is to chat with the participants between talk, at lunch, at diner, etc.
6:13:42
no-defun-allowed
is it going to cost anything if i manage to get a flight there? i've never been to europe or planned much myself
6:15:52
no-defun-allowed
the school does a lot of extracurricular stuff so maybe they'll let me go for the week
6:20:22
Shinmera
The story goes that it was initially supposed to be much more lisp-like before it got rushed out the door at netscape
6:28:11
jackdaniel
"In 1995, Netscape Communications recruited Brendan Eich with the goal of embedding the Scheme programming language into its Netscape Navigator."
6:54:13
shrdlu68
"But CL implementations! What big binaries you produce", said the Little Red Riding Hood.
7:00:27
Jachy
beach: was there much overlap in speakers between ELS and <Programming> the previous time they were co-located? I assume lots of attendees of ELS would go to <Programming> while they're there, did the reverse happen too?
7:05:26
jackdaniel
Jachy: unfortunately (imho) things were happening at the same time, so nobody would want to miss ELS to attend programming
7:18:39
aeth
no-defun-allowed: iirc it depends on the format and the direction because it's just a wrapper over other things.
7:22:50
|3b|
opticl has some options for fast, but like most pixel manipulation libs is easy to use slowly :)
7:23:54
|3b|
a large part of png reading time was zlib last i checked, so need to rewrite another lib to improve it much :(
7:25:30
|3b|
ACTION wants a version that can do streaming reads, so i can just start a bunch of threads and ignore some of the speed problems :p
7:26:13
|3b|
while limiting memory usage if i happen to want to load a bunch of 4kx4k images at once or something
7:28:36
jackdaniel
lucky for us it is an opensource library which we can improve to read all of them! :)
7:28:37
no-defun-allowed
it feels slightly faster with a custom ppm writer but i'm Totally Not Biased
7:38:29
no-defun-allowed
i've tried some type definitions and i've got sbcl's (speed 3) down to 3 complaints
7:39:42
no-defun-allowed
i haven't updated cl-vep recently with new stuff but it lives [here](https://gitlab.com/Theemacsshibe/cl-vep)
7:43:23
aeth
asarch: Either you have a sequence of length 9 and by convention treat it as a 3x3 matrix or you have a (3 3) array
7:47:14
aeth
asarch: If you're using SBCL for a lot of 2D array code, SBCL 1.4.10 and later has fixes for bounds checking on 2D arrays
7:48:40
|3b|
use :report :graph (or :type :graph if you call report directly instead of just with-profiling)
7:49:31
|3b|
if you end up using it much, probably also want to install and use the slime slime-sprof contrib, easier to navigate, but harder to paste for other people to look at :)
7:50:31
|3b|
(and you probably want other people to look at it when you start out, not obvious what the various internal things indicate about your code)
7:51:20
no-defun-allowed
i'm fairly sure it's gonna be the image read/write though, i've tuned the effects very carefully
7:53:41
|3b|
also, if you are starting threads inside with-profiling, make sure you wait for them to finish before with-profiling exits
7:55:18
|3b|
ACTION usually tries to profile 5-10sec of work, so process a few images, or do one in a loop or whatever
8:00:11
no-defun-allowed
by the looks of things there are still some array references that didn't get inlined
8:09:01
|3b|
in each block, the line with #s under 'total.' (and name shifted to the left a bit) is the function being measured in that block, above it shows what called it with how much to the left
8:10:12
|3b|
and in first block you can see same info, and that mask-blend is also making full calls to +,-,*
8:12:34
no-defun-allowed
every time i run a parallel-mapvideo, it complains the worked on video is invalid and crashes
8:14:14
no-defun-allowed
also it seems i should have taken a background picture for every clip i masked
8:16:19
shrdlu68
dim: Getting "Failed to connect to pgsql...Can't resolve foreign symbol "SSL_load_error_strings""
8:21:21
no-defun-allowed
also i have to do a beach and set the heap size relatively high to get the worker->writer cache working well enough
8:25:12
|3b|
hmm, actually i guess i've been building with 16gb heap out of 32gb ram lately, i should raise that
8:34:35
shrdlu68
dim: Runnning the dimitri/pgloader container fails with "The alien function "CRYPTO_num_locks" is undefined".
8:34:37
no-defun-allowed
i do need to work on the cache though, and make it use a "maximum pixel count" instead of "maximum frame count"
8:35:27
no-defun-allowed
i also need to work on cl-decentralise but that's not doing any school projects for me
8:37:12
makomo
shrdlu68: i remember having to install an older version of SSL (1.1.0, as opposed to 1.1.1, i think)
8:37:34
makomo
and then there was the problem of cl-plus-ssl not having the correct name of the shared lib in its CFFI defs
8:38:02
shrdlu68
makomo: ..this is in the docker image, shouldn't that bundle up everything it needs?
8:40:02
svillemot
shrdlu68: you may be interested in the patch I wrote to make cl+ssl work with OpenSSL 1.1 https://sources.debian.org/src/cl-plus-ssl/20180328-3/debian/patches/openssl-1.1.patch/
8:42:32
shrdlu68
I'll maybe look at it later, right now I need a pgloader-like thing to complete some task.
8:48:29
Xof
flip214: this? <http://christophe.rhodes.io/notes/blog/posts/2018/algorithms_and_data_structures_term2/> (not complete, haven't actually done the exam analysis, argh)
8:48:30
minion
Xof, memo from flip214: please send me the link with the full results (whole year) of your 'Using Lisp-based pseudocode to probe student understanding' Moodle experiments; ISTR that you posted that in June or so. I'd like to forward that. Thanks a lot!
10:37:10
xificurC
I have an unknown number of lines I'm reading. I want to work through the entries on-the-fly, not accumulating everything first, because the volume will not fit in RAM. So there's no way to tell the final amount. A line needs to be parsed (json) and processed. The processing is fast, the parsing is slow. I want to parallelize this so I picked up lp
10:37:10
xificurC
arallel. I cannot find a solution that would ensure the tasks pushing onto the queue will really all be processed. This is what I've come up with but it still doesn't ensure all messages get processed. http://sprunge.us/1dK2Yw
10:40:07
|3b|
ACTION would probably have the worker loop waiting on the queue, and exit loop when it gets an :exit message (or anything distinguishable from input)
10:40:56
xificurC
|3b|: I thought about that, but who should send that message? How can I make sure that message gets on the queue last?
10:42:47
xificurC
does lparallel.queue guarantee that (submit-task channel (lambda () (push-queue :first))) (push-queue :second) (eq :first (pop-queue)) (eq :second (pop-queue)) ?
10:43:06
|3b|
ok, with submit-task, i think reader needs to count tasks and loop on (Receive-result) until it gets that many
10:43:43
phoe
if you have eight processors, then you'll need to (finally (loop repeat 8 do (push-queue :exit)))
10:44:43
|3b|
with submit-task you don't need a specific :exit task, since each line is a separate task, you just need to wait for all of them to finish
10:45:40
|3b|
in either case, if lines are independent and you want more than 1 parser task, you can just add a serial # and sort results at end or something
10:48:08
phoe
in that case program your code to handle the error somehow and return a result anyway.
10:48:30
phoe
and then when you count the results, you can catch the "error happened" results and filter them out.
10:49:33
xificurC
(let ((channel (make-channel)) (x 0)) (loop repeat 1000 do (submit-task channel (lambda () (incf x) :done))) (loop repeat 1000 do (receive-result channel)) x) --> anything between 960 an 1000
10:49:37
|3b|
return success/fail and count in the receive results and you don't have to use atomics
10:50:30
|3b|
it has to read, add and write. another thread could read before the write, and then overwrite the first write
10:50:57
|3b|
most threaded lisps have an atomic incf that only works on specific types of data (since otherwise it would need locks)
10:53:43
xificurC
yeah, so to sum up the only way to make sure is to count the lines that were processed and receive that many results. Counting needs to be done either atomically or with some locking mechanism. So I need to look at sbcl if it provides an atomic increment or go for a lock
10:57:27
xificurC
I'm reading lines from a stream. I can't wait for the stream to be read completely, so I have to fire up the receiver loop before there's anything to receive
10:57:29
|3b|
actually, reading closer, you were using both strategies... you submit a task to add to the queue
10:58:45
xificurC
|3b|: I was using a queue inside because I need to start the receiver loop before reading all lines
10:59:30
shka_
ok, so if it was a file, i would recommend cl-ds line-by-line -> process -> in-parallel pipe
10:59:43
|3b|
(unless you want stats while reading or something, but even then just counting results is probably fast enough to do between lines)
11:00:57
shka_
xificurC: you will need thread to read stream, lparallel:future that will do processing, and lparallel.queue with fixed size to get the result
11:01:02
xificurC
I can't have the 1 loop that is reading the stream both submit and receive because then I'm blocking and processing line by line again
11:01:11
|3b|
(like printing progress report, or writing output if parsed data won't fit in ram or whatever)
11:02:28
shka_
so basicly do the following -> on thread A: read line, create future, push it into fixed size queue
11:04:12
shka_
basicly two independent loops, in two threads, processing on the lparallel worker, and threads are talking to each other with queue
11:04:14
xificurC
shka_: I'm ok with the processing part, not ok how to ensure all lines get processed
11:05:24
shka_
or you can handle error in the worker, and return tuple of results, where first value is processing result, and the second is the processing status
11:06:07
shka_
don't have time right now to demonstrate how this works, but maybe in the evening if you want to
11:06:12
|3b|
looking at code some more, you are probably losing results due to lack of atomics/locks too
11:07:05
shka_
anyway, this approach maintains upper memory usage, does not require explicit locks on your side and usually works good enough