freenode/#lisp - IRC Chatlog
Search
13:14:35
gendl
shka_: also check uiop. One of uiop's stated goals is to obsolete little utility libraries like cl-fad.
13:26:00
Demosthenex
so i'm trying to make my restful api downloader multithreaded with lparallel, i've got most of it ok except the rate limiter. each rest request return how many requests against the limit in the headers, and i was calculating a delay based on the returned value which is local for threads, but now i'm hitting the rate limit constantly.
13:27:46
Demosthenex
though maybe there isn't much point, given i can hit the rate limit without threads, i spend most of my time waiting.
13:28:03
|3b|
if just a shared number is enough (and you can find a portability lib or limit implementations), atomics, otherwise a mutex and shared variable
13:28:41
|3b|
yeah, if your rate limit is less than processing rate, threads just give you a bit higher burst rate
13:32:04
Demosthenex
given that i was calculating a sleep from the response header, i thought i was safe :P
13:33:08
xificurC
getting number of CPUs, do I really need an external library like - https://github.com/muyinliu/cl-cpus/
13:37:04
|3b|
ouch, ircbrowse hasn't updated since april, guess that just leaves http://log.irc.tymoon.eu/freenode/lisp
13:41:48
AeroNotix
xificurC: in the SBCL source they use the exact same syscall on Linux to achieve this but there are no public interfaces that expose this value directly.
13:42:51
AeroNotix
xificurC: https://gist.github.com/AeroNotix/7179e3715320aafa6034164748e87fa5 here is a file from sbcl that uses this syscall
13:47:14
AeroNotix
xificurC: if you're dead against using a library for this (don't know why you would be realistically, we have lots of RAM these days) then you can c+p the code which sbcl uses?
13:51:11
AeroNotix
xificurC: It's not a bad thing. If sbcl itself provided a function for this your interface to it would be exactly the same
13:53:34
AeroNotix
xificurC: the main "issue" is that simply the standard didn't mandate a function for exposing the number of CPUs on the host OS. So it doesn't exist in implementations. That library provides that function.
13:53:44
AeroNotix
I don't know what else that library should do aside from what it says on the tin :)
13:57:10
AeroNotix
Can someone run this for me: (progn (ql:quickload :stmx) (and (stmx:hw-transaction-supported?) (= stmx.asm:+transaction-started+ (stmx.asm:transaction-begin)))) ?
14:05:40
xificurC
AeroNotix: doing, on 1.4.5. btw that progn doesn't work, the reader barks package does not exist
14:06:44
AeroNotix
xificurC: derp, yeah I already had the package loaded in my repl. Inside the progn that won't work you're right
14:10:55
xificurC
|3b| shka_, I managed to rewrite my code to indeed run in parallel, all cores are running on 100% after the changes. The only problem is that it's giving wrong results :) http://sprunge.us/lk0LeM
14:13:40
shka_
also (lparallel:future (lparallel.queue:push-queue (ignore-errors (parse-line line)) queue))
14:14:32
|3b|
it fills queue with futures, then remaining futures just sit there doing nothing, and nothing adds to queue
14:14:47
Shinmera
shka_: I don't think pathname-utils already has what you wanted, but it has very similar functions.
14:14:53
|3b|
actually i guess you get 100 results (and any more that happen to fit in before read finishes)
14:19:30
xificurC
just remember that add-slowlog and add-groklog modify slowlogs/groklogs by setfing on it
14:20:22
|3b|
because it might not be a new binding each iteration, so could get overwritten by next iteration before it is processed
14:22:35
|3b|
create a new binding every time yourself, so the closure captures that instead of the (possibly) reused loop variable
14:24:31
xificurC
where can I read about this issue? I'd need a more detailed explanation because I didn't understand it from the short description you gave |3b|
14:26:52
xificurC
|3b|: I "know" C. I didn't write much of it but can read it to some extent. I know some assembly too if that helps
14:28:07
|3b|
imagine if your loop allocates a buffer once, then stores a pointer to that buffer each time it sends work to other thread, then next iteration overwrites the buffer, so all the threads are looking at same data
14:28:21
xificurC
shka_: you mean the part about "dynamic variables and worker context"? Is the loop variable dynamic?
14:29:34
|3b|
that's effect you get when loop only creates 1 binding and just assigns to it every iteration. closure saves that binding, so all threads are looking at same thing (though you are more likely to move it into a new binding within the thread than passing around pointers in c, so less likely to see partial updates)
14:29:47
djeis[m]
The loop variable is lexical, it just might be allocated once at the start of the loop and then shared by each future.
14:30:32
|3b|
other option (and only correct one in C) is to allocate a new buffer (and new pointer) every iteration
14:31:11
|3b|
if LOOP is implemented to create a new binding every iteration, then each closure would only see the binding active when the closure was created, giving same effect
14:31:39
|3b|
CL doesn't specify which way LOOP is implemented, so you have to manually create a new binding yourself to make sure
14:32:13
|3b|
(equivalent to manually making a copy of the pointer to pass to other threads in the single-buffer C example)
14:33:37
xificurC
shka_: your implementation keeps order of lines. Forcing each future in this fashion might block the main thread for quite some time, no? Depending on how lparallel handles the threads there might be more stalls. Although the main thread doesn't do much work so there shouldn't be much time lost
14:34:16
|3b|
(if in CL you reused the same array every iteration, that would actually be closest to the single-buffer C example, and still wouldn't work even with new bindings for each closure, since each binding would still store same array object)
14:34:55
|3b|
xificurC: forcing the future will either parse it in that thread, or get results calculated by a worker thread
14:36:03
|3b|
so it will sometimes duplicate some of the work done by workers if it happens to catch up with them
14:38:54
makomo
hm, does DEFINE-MODIFY-MACRO accepts symbols which name macros as its FUNCTION argument? http://clhs.lisp.se/Body/m_defi_2.htm
14:40:04
makomo
it says "(...) /function/ is applied (...)" but i don't know whether that includes functions only or?
14:40:08
specbot
Constraints on the COMMON-LISP Package for Conforming Programs: http://www.lispworks.com/reference/HyperSpec/Body/11_abab.htm
14:40:50
djeis[m]
ACTION sent a long message: < https://matrix.org/_matrix/media/v1/download/matrix.org/ugKnxdRvPzmLYEuRZwNhMDnb >
14:41:06
makomo
jackdaniel: i mean, what part of that page hints that /function/ could name a macro as well?
14:41:50
makomo
jackdaniel: i guess one could infer it from "the expansion of a define-modify-macro is equivalent to the following: (...)" but i always end up wondering if they thought about all of the implications of the example
14:42:16
xificurC
|3b|: "A future is a promise which is fulfilled in parallel. When a future is created, a parallel task is made from the code passed."
14:42:30
jackdaniel
examples per se are not part of the standard, but this is not an example but purt of the description
14:43:01
xificurC
based on that I would assume a future will be fulfilled by BODY, and forcing means blocking waiting for it. In this case I'm blocking waiting for it's result of pushing on the queue
14:43:02
|3b|
makomo: i don't see anything in the spec requiring it to name anything, function or otherwise :)
14:46:57
|3b|
and maybe print a dot or something to see if it gets stuck, since large enough sleep might make it take too long overall if it doesn't
14:48:02
|3b|
problem i see is that if queue fills up and you try to push onto it, you will block waiting for something to pop the queue, which happens on the blocked thread
16:59:21
Spaceman77
But where should i look if i want to analyze an image pixel-by-pixel and apply filters
17:01:18
Spaceman77
I intend to create a `simultaneous localization and mapping` system that uses only 1 or 2 cameras. I try to create 3D maps from these images and determine the position of the robot
17:04:18
dim
you might like clasp, an experimental CL compiler that uses LLVM so that you can build apps in both C++ and CL
17:06:15
Spaceman77
I have some experience in programming, but on the other hand, Lisp truly looks like it's `the way aliens program`
17:06:48
dim
maybe the cando docker image is usable (https://hub.docker.com/r/drmeister/cando/), do you know Shinmera?
17:07:52
Shinmera
dim: anyway the problem is primarily that a lot of lisp libraries don't run on clasp yet
17:07:54
dim
Spaceman77: it got me about a week of everyday programming to get past the parens/syntax and the basics stuff, including the *whole* syntax and a first approach at the standard api
17:08:29
djeis[m]
Spaceman77: You actually get used to the syntax fairly quickly, and after that it’s not that much weirder than any other language for the usual stuff. The difference is the rabbit hole can go much deeper :)
17:09:13
Spaceman77
shrdlu68: Nothing and everything. I've never seen a single language being praised so much by some high profile people
17:11:40
djeis[m]
Spaceman77: well, every language you learn will help with that, especially languages that work differently than you’re used to. Lisp is definitely a good choice for expanding your horizons tho.
17:12:24
Spaceman77
I like programming. I like solving problems and thinking about abstractions and whatnot until what i have is the most elegant and readable solution.
17:12:46
dim
Spaceman77: a very nice intro to the language is found in the PAIP book, see https://github.com/norvig/paip-lisp
17:13:13
dim
seems like https://github.com/norvig/paip-lisp/blob/master/docs/chapter1.md is readable directly
17:15:38
Spaceman77
I want to learn lisp. I am studying and intend to work in robotics, and it seems like such a "doomed" field. Every problem is almost impossible to solve. There is no true AI, sensors lie etc. etc. etc.
17:17:48
dim
the more you dig into the details, the more it seems impossible that anything works, usually saying that a domain is more complex than another one only means that you've been digging more in that first domain ;-)
17:23:15
Spaceman77
I just want to tackle some impossible problems. Is Lisp an adequate language to tackle impossible problems?
17:24:06
pjb
Spaceman77: AFAIK, it's the only one. (well, perhaps ometa too, but I'm cheating here).
17:24:37
aeth
Spaceman77: What you get in Lisp is being able to trivially do anything at compile time that you can do at run time.
17:25:01
aeth
I would say "Well, within reason." but you can even get input in the user at compile time, but then you basically turn compile time into run time.
17:26:35
Spaceman77
Oh, this is curious. I've heard of this wacky stuff you can do with Lisp. Is it true that Lisp can interpret itself and change itself?
17:26:57
pjb
(defparameter *version* #.(progn (format *query-io* "What version are we compiling? ") (finish-output *query-io*) (read-line *query-io*)))
17:29:15
dlowe
Spaceman77: Well, it means that we don't have to have another language when we want to write code that operates on code
17:29:51
dim
Spaceman77: e.g. in pgloader I benefit from that by parsing the command language that the users give me into a lisp program, that I compile at run-time and then execute
17:30:19
dlowe
Spaceman77: the source code is made of nested lists. The text files that are parsed into nested lists are a convenience for humans.
17:30:33
|3b|
Spaceman77: are you familiar with c++ template metaprogramming? if so, imagine being able to use a sane programming language instead of template hacks
17:30:35
dim
if you've been doing some C/C++ before, imagine that the preprocessor would be in C/C++ rather than this #define pseudo-language
17:30:36
dlowe
Spaceman77: therefore, manipulating the soruce code programmatically is manipulating nested lists
17:34:30
Spaceman77
but "The Book" i keep stumbling upon whenever i see people discussing learning lisp is SICP
17:34:53
dim
Spaceman77: I think PAIP first chapter is a good start, and you can read it online at https://github.com/norvig/paip-lisp/blob/master/docs/chapter1.md
17:41:08
dlowe
yeah, but if you made an assembler, it's possible that even interpreted implementations would let you jump into it
17:45:25
Shinmera
anyway, for sbcl, consider https://www.pvk.ca/Blog/2014/03/15/sbcl-the-ultimate-assembly-code-breadboard/
17:45:49
Bike
putting assembly into different languages is kind of a pain. you know how gcc does it, right? this weird shit with strings and dependencies? not great
17:46:30
dlowe
it looks like https://github.com/sile/cl-asm will do x86 assembly and will execute it in an image with sbcl
17:46:37
AeroNotix
Kind of wish the VOP stuff was more documented with a few more examples/tutorials
17:47:50
pfdietz
You don't want it to (just) be documented, you want it to be specified and formally delivered, so it won't change out from under you.
17:48:41
Shinmera
anyhow, might be easier to just write the asm separately, compile to a shared object and then use cffi
17:49:04
AeroNotix
just would like to see how far it's possible to go within just CL itself rather than resorting to that
17:52:48
Shinmera
well, since you're adding a new operator to the compiler, it needs quite a bit of information to know how and when to use it.
17:53:16
_death
played with sb-assem a while ago.. https://gist.github.com/death/5ec259ef473b982898a3c5e36b21b1cd
17:54:59
Shinmera
I did some experiments with the ssa stuff but never took it beyond experiments https://github.com/Shinmera/3d-vectors/blob/master/ssa.lisp
17:57:02
_death
what does "ssa" mean? to me it flashes up as "single static assignment", but here it looks like "sse"
17:59:21
AeroNotix
https://www.pvk.ca/Blog/2014/08/16/how-to-define-new-intrinsics-in-sbcl/ <- useful
18:12:46
phoe
AeroNotix: https://www.pvk.ca/Blog/2014/03/15/sbcl-the-ultimate-assembly-code-breadboard/ <- same author, other interesting post