freenode/#clasp - IRC Chatlog
Search
21:01:26
kpoeck
can you also try from a repl with (time (compile-file "sys:kernel;lsp;generated-encodings.lsp"))?
21:03:17
drmeister
kpoeck: The buildbot hasn't successfully built clasp for the last day or so - first because generated-encodings.lsp was causing it to timeout and now because of some kind of bordeaux-threads issue? Argh
21:04:06
drmeister
Note - using (compile-file ...) uses compile-file-parallel. The build system uses compile-file-serial
21:05:09
kpoeck
(time (cmp:compile-file-serial "sys:kernel;lsp;generated-encodings.lsp")) takes 6 seconds on my mac
21:05:55
kpoeck
But in order to allow building again. I proposed earlier that I convert generated-encodings to lazy loading at runtime
21:06:38
kpoeck
Would that be fine with you? That should bring back the compile-times to few seconds at the cost of 3 seconds wait for the first use on a non-standard encoding
21:08:47
kpoeck
no: only load the enconding tables when it is first used, so pass the problem to runtime
21:09:56
drmeister
I can't replicate the problem yet on the one machine that really shows the problem because of other noise.
21:14:28
drmeister
This isn't surprising anymore - it's just sad. We need to reduce throwing C++ exceptions.
21:16:45
drmeister
https://github.com/clasp-developers/Eclector/blob/master/code/reader/read-common.lisp#L108
21:17:44
drmeister
It's a return-from - it's nothing fancy. It's a straightforward return-from from an inner function returning from an outer scope.
21:18:31
karlosz
the analysis isn't powerful enough to handle when terminat-token happens in multiple inner functions
21:18:40
drmeister
That's not to put the problem on you. I'm just looking at the code and wondering why contify didn't deal with this.
21:19:44
karlosz
you can declare it explicitly inline. it will cause about 9 copies of the inner function to be duplicated, but it's worth it because return-from is much slower than that
21:21:20
drmeister
What about converting this into a setjmp/longjmp - could we detect that there can be no C++ code between the read-token and terminate-token?
21:22:30
karlosz
because while contify can get rid of a lot of cases, ultimately it can't handle every return-from, because it only applies to single return-site stuff
21:23:18
drmeister
kpoeck: There is something going on with stack unwinding - on some systems it's not too bad on other it's TERRIBLE.
21:24:15
drmeister
It's a massive, unpredictable delay thrown into the code - it makes our lives very difficult.
21:25:27
kpoeck
but perhaps the problem is really (eval-when (:compile-toplevel) (process-encodings-file))
21:26:58
kpoeck
although (time (ext::process-encodings-file)) -> Time real(2.774 secs) on my machine
21:27:45
drmeister
kpoeck: It's not a problem on your machine though. It's a highly variable, machine dependent, operating system dependent, unpredictable cost of C++ stack unwinding that can vary from an insignificant to a MASSIVE delay.
21:28:43
drmeister
Note yesterday we were seeing a 3000 second compile time on the build bot and low double digit (3-20) seconds on my machine and yours.
21:29:44
drmeister
https://github.com/clasp-developers/Eclector/blob/master/code/reader/read-common.lisp#L70
21:31:11
drmeister
Here's another example - I'm building in docker on the iMacPro with 18 cores given to docker. It's sitting for the last half hour compiling one file.
21:37:11
drmeister
With compile-file-serial it is 34.6 seconds. I think I also need to inline read-char-handling eof.
21:38:15
drmeister
It's still terrible performance - now with compile-file-serial it is 33.9 seconds.
21:40:13
drmeister
I'm using C-c k to compile the entire file and then (time (cmp:compile-file-serial "sys:kernel;lsp;generated-encodings.lsp"))
21:42:53
drmeister
Is it missing inlining because of nested functions being created by the compiler?
21:43:03
karlosz
you were right to also inline read-char-handling-eof, but as far as i can tell that should do the trick
21:47:08
karlosz
drmeister: what if you tried putting the inline right after the defun into the body of the labels?
21:48:39
karlosz
there will always be cases where contification and inlining can't remove that, for example if you passed a (lambda () (return-from)) somewhere
21:49:38
karlosz
and if someone declares a function in quicklisp somewhere as explicitly notinline, and there's a return-from, neither contiication nor inlining will ever kick in, so kablooey
21:51:16
karlosz
yes, i think the real problem will be guaranteeing that there is no intervening C++ code
21:52:27
drmeister
Compiling a defun like this you get a big HIR graph with UNWIND-INSTRUCTION in there.
21:52:51
drmeister
We don't have call-with-variable-bound or funwind-protect anymore. There can't be any intervening C++.
21:55:43
drmeister
With call-with-variable-bound and funwind-protect I think it was a problem. But now...
21:58:28
drmeister
Hmm simple counter example - let's say you have (defun foo (x) (some-cxx-function (lambda () (return-from #'foo nil))
22:01:25
karlosz
and it would solve cases like these, where you don't pass random lambda's around, you are only using return-from in a context that setjmp and longjmp work
22:01:50
drmeister
Yeah. Is that the same as analyzing the paths between UNWIND and its destination and checking if there is a closure/FUNCTION between them. I might be talking nonsense here.
22:02:39
karlosz
nope, there's no need to do that. the same machinery that checks whether a function is eligible to be inlined should work
22:03:53
karlosz
we can make this a clasp specific optimization, because return-from is only really a clasp issue
22:03:56
drmeister
What about (defun foo () (cxx-function (lambda () (flet ((bar () (return-from foo nil))) (bar))) ?
22:07:16
karlosz
so, as long as no function that contains the return-from is passed around as a first class value...
22:07:54
karlosz
which, still applies in this case. I think a return-from would have to be enclosed by some kind of local function for C++ to get a handle on it. thinking... for a potential counterexmaple
22:09:22
karlosz
i.e. check all local functions that contain return-from foo are not passed around as first class values
22:13:08
drmeister
Right - think on this. Let's see what Bike thoughts are - he might have thought this through already.
22:14:12
drmeister
With cases like mapc we might be able to flag special functions as being safe in that they don't use CFFI
22:40:59
kpoeck
drmeister could you please check https://github.com/clasp-developers/clasp/pull/1012
22:53:29
karlosz
yeah, there really is no way to contify that terminate-token function, since it's not really possible to have two functions share the same body in HIR or LLVM IR
22:53:57
Bike
there are some lisp special operators that still use intervening C++ functions, like progv and complicated multiple-value-call
22:56:17
karlosz
because, if i understand correctly, you pass first class local functions to progv and m-v-c
22:56:55
Bike
you say "call position", but things like (flet ((foo ...)) (bar (lambda () ... (foo ...)))) have to be ruled out, right?
22:57:22
karlosz
call-position, as in transivitevly including every local function that encloses a RETURN-FROM
22:58:39
Bike
i mean in t his case foo i'd say foo is in "call position" but i don't think we could use set/longjmp
22:59:10
karlosz
yeah, foo is in call-position, but it's "inside" a local function that's not in call-position
23:02:10
karlosz
i guess it's harder for HIR to know what "higher" than a BLOCK means, but should still be possible by analyzing dynenvs
23:02:47
Bike
yeah, i mean, for this case probably all you need to know is that the dynenv of the call is a child of the dynenv of the block
23:03:00
Bike
so like, in (lambda () (foo ...)) it terminates at the lambda's dynamic environment output instead
23:03:55
Bike
and depending how sophisticated it is, either there are no bind or unwind protect environments, or that they're all the same
23:04:32
karlosz
but maybe it blocks too much, since looking at the "terminate-token", case, it would also block that
23:06:19
Bike
hm, well in this case you could go a bit deeper with the analysis. like, terminate-token call dynenvs terminate with read-char-handling-eof's, now how about read-char-handling-eof? and it looks like in that case it does get back to the read-token environment
23:07:27
karlosz
yeah, and you'd need to pair that with the passed-as-first-class-value thing to not conflate it with the example you gave
23:08:06
Bike
i mean yeah you'd also need to know that the only instructions it's fed to are assignments and bla bla funcall as only the callee.
23:09:19
karlosz
i'm thinking it would be pretty straightforward to do this as a pass somewhere in hir or mir, that just change-class's the catch-cont
23:09:56
karlosz
then translate-instruction just does its normal unwind thing in the general case, but setjmp/longjmp if the catch-cont is eligible for it
23:11:34
Bike
it might still be interesting in some cases. for example if you know there are no intervening unwind-protects/binds, in sicl you could skip calling the unwinder function
23:12:06
Bike
anyway, implementing this for the runtime might be annoying. we'll have to figure out what type a jmp_buf has and then probably actually call setjmp and longjmp the C functions
23:13:33
Bike
i mean, even with the C++ unwinding there's no actual unwind instruction, you're supposed to call __cxa_throw or whatever
23:15:37
Bike
quick check on godbolt has jmp_buf has {[8 x i64], i32, %struct.__sigset_t} which is not a five word buffer
23:16:41
Bike
we also might not have the destination address... i think llvm lets you refer to the addresses of code blocks sometimes, but i don't know if that works outside of the jump table thing
23:18:07
karlosz
"For SJLJ based exception handling, this intrinsic forces register saving for the current function and stores the address of the following instruction for use as a destination address by llvm.eh.sjlj.longjmp."
23:21:22
Bike
i'm just a little antsy about using something that's "used internally within LLVM's backend" and is specifically for SJLJ exception handling, which we are not doing
23:28:37
Bike
"returns_twice: This attribute indicates that this function can return twice. The C setjmp is an example of such a function. The compiler disables some optimizations (like tail calls) in the caller of these functions." that's a point i hadn't considered
23:28:57
Bike
i guess you can't reuse the stack space for a tail call since the code after the second return will still want access to the frame
23:32:26
Bike
yeah ok, since returns_twice is the only annotation for this and it's only for functions... yeah, that's difficult, since you'd want the same mark for the equivalent to catch-instruction
23:35:55
Bike
i mean that returns_twice on a function is the only way llvm has to indicate that it can't do tail calls and stuff, as far as i can tell
23:36:15
Bike
so you'd need anything that checks for a returns_twice call to check for this instruction too, i guess?
23:38:16
karlosz
oh, you mean because it's just a bare instruction and not a function that you can annotate return_twice on
1:09:35
yitzi
I removed the line that executed cando in the root user and forced all quicklisp activities to be in the user account
1:12:04
yitzi
I'll have to write a custom installer or tweak cl-jupyters installer since it thinks that cando is clasp.
1:17:16
yitzi
I'm probably done tonight. I'll detangle the dockerfile and fix the kernel installer tomorrow. Just thought you'd like to know that progress is being made.
1:21:09
drmeister
It means I have to go through the whole rigamarole to set up /opt/clasp and set permissions and so on.
1:36:59
karlosz
okay, i coded up a slightly more conservative analysis for when to do setjmp that should be correct
1:38:34
karlosz
Look at every catch-instruction. Look at every unwind that uses that catch-instruction. If every intermediate function of the catch and the unwind are in call-position, there is no problem
1:39:32
karlosz
it's simple and can be improved later, but now the next step is how to actually emit setjmp and longjmp
1:41:59
karlosz
so, anything that is of the class "escaping instruction" should fall back to using c++ unwinding
1:59:20
Bike
i mean, it's the same with unwind-protect, the protection functions need to be executed on the way out
1:59:55
karlosz
i thought we were just handling the cases where we could substitute in setjmp for unwind
2:00:46
Bike
we could just stick to cases with no intervening binds or protects, that should be statically detectable
2:01:08
Bike
doing them in a different dynamic context may be difficult, the way they work now is C++-like and not amenable to it
2:52:56
Bike
also, i fixed up the ast interpreter so that it only maps the ast once, and only converts csts to asts once even when it gives up and uses the compiler
2:53:34
Bike
i'm thinking the "children" relation on ASTs may form an actual tree so we don't need to do the hash table thing for the mapping, though
3:06:30
Bike
but i don't think it's in the "children". maybe it used to be, i remember it being an issue
3:06:54
karlosz
oh, i see.i would have just expected it to be a block tag, rather than the actual full ast itself
3:10:45
Bike
i think i'll look at ensuring the AST is actually a tree, and then this hash table thing can be gone. could speed up a couple different things, really, and i don't see any need to make it not a tree
3:11:23
karlosz
yeah. though its still kinda weird that that is the problem because of how much the compiler uses map instructions with hash tables
3:12:03
Bike
doing some flamegraphs with the new system might be good. it only maps once so the time usage there should be obvious
3:12:06
karlosz
and its not like its the gf dispatch either, because ast->hir basically walks the same thing
3:18:54
Bike
could also save consing by, instead of having the CHILDREN generic function, have a MAP-CHILDREN function that doesn't cons, and derive CHILDREN from it
5:11:56
drmeister
This unwinding problem is biting me bad today - I've lost at least three hours waiting for this docker image with 18 cores! of an iMacPro to compile generated-encodings.lisp.
5:12:30
drmeister
kpoeck: It's not that your code was wrong - it was fine - we have a really bad problem with unwinding that bites us at the worst times.
5:13:12
drmeister
It's been compiling generated-encodings.lsp in this docker image for about an hour now - I'm giving up and going to bed.
5:14:05
drmeister
I don't think yitzi sees these looooooooooooooooong compilations on his docker image.
5:14:24
drmeister
And this same file on the same iMacPro in macOS only takes a few seconds to compile.
7:12:26
beach
If you are new to Lisp, I suggest you learn it first, independently of C++ integration.
7:13:30
beach
I am answering you because most Clasp developers are in the US, and asleep at this time.
7:58:35
beach
You need to wait for the developers to wake up for that. I know very little about C++ integration. I hang out here because Clasp uses the compiler of a project that I started.
7:59:39
beach
But, again, if you plan to use Common Lisp, I strongly suggest you learn it first, independently of C++. Otherwise, you might be very confused.
8:00:00
beach
The semantics of C++ and Common Lisp are very different, so the integration is nontrivial.
8:08:40
minion
kapil_: please see PCL: pcl-book: "Practical Common Lisp", an introduction to Common Lisp by Peter Seibel, available at http://www.gigamonkeys.com/book/ and in dead-tree form from Apress (as of 11 April 2005).