freenode/#clasp - IRC Chatlog
Search
11:38:25
kpoeck
to try (ql:quickload :3bgl-shader-example :verbose t) and (3bgl-shader-example:run-example)
11:39:32
kpoeck
I believe that was yesterdays fix from drmeister and another fix from Bike (but can identify which one)
14:25:42
Bike
so i wrote the code so that discrimination works like i've been talking about, not using where tags. it's slightly slower if the generic function call history includes both instances and funcallable instances, but otherwise is faster, and by a higher percentage
14:26:07
Bike
however, it takes like 150% the time to compile, and if i force clos to always compile that means practically speaking a 50% increase in build time
14:53:49
Bike
well, i'll try it with the dtree interpreter, which is what we're actually using anyway. hopefully shouldn't matter since interpreted etc
16:06:58
Bike
it kind of goes both ways, since it takes more \time to start the image and start compiling
16:16:22
drmeister
We do the binary search - can we reduce it to a jump table or whatever llvm switch compilation is doing?
16:16:46
drmeister
llvm needs to solve the general case. We know a lot about what we need to compile.
16:19:27
Bike
the overall goal here is just to make the discriminating function code as fast as possible so we can use it for any kind of type differentiation. subgoal is using llvm switches and, necessarily, making stamps contiguous so llvm can merge them.
16:19:35
drmeister
What does "generic function call history includes both instances and funcallable instances" mean? When do we include instances and funcallable instances?
16:20:25
Bike
like if you have a function that's called with both a generic function and a HIR instruction as n argument
16:22:34
drmeister
I'm with you on making GF dispatch as fast as possible so we can use it for type differentiation
16:30:23
Bike
i'm also not totally sure this compile time thing is actually why build time is slower. i'm guessing based on smaller metrics
16:30:30
drmeister
Can you dump what a GF dispatcher looks like - the llvm-IR for a general example?
16:32:18
Bike
i moved to using llvm switches instead of building our own binary search since hey, llvm should be specialized for that, but it's super slow i guess? i don't understand what it could possibly be doing that's slower than our own building up a search tree, our process conses a ton
16:36:45
Bike
http://ix.io/2nju here's the disassembly for a function that returns T if it's given an instance of one of a couple disparate types, and otherwise nil
16:37:34
Bike
something like (lambda (x) (typep x '(or cons core:hash-table-eq standard-generic-function static-gfs::constructor-cell standard-method-standard-class clos:standard-effective-slot-definition)))
16:39:44
Bike
first it checks for the cons tag, then the general tag, then it checks the header against Instance_O, FuncallableInstance_O, and HashTableEq_O, and then it checks against the class stamps in the first two cases.
16:55:18
Bike
actually, let me double check, i might still have that off since your closurette thing
16:56:46
drmeister
I've added a lot of complicated stupid code over the years to try and dump the JITted modules.
16:57:05
drmeister
https://github.com/clasp-developers/clasp/blob/master/src/lisp/kernel/cmp/jit-setup.lsp#L731
16:58:44
Bike
also if we sorted the cases that would just mean we spend time on sorting instead of llvm, so i doubt that would make an impact
16:58:44
drmeister
I was about to say that we should figure out an API to capture JITted modules so we can examine them.
16:59:09
drmeister
Ideally DISASSEMBLE would give us a module that is identical to what gets compiled.
16:59:55
Bike
really, that's why i don't get the extreme impact on build time, the effect on compile time doesn't even seem that bad. maybe it gets exponentially worse with more cases, i didn't look too closely
17:00:05
drmeister
I have this stupid (quick-module-dump <whatever>) and I've forgotten how to use it.
17:00:15
Bike
i could just compile-file this instead of you want. i don't want to go down some jit based rabbit hole.
17:02:35
Bike
i've broken things up so that we can just use a "discriminate" macro. since that's what we want for type tests. so there's no methods involved, it just returns t or nil.
17:02:39
selwyn
what's the status of demo-clasp-cxx-interop on linux? has quickclasp been working ok?
17:02:41
drmeister
We should also profile the compilation of discriminating functions and see where the time is being spent.
17:04:32
kpoeck
I can confirm that demo-clasp-cxx on macosx even runs on machines not installed by drmeister :-)
17:04:44
drmeister
I did notice that usocket stopped working because it couldn't find the 'sb-bsd-socket' system. Clasp provides it internally - so it's something that I "solved" by adding...
17:04:46
drmeister
(asdf:register-immutable-system :sb-bsd-sockets) ; already provided by the system
17:05:48
drmeister
Right now demo-clasp-cxx is kind of a hack. With the coming JITLink it will be better.
17:06:27
drmeister
Currently the problem is you can only load the binding code once. You have to shutdown clasp and start it again if you want to reload.
17:17:08
drmeister
Yeah - it's because static initializers don't work yet on Linux. I'm told they will soon.
17:17:57
drmeister
Without static initializers I am forced to expose an external linkage symbol to initialize the bindings and the external symbol needs to be hard coded into the Lisp code that loads the bindings.
17:18:58
drmeister
With static initializers everything can be done with internal binding symbols and there is no external symbols and no danger of symbol collisions.
17:20:27
drmeister
Hmm, so what's wrong with the DISASSEMBLE output then? When I delete the offending line from the IR (it's not referenced anywhere else) the error moves to the next line.
17:20:46
drmeister
That makes me suspect the problem isn't the line that it says it is. I don't see anything wrong with the line either.
17:21:13
drmeister
I've deleted four globals now (they are not referenced from the code - so it's ok to do that). Now I get...
17:22:37
drmeister
Side note/question: I think someone told me what these were before but I forgot - what are these?
17:24:44
drmeister
Let's ponder this for a moment... https://usercontent.irccloud-cdn.com/file/yKr1SfA4/TEST%5ECOMMON-LISP-USER%5EFN%5E%5E.pdf
17:25:31
drmeister
1. Ok, dumping register arguments - we should do something about that under control of DECLARE.
17:28:16
drmeister
kpoeck: Did you do anything to tell ASDF about sb-bsd-sockets? Clasp puts it in *modules* but I still had problems with it being missing until I registered it with asdf as an immutable system.
17:32:02
Bike
this is post optimization, i think, since we don't actually generate a switch for the tag
17:37:24
Bike
that code is the same used by the existing discriminating function thing, so it's not the problem
17:38:34
drmeister
I'm thinking it's more llvm-IR instructions that need to be compiled. The generated code hopefully ends up as a single displaced read but we could eliminate two llvm-IR instructions if you use a GEP-9
17:39:33
Bike
that's what the core::header-stamp special operator is. irc-read-tagged-general-header
17:40:00
Bike
if i force fastgf to always use the compiler with the discriminating function code in master, build takes about 23 minutes
17:40:26
Bike
the part you're talking about is the same before and after, so i'm not really concerned with it right now
17:41:38
drmeister
I understand that it's the same before and after - I mean for better performance all around - let's change irc-header-stamp to use a GEP-9.
17:42:18
drmeister
Can you compile discriminating functions the old way and the new way in a loop and time it?
17:43:14
drmeister
Because if it is then it's an even bigger difference because half the time is spent compiling C++.
17:44:08
drmeister
Get a branch built that uses the old approach. Then compile some discriminators in a loop and let's profile it.
17:44:15
Bike
compiling something that checks (or standard-generic-function cleavir-ir:enter-instruction) with the master code 100 times takes, say, 1.8s
17:47:46
drmeister
I can do it from the console if you like. Just put it in two separate directories and I'll profile and show the results.
17:48:30
drmeister
The only challenge for you is getting the extra terminals open and then displaying the svg files.
17:58:57
drmeister
selwyn: What do I do to remove a system from quickclasp. edit something something run something something?
18:01:10
selwyn
and then cd /home/selwyn && ./rebuild-dist to make a new version of the distribution
18:04:26
selwyn
executing it as sudo may not load the correct .sbclrc file which is where i usually load quicklisp
18:06:21
selwyn
http://thirdlaw.tech/quickclasp/quickclasp.txt will always contain information about the most recent dist
18:08:29
selwyn
and then, (ql:update-all-dists) on the client to pull the latest distribution, it won't update automatically
18:19:56
Bike
alright compile times are different now but the one is still slower, so i guess it's fine
18:29:04
drmeister
Can you dump the long backtrace? It's a brittle tool with multiple failure modes.
18:31:47
Bike
you wanna do it? i can just set it running and you can do it. i can't view svgs anyway.
18:32:43
drmeister
Do you have this in your wscript.conf? DEBUG_OPTIONS = [... "DEBUG_JIT_LOG_SYMBOLS", ...]
18:43:41
drmeister
You said "alright compile times are different now but the one is still slower, so i guess it's fine" - I interpreted that as the difference in speed was less noticible.
19:06:42
kpoeck
decribed in https://github.com/clasp-developers/clasp/wiki/Using-the-profiling-tools
19:07:33
Bike
still don't really have any way to view it. i could scp it back but there's annoying to do. hm
19:12:37
Bike
cannot see shit. maybe someday i'll be able to use a computer effectively, but not today
19:17:58
drmeister
If it didn't cut off the tips it would take forever to render. It's been a problem profiling the compiler.