freenode/#clasp - IRC Chatlog
Search
16:32:58
attila_lendvai
I would like to retain the ability to bootstrap off of clang, and the longer I wait, the more probable that you will eventually jump the sharks and start using clasp.exe for further development. I want to come up with something before that... :)
16:45:36
drmeister
karlosz: Type (1) 'make' (2) a bunch of stuff happens really fast (3) you have a working clisp executable.
16:50:20
drmeister
Just so that I have this absolutely clear. You compile cleavir with clisp and then compile cleavir with that. Then you can run the result of that and compile cleavir again?
16:51:43
drmeister
Because we are comparing cleavir generated clisp bytecode to cleavir generated llvm-ir lowered to native code.
16:52:51
karlosz
like, the speed gain you have from having better generated code is a microoptimization compared to not having to do llvm optimizations
16:53:54
karlosz
but that doesnt impact the performance much either, since all ssa optimizations run in linear time
16:54:07
drmeister
Well, we've been talking about doing that. Adding your hir/mir->ssa conversion and then lowering ssa to llvm-ir. It would be less work for llvm.
16:56:19
drmeister
We can profile Common Lisp generated code alongside C++/C code. Nothing has stood out.
16:57:15
drmeister
I've had a nagging feeling for years that something is wrong hidden in the runtime. This might help us suss it out if it's really there.
16:57:48
karlosz
right. to be absolutely certain, the way im self compiling is to do (dolist (system '(cleavir-generate-ast cleavir-ast-to-hir sicl-boot cleavir-dominance))
17:07:50
drmeister
I have a recent asdf - but it has a single asdf/build/asdf.lisp and uiop is included in that - isn't it?
17:08:29
drmeister
I commented out the uiop.lisp load and use /Users/meister/Development/cando/src/lisp/modules/asdf/build/asdf.lis
17:11:37
drmeister
Sorry for the basic questions - I know Clasp - other lisps with asfd/quicklisp - I'm a babe in the woods.
17:17:24
karlosz
do this: https://github.com/quicklisp/quicklisp-client/commit/957e342a455fee4f8e708c66b8cf25825b19e280
17:25:57
drmeister
karlosz: LOAD: A file with name /Users/meister/Development/clisp/src/cleavir/ssa-constant-prop.lisp does not exist
17:43:34
drmeister
Ok, I did it twice - how do I know that the second time it's cleavir generated bytecode that is building everything?
17:45:13
karlosz
also, for me at least, once cleavir starts compiling itself i just get a bunch of expected at most nil warnings
17:54:45
drmeister
If I then want to compile-file some test file using cleavir - is there anything I need to tell compile-file to use cleavir?
17:58:01
karlosz
so i need that for genereate ast, which doesnt accept (FUNCTION ... ...) with two arguments
17:59:22
karlosz
in check-special-syntax.lisp in generate ast, #-cleavir goes above the method that specializes on (eql 'function)
17:59:45
drmeister
https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/environment-query.lisp#L3
18:00:05
karlosz
https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/check-special-form-syntax.lisp
18:00:43
drmeister
Here? https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/check-special-form-syntax.lisp#L66
18:03:48
karlosz
beach: check-special-form-syntax in generate-ast doesnt accept a system parameter like convert-special does. this ends up being a problem for implementations that let function take two arguments, for example. is it worth changing now?
18:16:40
drmeister
There is a lot of output being generated - it's slowing things down. Is this expected?
18:18:42
karlosz
weird. i get that much output and it can still do it in about a minute and a half for me
18:18:58
karlosz
im not quite sure whats causng it. if it really were miscompiling, it would have segfaulted by now
18:19:55
karlosz
is your terminal slow or something? i do this in slime in emacs and it can cope fine
18:34:48
drmeister
Back in 2015 I posted this: https://drmeister.wordpress.com/2015/07/30/timing-data-comparing-cclasp-to-c-sbcl-and-python/
18:36:04
drmeister
10,000,000 calculations of the 78th Fibonacci number. --> 2.9 seconds with Clasp on an older laptop.
18:37:28
drmeister
We've been messing around with things - we will have to take a look at it though.
18:59:34
drmeister
I'll be very surprised if there has been a significant drop in the efficiency of the code since last year. The major changes that do slow things down are being corrected using HIR inlining by Bike and aren't even active at the moment.
19:02:08
drmeister
After running that script when I use compile-file - it's cleavir compiled code running cleavir. I want to compare timing between cleavir+clasp and cleavir+clisp
19:05:12
drmeister
Here's how I think things will shake out. For this microbenchmark clasp's generated code is at least 25x faster than cleavir+clisp's that's native code vs bytecode.
19:06:00
drmeister
For compilation however, clasp is much, much slower than cleavir+clisp. Right now I have no idea why.
19:07:26
drmeister
Right - but why the compilers are so different in speed is a mystery to me. Clasp is running with native code everywhere.
19:09:45
karlosz
so the generated code probably isnt that different in speed when comparing lots of generic fucntion dispatch style stuff
19:10:48
drmeister
Your code is doing lots of generic function dispatch as well - exactly as much as Clasp's
19:11:50
karlosz
right, but i mean i dont think native code vs bytecode matters that much when it comes to those types of things
19:16:19
drmeister
Oh - nope - wrong common lisp - I compared ours to sbcl, ecl and ccl - not clisp.
19:31:19
karlosz
i might as well see if i can squash the warnings while im at it, but at least the numeric code is correct now
19:52:02
karlosz
the latest fixes your fib function but im in the middle of squashing the warnings right now
20:04:09
karlosz
drmeister: if you pull the latest and write a #-cleavir here: https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/convert-form.lisp#L205
20:31:02
drmeister
kpoeck: Thanks - there is a switch in wscript called DEBUG_CCLASP_LISP - if you comment that out it won't force generation of debug frames for every function.
20:31:41
drmeister
A debug frame means a call on entry to the function and another on exit to push and pop a shadow stack frame.
20:46:36
drmeister
https://usercontent.irccloud-cdn.com/file/BUyEX5K4/cfg.FIBN%5ECOMMON-LISP-USER%5EFN%5E%5E.dot.pdf
21:03:49
Bike
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/lsp/evalmacros.lsp#L131-L132 well that's still there
21:05:32
Bike
oh, it does have one? maybe defun inline hook is just messed up during build or something.
21:06:35
drmeister
This would explain why sometimes incremental build breaks with errors about inlined-two-arg-+ being missing.
21:07:00
drmeister
I didn't realize that inlined-two-arg-+ was even tied to a function - I thought it was just a symbol.
21:08:09
Bike
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/clasp-builder.lsp#L573-L576 there's always this crap
21:09:49
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cleavir/inline-prep.lisp#L130
21:13:32
drmeister
I see - inlined-two-arg-+ HAS to have a definition - at least once otherwise there would be no AST.
21:14:32
drmeister
Or set the fdefinition of inlined-two-arg-+ to a function that generates an error?
21:15:25
drmeister
I don't see a reason why it's broken at the moment. It's clearly being defined and so the AST should be generated - wth?
21:17:03
drmeister
On another topic - karlosz says that he has cleavir compiling cleavir and I have a copy of it.
21:17:56
drmeister
In this microbenchmark - even without inlining clasp generated code is 7x faster.
21:19:40
drmeister
There are a lot of warnings - he squelched most of them - but I'm running in a macOS terminal so that it's hopefully not such a big deal.
21:29:18
drmeister
After that running: (time (let ((sys::*load-compiling* t)) (load "cleavir/load.lisp")))
21:30:45
karlosz
otherwise it would just be loading those files with the clisp interpreter again and again
21:41:30
karlosz
not sure how much is there but that includes stuff like alexandria, eclector, etc...
22:25:11
drmeister
In clasp (asdf:load-system :alexandria :force t) returns immediately after the first time I run it.
22:29:00
karlosz
thats about the time i had for clisp+cleavir before i switched my basic block representation
22:32:58
drmeister
If the "LLVM time" and "clang link time" is accurate (and I'm not sure they are) then 37.3 seconds is spent in llvm/clang
22:35:10
drmeister
So Clasp running cleavir is about 3x slower than clisp running cleavir and it uses about 2.5x the memory.
22:36:45
drmeister
I need to spend some time looking at how llvm time and clang link time are calculated.
22:45:47
drmeister
Would it be interesting to add a facility to TRACE that reports time and memory allocation?
22:46:53
Bike
you mean in the course of the call? that's kind of weird, but i guess it could be interesting
22:48:19
Bike
maybe start with an ability for TIME to do detailed memory breakdowns and use it in trace if we want
22:54:06
drmeister
Sounds like it could use an API for implementation dependent memory allocation tracking.
23:17:49
karlosz
apparently there is also a JIT compiler in clisp that takes bytecode and turns it into machine code on the fly
23:18:04
karlosz
no idea how that works, though it might be worth investigating to compare with clasp more fairly
23:47:40
drmeister
'clang link time' is acurate - it's the time spent in the system call that runs the clang link
23:50:56
Bike
another thing is that clisp cleavir cleavir's 15 seconds for alexandria is still super slow
23:52:56
Bike
karlosz set up cleavir to run in it, so we're comparing how well it does against clasp.
0:16:00
drmeister
I moved the code that keeps track of llvm time out of C++ into Common Lisp - so I'm a bit more certain of what it is tracking.
0:27:40
drmeister
Bike: Can cleavir running in sbcl do this yet? Compile itself and then compile itself again?
0:40:22
drmeister
As we discussed - I added a slot to classes REF_CLASS_ALLOCATION_COUNTER - its for optionally tracking how many instances of that class are allocated.
0:41:41
drmeister
It's a slot that will store a fixnum and be incremented atomically by the allocation machinery.
0:43:40
Bike
though something more functional like atomic incf of specifically a slot would be nice.
1:03:19
drmeister
Hmm, from reading a bit it sounds like a bad idea to try to cast non-atomic values to atomic ones.
1:07:43
drmeister
I think it's possible - but it's dangerous because you might read the slot non-atomically
1:12:01
drmeister
Nevermind the spin lock - I'm just going to be disciplined wrt the allocation counter.
1:41:02
drmeister
There are issues with alignment and aliasing that makes it look like a bad idea to try to cast non atomic data to atomic data.
1:44:42
drmeister
Ideally we want these counters to be thread-local. We don't want counts from other threads adding in - that would wreck it.
1:45:34
drmeister
Yeah - maybe in something like the thread-local bindings for special variables. Indexed by stamp.
1:51:38
drmeister
But every object allocation would also allocate a CONS cell and the CAR would point to the object.
1:52:14
drmeister
Then you can walk the list to figure out what was allocated - you'd have the order as well.