freenode/#clasp - IRC Chatlog
Search
16:06:48
attila_lendvai
drmeister: I'm working towards my staged bootstrapping idea. I already have stage-1 (iclasp) building, and now I'm reducing the codebase for the stage-2 branch (aclasp). I'm envisioning that these early stages will be much simpler, with much fewer files. and the build scripts will also be much simpler, e.g. all of the stage_char complexity can go away from them.
16:09:13
attila_lendvai
drmeister: it'll be a long ride, but I have a rather exciting vision. everything will go into my own repo until its status/demo gets you excited, too... :)
16:12:50
attila_lendvai
it will have its costs, e.g. 3-4 potentially diverging codebases, but if things work out as I expect them to, then touching the earlier stages will not be necessary too often, and in return it will make bootstrapping much more understandable and debuggable.
16:14:22
attila_lendvai
there'll be a "toplevel" bootstrap.sh or equivalent that checks out and builds the stages under build/, and the toplevel dir will be used to build the currently latest stage
16:15:03
attila_lendvai
I'm also considering how to introduce two "starting points" for the bootstrap, e.g. another one started off of sbcl+cleavir.
16:16:28
attila_lendvai
I should have a much more detailed understanding of the bootstrap process, the lack of which will bite me a few times, but... how else to learn its intimate details?
16:17:49
drmeister
The sbcl+cleavir one Bike and I have talked about a lot - it requires a lot of clasp runtime to be replicated in sbcl.
16:18:28
drmeister
I think if you do that one - that might be the only one to focus on. If you could compile the cclasp code from sbcl+cleavir then why do anything else?
16:18:46
attila_lendvai
I won't attempt it, just thinking ahead to set things up so that we can accommodate for that
16:19:40
attila_lendvai
drmeister: for the beauty of it? of being able to bootstrap the same system based off of two distinct languages and ending up with reproducible builds (same output executable from both base languages)
16:19:47
drmeister
We were thinking that sbcl+cleavir would compile to a list of generic function calls that would be run in the iclasp-boehm interpreter to invoke the llvm functions to generate code.
16:21:07
drmeister
I'm all for beauty - but I don't want anyone wasting time on something that won't be used. There aren't a lot of us.
16:22:25
drmeister
I don't have a lot of energy to talk anyone out of anything. But please, really think hard about spending a lot of time on something that won't be useful.
16:23:18
drmeister
Potentially diverging codebases makes me anxious. However, we are bumping into constraints wrt what we can optimize because of the way clasp builds.
16:24:55
attila_lendvai
it's also a bit of a security concern, although somewhat far fetched... see the http://wiki.c2.com/?TheKenThompsonHack (a trojan in gcc that reproduces itself when bootstrapping gcc from sources with a gcc.exe, and also "miscompiles" the sources of login to implement a backdoor)
16:25:21
drmeister
How about you talk with Bike about this more? He has thought about the sbcl+cleavir compilation a lot. If you two could come up with a solid plan for a better build system that (1) is faster and (2) doesn't constrain optimizations - we could support you to do it.
16:26:41
attila_lendvai
drmeister: my hope is that we will be able to easily and cheaply "abandon" stages and "escape forward", which will make the diverging codebases less of an issue (unless we manage to introduce a bug that manifests several stages later, but I'm not sure I could even create one if I was asked to)
16:27:20
drmeister
Yeah - I'm aware of the compiler trojan issue - I worry about it like I worry about malevolent AI. All I say is - just don't doesn't mess with my molecules.
16:28:06
attila_lendvai
I'll keep you guys updated here, but first I need to have a proof of concept that can rebuild clasp as it is currently to convince myself that this is viable and worth it
16:28:18
drmeister
Well, it's more of a concern. We could use cclasp to build cclasp right now - we don't because we are worried about the compiler spinning out of control with a bug.
16:32:58
attila_lendvai
I would like to retain the ability to bootstrap off of clang, and the longer I wait, the more probable that you will eventually jump the sharks and start using clasp.exe for further development. I want to come up with something before that... :)
16:45:36
drmeister
karlosz: Type (1) 'make' (2) a bunch of stuff happens really fast (3) you have a working clisp executable.
16:50:20
drmeister
Just so that I have this absolutely clear. You compile cleavir with clisp and then compile cleavir with that. Then you can run the result of that and compile cleavir again?
16:51:43
drmeister
Because we are comparing cleavir generated clisp bytecode to cleavir generated llvm-ir lowered to native code.
16:52:51
karlosz
like, the speed gain you have from having better generated code is a microoptimization compared to not having to do llvm optimizations
16:53:54
karlosz
but that doesnt impact the performance much either, since all ssa optimizations run in linear time
16:54:07
drmeister
Well, we've been talking about doing that. Adding your hir/mir->ssa conversion and then lowering ssa to llvm-ir. It would be less work for llvm.
16:56:19
drmeister
We can profile Common Lisp generated code alongside C++/C code. Nothing has stood out.
16:57:15
drmeister
I've had a nagging feeling for years that something is wrong hidden in the runtime. This might help us suss it out if it's really there.
16:57:48
karlosz
right. to be absolutely certain, the way im self compiling is to do (dolist (system '(cleavir-generate-ast cleavir-ast-to-hir sicl-boot cleavir-dominance))
17:07:50
drmeister
I have a recent asdf - but it has a single asdf/build/asdf.lisp and uiop is included in that - isn't it?
17:08:29
drmeister
I commented out the uiop.lisp load and use /Users/meister/Development/cando/src/lisp/modules/asdf/build/asdf.lis
17:11:37
drmeister
Sorry for the basic questions - I know Clasp - other lisps with asfd/quicklisp - I'm a babe in the woods.
17:17:24
karlosz
do this: https://github.com/quicklisp/quicklisp-client/commit/957e342a455fee4f8e708c66b8cf25825b19e280
17:25:57
drmeister
karlosz: LOAD: A file with name /Users/meister/Development/clisp/src/cleavir/ssa-constant-prop.lisp does not exist
17:43:34
drmeister
Ok, I did it twice - how do I know that the second time it's cleavir generated bytecode that is building everything?
17:45:13
karlosz
also, for me at least, once cleavir starts compiling itself i just get a bunch of expected at most nil warnings
17:54:45
drmeister
If I then want to compile-file some test file using cleavir - is there anything I need to tell compile-file to use cleavir?
17:58:01
karlosz
so i need that for genereate ast, which doesnt accept (FUNCTION ... ...) with two arguments
17:59:22
karlosz
in check-special-syntax.lisp in generate ast, #-cleavir goes above the method that specializes on (eql 'function)
17:59:45
drmeister
https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/environment-query.lisp#L3
18:00:05
karlosz
https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/check-special-form-syntax.lisp
18:00:43
drmeister
Here? https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/check-special-form-syntax.lisp#L66
18:03:48
karlosz
beach: check-special-form-syntax in generate-ast doesnt accept a system parameter like convert-special does. this ends up being a problem for implementations that let function take two arguments, for example. is it worth changing now?
18:16:40
drmeister
There is a lot of output being generated - it's slowing things down. Is this expected?
18:18:42
karlosz
weird. i get that much output and it can still do it in about a minute and a half for me
18:18:58
karlosz
im not quite sure whats causng it. if it really were miscompiling, it would have segfaulted by now
18:19:55
karlosz
is your terminal slow or something? i do this in slime in emacs and it can cope fine
18:34:48
drmeister
Back in 2015 I posted this: https://drmeister.wordpress.com/2015/07/30/timing-data-comparing-cclasp-to-c-sbcl-and-python/
18:36:04
drmeister
10,000,000 calculations of the 78th Fibonacci number. --> 2.9 seconds with Clasp on an older laptop.
18:37:28
drmeister
We've been messing around with things - we will have to take a look at it though.
18:59:34
drmeister
I'll be very surprised if there has been a significant drop in the efficiency of the code since last year. The major changes that do slow things down are being corrected using HIR inlining by Bike and aren't even active at the moment.
19:02:08
drmeister
After running that script when I use compile-file - it's cleavir compiled code running cleavir. I want to compare timing between cleavir+clasp and cleavir+clisp
19:05:12
drmeister
Here's how I think things will shake out. For this microbenchmark clasp's generated code is at least 25x faster than cleavir+clisp's that's native code vs bytecode.
19:06:00
drmeister
For compilation however, clasp is much, much slower than cleavir+clisp. Right now I have no idea why.
19:07:26
drmeister
Right - but why the compilers are so different in speed is a mystery to me. Clasp is running with native code everywhere.
19:09:45
karlosz
so the generated code probably isnt that different in speed when comparing lots of generic fucntion dispatch style stuff
19:10:48
drmeister
Your code is doing lots of generic function dispatch as well - exactly as much as Clasp's
19:11:50
karlosz
right, but i mean i dont think native code vs bytecode matters that much when it comes to those types of things
19:16:19
drmeister
Oh - nope - wrong common lisp - I compared ours to sbcl, ecl and ccl - not clisp.
19:31:19
karlosz
i might as well see if i can squash the warnings while im at it, but at least the numeric code is correct now
19:52:02
karlosz
the latest fixes your fib function but im in the middle of squashing the warnings right now
20:04:09
karlosz
drmeister: if you pull the latest and write a #-cleavir here: https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/convert-form.lisp#L205
20:31:02
drmeister
kpoeck: Thanks - there is a switch in wscript called DEBUG_CCLASP_LISP - if you comment that out it won't force generation of debug frames for every function.
20:31:41
drmeister
A debug frame means a call on entry to the function and another on exit to push and pop a shadow stack frame.
20:46:36
drmeister
https://usercontent.irccloud-cdn.com/file/BUyEX5K4/cfg.FIBN%5ECOMMON-LISP-USER%5EFN%5E%5E.dot.pdf
21:03:49
Bike
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/lsp/evalmacros.lsp#L131-L132 well that's still there
21:05:32
Bike
oh, it does have one? maybe defun inline hook is just messed up during build or something.
21:06:35
drmeister
This would explain why sometimes incremental build breaks with errors about inlined-two-arg-+ being missing.
21:07:00
drmeister
I didn't realize that inlined-two-arg-+ was even tied to a function - I thought it was just a symbol.
21:08:09
Bike
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/clasp-builder.lsp#L573-L576 there's always this crap
21:09:49
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cleavir/inline-prep.lisp#L130
21:13:32
drmeister
I see - inlined-two-arg-+ HAS to have a definition - at least once otherwise there would be no AST.
21:14:32
drmeister
Or set the fdefinition of inlined-two-arg-+ to a function that generates an error?
21:15:25
drmeister
I don't see a reason why it's broken at the moment. It's clearly being defined and so the AST should be generated - wth?
21:17:03
drmeister
On another topic - karlosz says that he has cleavir compiling cleavir and I have a copy of it.
21:17:56
drmeister
In this microbenchmark - even without inlining clasp generated code is 7x faster.
21:19:40
drmeister
There are a lot of warnings - he squelched most of them - but I'm running in a macOS terminal so that it's hopefully not such a big deal.
21:29:18
drmeister
After that running: (time (let ((sys::*load-compiling* t)) (load "cleavir/load.lisp")))
21:30:45
karlosz
otherwise it would just be loading those files with the clisp interpreter again and again
21:41:30
karlosz
not sure how much is there but that includes stuff like alexandria, eclector, etc...
22:25:11
drmeister
In clasp (asdf:load-system :alexandria :force t) returns immediately after the first time I run it.
22:29:00
karlosz
thats about the time i had for clisp+cleavir before i switched my basic block representation
22:32:58
drmeister
If the "LLVM time" and "clang link time" is accurate (and I'm not sure they are) then 37.3 seconds is spent in llvm/clang
22:35:10
drmeister
So Clasp running cleavir is about 3x slower than clisp running cleavir and it uses about 2.5x the memory.
22:36:45
drmeister
I need to spend some time looking at how llvm time and clang link time are calculated.
22:45:47
drmeister
Would it be interesting to add a facility to TRACE that reports time and memory allocation?
22:46:53
Bike
you mean in the course of the call? that's kind of weird, but i guess it could be interesting
22:48:19
Bike
maybe start with an ability for TIME to do detailed memory breakdowns and use it in trace if we want
22:54:06
drmeister
Sounds like it could use an API for implementation dependent memory allocation tracking.
23:17:49
karlosz
apparently there is also a JIT compiler in clisp that takes bytecode and turns it into machine code on the fly
23:18:04
karlosz
no idea how that works, though it might be worth investigating to compare with clasp more fairly
23:47:40
drmeister
'clang link time' is acurate - it's the time spent in the system call that runs the clang link
23:50:56
Bike
another thing is that clisp cleavir cleavir's 15 seconds for alexandria is still super slow
23:52:56
Bike
karlosz set up cleavir to run in it, so we're comparing how well it does against clasp.
0:16:00
drmeister
I moved the code that keeps track of llvm time out of C++ into Common Lisp - so I'm a bit more certain of what it is tracking.
0:27:40
drmeister
Bike: Can cleavir running in sbcl do this yet? Compile itself and then compile itself again?
0:40:22
drmeister
As we discussed - I added a slot to classes REF_CLASS_ALLOCATION_COUNTER - its for optionally tracking how many instances of that class are allocated.
0:41:41
drmeister
It's a slot that will store a fixnum and be incremented atomically by the allocation machinery.
0:43:40
Bike
though something more functional like atomic incf of specifically a slot would be nice.
1:03:19
drmeister
Hmm, from reading a bit it sounds like a bad idea to try to cast non-atomic values to atomic ones.
1:07:43
drmeister
I think it's possible - but it's dangerous because you might read the slot non-atomically
1:12:01
drmeister
Nevermind the spin lock - I'm just going to be disciplined wrt the allocation counter.
1:41:02
drmeister
There are issues with alignment and aliasing that makes it look like a bad idea to try to cast non atomic data to atomic data.
1:44:42
drmeister
Ideally we want these counters to be thread-local. We don't want counts from other threads adding in - that would wreck it.
1:45:34
drmeister
Yeah - maybe in something like the thread-local bindings for special variables. Indexed by stamp.
1:51:38
drmeister
But every object allocation would also allocate a CONS cell and the CAR would point to the object.
1:52:14
drmeister
Then you can walk the list to figure out what was allocated - you'd have the order as well.