freenode/#clasp - IRC Chatlog
Search
11:18:29
kpoeck
Drmeister: I would like to fix interpret_token_or_throw_reader_error to allow all possible float-types in _STARreadDefaultFloatFormatSTAR
11:23:12
kpoeck
Am now compiling under the assumption that I can read both a DoubleFloat and a Long with strtod
11:24:18
drmeister
No DoubleFloat and LongFloat are the same. I've toyed with the idea of using the GMP infinite precision float for long-float.
11:24:55
kpoeck
Still puzzled with SingleFloat is read with strod, since clasp_make_single_float is defined to accept a float and is passed a double
11:25:46
drmeister
ShortFloat and SingleFloat are also confused - SingleFloat is a tagged C 'float'. I think (but I may have forgotten) ShortFloat is still managed by the GC.
11:29:45
drmeister
Yes. SingleFloat values are not stored on the heap at all - they are coded into the tagged pointer itself. Like fixnum or character.
11:30:15
drmeister
They are shifted a few bits, have tag written into the lower three bits and that's how they are represented.
11:31:38
drmeister
They could be aliased to SingleFloat's... Are there any other ideas of what to do with them?
11:32:18
drmeister
What about LongFloat - I'd like to implement them with GMP arbitrary precision floats.
11:33:55
kpoeck
In ccl, for number crunching I used short-floats instead of single-float, since they would not cons
11:33:58
drmeister
kpoeck: Right - I did that with that in mind - I didn't tidy up ShortFloat and LongFloat because I didn't know what to do with them at the time. So I just left them.
11:36:21
drmeister
Cleavir has a way of doing a lot of math without boxing/unboxing operations that I am looking forward to using.
11:39:32
kpoeck
So I just extend interpret_token_or_throw_reader_error for short-float and long-float (testing right now)
11:40:31
kpoeck
On a different matter, could you perhaps test whether issue #531 also crashes your clasp?
11:42:43
heisig
drmeister: Currently, with the whole machine learning hype, 16bit half-floats have become fashionable again. So that could be a use case for ShortFloat_O.
11:50:58
kpoeck
Meanwhile running ansi-tests with the extended interpret_token_or_throw_reader_error
12:14:56
drmeister
heisig: What do you think about arbitrary precision float's - would it be a good idea to expose them as long-float?
12:35:12
heisig
Bike: This is wikipedia on 16bit floats: https://en.wikipedia.org/wiki/Bfloat16_floating-point_format
12:36:17
Shinmera
"The bfloat16 format is utilized in upcoming Intel AI processors, such as Nervana NNP-L1000, Xeon processors, and Intel FPGAs,[1][2][3] Google Cloud TPUs,[4][5][6] and Tensorflow.[6][7]"
12:36:58
heisig
drmeister: Arbitrary precision long-floats sound like a great idea. I know applications where this would be extremely useful.
12:38:33
drmeister
Ok then. short-float will be 16bit floats - I'll need to tweak the tagging scheme. long-float's will be arbitrary precision floats
12:43:21
drmeister
That is correct - single-floats do not cons - they are stored in tagged pointers.
12:46:32
drmeister
I was going to use them as pointers into aligned C++ memory - but that never panned out. Actually, I'll wait until Martin comes on board - he may have some use for another kind of tagged immediate.
12:47:15
stassats`
i think the only thing clasp will benefit from with 16 bit floats is reduced storage
12:48:09
stassats`
otherwise you'll need clasp to run on these TPUs or support SIMD or wherever these floats are used
12:48:40
drmeister
Well, there is interoperability with C++ machine learning libraries if we need to pass 16bit floats back and forth. Interoperability - as you mentioned above.
12:57:59
drmeister
What's up with a language-server-protocol server for Common Lisp? Does anyone use one?
13:00:02
drmeister
jupyterlab appears to be going in that direction - to use language server protocol.
13:00:44
drmeister
ACTION is rummaging around inside of jupyterlab discussion groups and github issue trackers
13:05:10
attila_lendvai
hi! I could use some insights/pointers... so, iclasp is a CL interpreter written in C++. aclasp is iclasp loading a bunch of files implementing a minimal CL system. what I don't understand is that compile-aclasp seems to be actually compiling stuff. how is that happening? what kind of compilation can iclasp do?
13:06:47
Bike
iclasp is the C++ interpreter. then it loads a bunch of files including a compiler, and compiles itself
13:09:31
drmeister
attila_lendvai: If you start from ./waf distclean configure build_iboehm <-- it builds the C++ interpreter iclasp-boehm
13:10:31
drmeister
Then you can use ./waf build_rboehm <-- This loads the C++ interpreter and has it load the aclasp Common Lisp source code as interpreted code - absolutely no Common Lisp compilation happens up to this point.
13:11:09
drmeister
stassats`: You got me: ./waf distclean configure build_iboehm <-- it builds the C++ interpreter of a subset of Common Lisp programming language: iclasp-boehm
13:12:22
drmeister
So: ./waf build_rboehm puts you in a REPL that loaded the aclasp/bclasp Common Lisp compiler as interpreted code. At this point you can compile-file a subset of Common Lisp code.
13:14:10
drmeister
./waf build_aboehm loads the same Common Lisp source files as interpreted functions and then runs 'compile-file' on all of the Common Lisp source files that ./waf build_rboehm and ./waf build_aboehm load.
13:16:07
drmeister
./waf build_aboehm then links the compile-file'd code into a fasl called clasp/build/boehm/fasl/aclasp-boehm-image.fasl <--- Run with this and you get what you get when you run ./waf build_rboehm but everything is compiled. It provides a subset of Common Lisp and can further be used to compile a complete Common Lisp: 'bclasp'
13:17:52
drmeister
The reason for all of this is that the interpreter written in C++ is an S-expression walking interpreter and it expands macros whenever they are encountered during evaluation. This is really, really slow when you start hitting a lot of macros as in loop and CLOS.
13:35:14
drmeister
As ./waf build_aclasp compile-file's it's own code - it LOAD's the fasl files. This replaces interpreted functions with compiled ones as they are generated. This happens in both serial and parallel build. The code is set up to be tolerant to hot-swapping interpreted code with compiled versions in any order.
13:35:58
attila_lendvai
and aclasp is compile-file'ing how? aclasp is already capable of calling the LLVM machinery at runtime?
13:36:28
drmeister
Yes - everything stage of Clasp building has full access to all of the llvm machinery.
13:37:30
drmeister
If you go iclasp-boehm -I -n (loads absolutely no CL code, either interpreted or compiled).
13:38:43
drmeister
You get a repl and a very restricted subset of the Common Lisp language - but everything that is exposed from C++ is exposed here.
13:41:50
drmeister
Its better to build aclasp with fewer parallel processes than more - because more of the code can then take advantage of the compiled fasls that replaced the interpreted code. I think I currently limit aclasp compilation to 8 parallel processes. I haven't timed it carefully.
13:42:54
attila_lendvai
FTR, it'll be here: https://github.com/clasp-developers/clasp/wiki/The-Build-Process/
13:43:50
attila_lendvai
there's docs/bootstrap.rst. maybe I'll extend that and point to it from the wiki. I'll think about it
13:58:35
attila_lendvai
registerClasses.py and src/common/build is obsoleted by the CL scraper that is run by sbcl and wscript, right? I'll delete anything in the codebase and docs related to it. let me know if that's not desirable!
14:00:08
attila_lendvai
or is the python groveler in a good enough state that it's worth keeping it in some tools-for-build/obsolete/ directory?
14:01:23
drmeister
Working backwards. The python groveler can be gently lowered into a grave and buried. It is dead.
14:03:07
attila_lendvai
drmeister: in the docs, and some of its output can be found checked into the repo (by rgrep registerClasses .)
14:07:18
attila_lendvai
drmeister: but I welcome your impulses for cleaning the codebase! it's very nice for the newcomers, thanks! :)
14:08:34
drmeister
It's probably better to do it slowly and methodically. I don't instantly recognize what is being used and what is not.
14:12:45
attila_lendvai
drmeister: I have the following left in src/common. let me know if any of those should also get "gently lowered into a grave"... :) analyzeLldbBacktrace.py build-json.lisp buildJSON.py copywrite.lsp metering-allocations.lisp peakMemory.cc
14:20:38
attila_lendvai
drmeister: I assume analyzeLldbBacktrace.py can also be disappeared. I moved the rest that seemed useful to tools/ and I'll delete src/common/
14:23:57
drmeister
Actually, can you move analyzeLldbBacktrace.py to tools? I have used that to analyze the size of stack frames to figure out if there are functions that allocate too much on the stack.
14:30:29
attila_lendvai
I've pushed it. let me know if I happened to delete anything that you want restored: https://github.com/clasp-developers/clasp/commit/61e2b47c0299baaf89e24c08b1b4a7159a860c64
15:02:37
karlosz
beach: its not explicitly written in the docs, but we are assuming that for phi functions, the order of the predecessors have to match the order of the inputs, right?
15:03:03
karlosz
some code you wrote seems to assume that, and otherwise we have to do the dominator calculations over and over again to find the corresponding input for each predecessor
15:03:33
karlosz
but things like delete-instruction shuffle the order of the predecessors, like if you deleted the first phi of a cluster
15:04:02
beach
Yes, but that is also the reason why I strongly dislike SSA. It imposes such an order whereas the flow graph itself does not. It ruins everything.
15:04:48
beach
And, you are right, I did not take predecessor order into account when I wrote the code to modify the graph.
15:06:17
karlosz
maybe phi instructions can have a mapping between inputs and predecessors as an extra slot?
15:06:47
beach
I especially dislike SSA because I am convinced (but I have yet to do the research) that basically none of the optimization techniques that claim to need SSA takes advantage of the fact that SSA is really S.S.A.
15:08:38
karlosz
beach: https://en.wikipedia.org/wiki/Sparse_conditional_constant_propagation this is the algorithm that i jjust implemented that requires SSA
15:09:04
karlosz
it runs in linear time with SSA, and i think itd be very diffciult to write without assuming definitions are unique and dominate uses
15:09:34
karlosz
the effects are folding something like https://paste.gnome.org/pp2hrvkia to just values 4 4 1 2 3 4
15:11:09
beach
Like SSA, but put an assignment in each incoming arc of what use to be the Phi instruction.
15:11:37
beach
All I am saying is that SSA ruins everything, and no algorithm I know take advantage of the fact that SSA is S.S.A.
15:12:48
beach
karlosz: But I have yet to do the research like I said. It would be a beautiful result though.
15:13:12
karlosz
i do the transformation you are talking aboutl. to actually implement the phi functions, i convert the phi to assignmnet instruction along each arc before lowering to code
15:13:40
karlosz
its standard though, and still considered SSA in the leiterature, because those phi functions still have to be implemented somehow
15:15:05
beach
karlosz: I have to at least convince myself that SFA is good enough for all the important published optimization techniques.
15:15:55
beach
But it is a beautiful research project. Take the literature on compiler optimization techniques that claim to need SSA, and show that they don't use the fact that SSA is S.S.A.
15:16:22
beach
I know of one algorithm that needs it, but then it was shown that the fact that it does makes it worse than others.
15:18:18
karlosz
"we can "implement" the phi-functions using a move instruction on each incoming edge, as shown in Section 19.6"
15:20:21
beach
But, since I am not yet sure that it can do all the interesting published optimization techniques, I am not willing to make such a declaration quite yet.
15:21:37
karlosz
you need to consider multiple defining instructions for those merge point locations
15:21:45
beach
That is entirely possible. But those algorithms should be examined to see whether they perhaps could even be simplified.
15:22:39
karlosz
so you can easily distinguish between normal assignments and thioose join point assignments
15:41:12
beach
karlosz: The optimization technique you gave a link to looks like one that should be included in Cleavir. In case I haven't mentioned it, I think Cleavir should ultimately contain a large collection of such techniques, so that client code can choose the ones that are appropriate for that particular client.
15:42:00
beach
Many of the published techniques need to be adapted to the fact that our language is more complicated than the C-like language that is assumed in published techniques.
15:50:51
karlosz
yeah, i adapted the technique by defining a generic function protocol on instructions that declare wehtehr they allow constant folding and methods for how to actually constant fold
15:58:33
attila_lendvai
drmeister: so, in the current build scheme the scanning of extensions/ only makes sense in the 'dclasp' stage, right?
16:02:55
drmeister
Correct, vanilla clasp never messes with extensions. However, vanilla clasp does automatically check for extensions. I like the "(1) clone an extension into clasp/extensions (2) it just builds
16:06:48
attila_lendvai
drmeister: I'm working towards my staged bootstrapping idea. I already have stage-1 (iclasp) building, and now I'm reducing the codebase for the stage-2 branch (aclasp). I'm envisioning that these early stages will be much simpler, with much fewer files. and the build scripts will also be much simpler, e.g. all of the stage_char complexity can go away from them.
16:09:13
attila_lendvai
drmeister: it'll be a long ride, but I have a rather exciting vision. everything will go into my own repo until its status/demo gets you excited, too... :)
16:12:50
attila_lendvai
it will have its costs, e.g. 3-4 potentially diverging codebases, but if things work out as I expect them to, then touching the earlier stages will not be necessary too often, and in return it will make bootstrapping much more understandable and debuggable.
16:14:22
attila_lendvai
there'll be a "toplevel" bootstrap.sh or equivalent that checks out and builds the stages under build/, and the toplevel dir will be used to build the currently latest stage
16:15:03
attila_lendvai
I'm also considering how to introduce two "starting points" for the bootstrap, e.g. another one started off of sbcl+cleavir.
16:16:28
attila_lendvai
I should have a much more detailed understanding of the bootstrap process, the lack of which will bite me a few times, but... how else to learn its intimate details?
16:17:49
drmeister
The sbcl+cleavir one Bike and I have talked about a lot - it requires a lot of clasp runtime to be replicated in sbcl.
16:18:28
drmeister
I think if you do that one - that might be the only one to focus on. If you could compile the cclasp code from sbcl+cleavir then why do anything else?
16:18:46
attila_lendvai
I won't attempt it, just thinking ahead to set things up so that we can accommodate for that
16:19:40
attila_lendvai
drmeister: for the beauty of it? of being able to bootstrap the same system based off of two distinct languages and ending up with reproducible builds (same output executable from both base languages)
16:19:47
drmeister
We were thinking that sbcl+cleavir would compile to a list of generic function calls that would be run in the iclasp-boehm interpreter to invoke the llvm functions to generate code.
16:21:07
drmeister
I'm all for beauty - but I don't want anyone wasting time on something that won't be used. There aren't a lot of us.
16:22:25
drmeister
I don't have a lot of energy to talk anyone out of anything. But please, really think hard about spending a lot of time on something that won't be useful.
16:23:18
drmeister
Potentially diverging codebases makes me anxious. However, we are bumping into constraints wrt what we can optimize because of the way clasp builds.
16:24:55
attila_lendvai
it's also a bit of a security concern, although somewhat far fetched... see the http://wiki.c2.com/?TheKenThompsonHack (a trojan in gcc that reproduces itself when bootstrapping gcc from sources with a gcc.exe, and also "miscompiles" the sources of login to implement a backdoor)
16:25:21
drmeister
How about you talk with Bike about this more? He has thought about the sbcl+cleavir compilation a lot. If you two could come up with a solid plan for a better build system that (1) is faster and (2) doesn't constrain optimizations - we could support you to do it.
16:26:41
attila_lendvai
drmeister: my hope is that we will be able to easily and cheaply "abandon" stages and "escape forward", which will make the diverging codebases less of an issue (unless we manage to introduce a bug that manifests several stages later, but I'm not sure I could even create one if I was asked to)
16:27:20
drmeister
Yeah - I'm aware of the compiler trojan issue - I worry about it like I worry about malevolent AI. All I say is - just don't doesn't mess with my molecules.
16:28:06
attila_lendvai
I'll keep you guys updated here, but first I need to have a proof of concept that can rebuild clasp as it is currently to convince myself that this is viable and worth it
16:28:18
drmeister
Well, it's more of a concern. We could use cclasp to build cclasp right now - we don't because we are worried about the compiler spinning out of control with a bug.
16:32:58
attila_lendvai
I would like to retain the ability to bootstrap off of clang, and the longer I wait, the more probable that you will eventually jump the sharks and start using clasp.exe for further development. I want to come up with something before that... :)
16:45:36
drmeister
karlosz: Type (1) 'make' (2) a bunch of stuff happens really fast (3) you have a working clisp executable.
16:50:20
drmeister
Just so that I have this absolutely clear. You compile cleavir with clisp and then compile cleavir with that. Then you can run the result of that and compile cleavir again?
16:51:43
drmeister
Because we are comparing cleavir generated clisp bytecode to cleavir generated llvm-ir lowered to native code.
16:52:51
karlosz
like, the speed gain you have from having better generated code is a microoptimization compared to not having to do llvm optimizations
16:53:54
karlosz
but that doesnt impact the performance much either, since all ssa optimizations run in linear time
16:54:07
drmeister
Well, we've been talking about doing that. Adding your hir/mir->ssa conversion and then lowering ssa to llvm-ir. It would be less work for llvm.
16:56:19
drmeister
We can profile Common Lisp generated code alongside C++/C code. Nothing has stood out.
16:57:15
drmeister
I've had a nagging feeling for years that something is wrong hidden in the runtime. This might help us suss it out if it's really there.
16:57:48
karlosz
right. to be absolutely certain, the way im self compiling is to do (dolist (system '(cleavir-generate-ast cleavir-ast-to-hir sicl-boot cleavir-dominance))
17:07:50
drmeister
I have a recent asdf - but it has a single asdf/build/asdf.lisp and uiop is included in that - isn't it?
17:08:29
drmeister
I commented out the uiop.lisp load and use /Users/meister/Development/cando/src/lisp/modules/asdf/build/asdf.lis
17:11:37
drmeister
Sorry for the basic questions - I know Clasp - other lisps with asfd/quicklisp - I'm a babe in the woods.
17:17:24
karlosz
do this: https://github.com/quicklisp/quicklisp-client/commit/957e342a455fee4f8e708c66b8cf25825b19e280
17:25:57
drmeister
karlosz: LOAD: A file with name /Users/meister/Development/clisp/src/cleavir/ssa-constant-prop.lisp does not exist
17:43:34
drmeister
Ok, I did it twice - how do I know that the second time it's cleavir generated bytecode that is building everything?
17:45:13
karlosz
also, for me at least, once cleavir starts compiling itself i just get a bunch of expected at most nil warnings
17:54:45
drmeister
If I then want to compile-file some test file using cleavir - is there anything I need to tell compile-file to use cleavir?
17:58:01
karlosz
so i need that for genereate ast, which doesnt accept (FUNCTION ... ...) with two arguments
17:59:22
karlosz
in check-special-syntax.lisp in generate ast, #-cleavir goes above the method that specializes on (eql 'function)
17:59:45
drmeister
https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/environment-query.lisp#L3
18:00:05
karlosz
https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/check-special-form-syntax.lisp
18:00:43
drmeister
Here? https://github.com/robert-strandh/SICL/blob/master/Code/Cleavir/Generate-AST/check-special-form-syntax.lisp#L66
18:03:48
karlosz
beach: check-special-form-syntax in generate-ast doesnt accept a system parameter like convert-special does. this ends up being a problem for implementations that let function take two arguments, for example. is it worth changing now?
18:16:40
drmeister
There is a lot of output being generated - it's slowing things down. Is this expected?
18:18:42
karlosz
weird. i get that much output and it can still do it in about a minute and a half for me
18:18:58
karlosz
im not quite sure whats causng it. if it really were miscompiling, it would have segfaulted by now
18:19:55
karlosz
is your terminal slow or something? i do this in slime in emacs and it can cope fine