freenode/clasp - IRC Chatlog
Search
17:46:56
drmeister
I'm still working on literals - I've got a good way to do it now that will be faster for bclasp and cclasp.
18:37:45
drmeister
The stack use is a bit heavy with several C++ frames for each C++ function call - and tail calls are not supported yet.
18:38:26
drmeister
I'm working on that stuff right now to speed up access to literals - so it's a good time to fix it.
18:39:13
drmeister
I just got a huge helping hand from someone in the #llvm chat room. I had a problem with the 'getelementptr' instruction and we came up with a solution. Phew.
18:40:32
drmeister
This one is pretty specialized. I create a 0 element array for literals and then generate code referencing that array and then once I'm done with the code I know how large the array needs to be so I create a new one and replace every use of the old one.
18:40:58
drmeister
It turns out I have to cast the pointer to the new array to the type of the 0 element original one.
19:04:03
attila_lendvai
does this make any sense: there are fundamentally two strategies to bootstrapping something like clasp: 1) keep the current three stages of aclasp, bclasp, cclasp, based off of C++. 2) set up a scripted time machine in the git repo, where each stage can be checked out, and stage n is built using stage n-1, and where stage 0 is written in C++.
19:04:50
attila_lendvai
if a bootstrap is needed all the way down from C++ for a new platform, then it can be done with a script that builds each necessary stage from the past to get to the current.
19:05:27
jackdaniel
ecl has basic runtime written in C (it's not CL, it is some lisp though), on which is loaded CL – now you have interpreted/bytecompiled CL environment
19:05:46
jackdaniel
then you load compiler (written in CL), and compile CL once more to native code (for performance reasons)
19:05:49
attila_lendvai
whenever the clasp compiler (written in lisp) wants to use a new lisp feature, then a new stage needs to be introduced
19:10:17
attila_lendvai
in a way 1) is a less generalized subset of 2) where someone needs to walk the stage0->stage2 at each build, and where the rebuild_cboehm target is basically when stage2 is built using another executable output of stage2 (which will only work if no new features/bugfixes are needed for the clasp compiler)
19:15:26
attila_lendvai
* is somewhat excited about the idea of setting up such a build script for cclasp
19:32:49
attila_lendvai
Shinmera: did you mean that in the context of the above bootstrapping context? if so what are image based upgrades? what I talked about is very similar to what exists now (i.e. not image based upgrades, although I'm not sure what that means).
19:34:08
Shinmera
attila_lendvai: A couple of lisps (like CCL) use an image of a previous versionf of themselves to "flash on" the upgrades to produce the newer version. This is usually a nightmare.
19:35:16
Shinmera
attila_lendvai: Otherwise, I don't see how your idea works with the C++ side of things, since that code frequently changes too, in which case I'm quite sure you have to start from scratch again.
19:35:22
attila_lendvai
what I meant was about a complete build involving LLVM and linux exe outputs
19:37:04
attila_lendvai
me neither. I should understand the build much better to be able to formulate it more clearly (or understand that it's not feasible)
19:37:18
Shinmera
As far as I know the build system /should/ support incremental updates to only rebuild the parts on top that are necessary to rebuild.
19:38:10
attila_lendvai
*if* it worked, then the benefit would be that when you add a new major feature (evolve the compiler), then you don't need to go down to the aclasp level of hand-written code to bootstrap the new feature, but you can start off of the current stage
19:38:33
Shinmera
And for people who are building for a first time from scratch, I don't see how it would help, since the C++ base is exactly what needs to be tailored to the setup you're running, so you have to start from 0 anyway.
19:39:20
Shinmera
Like-- who are you seeing as the target audience for this? The devs? The testers? The end-users?
19:40:16
Shinmera
And if it's not, then the incremental building capabilities of it have to be improved. No need for fancy stashing of binaries or whatever.
19:40:40
attila_lendvai
e.g. imagine the situation where you want to extend the scraper that would generate code that aclasp/bclasp couldn't deal with, only the current cclasp
19:42:51
attila_lendvai
you can add the new complexity into the current setup, separating the two scraper code and output in the repo/build dir... or you can add the new complexity to the build script that checks out stages from the git repo and builds them into separate build/ directories, and you can just edit/extend the scraper in a stage-n+1 location in git, not being concerned about aclasp/bclasp.
19:43:29
attila_lendvai
you can always bootstrap from 0 again, this setup is already implemented in gambit scheme.
19:44:01
Shinmera
I've read what you said just now four times and completely failed to understand each time.
19:45:06
Shinmera
I really don't get how stashing away any part of the build in resulting binary form would help for anything.
19:45:13
attila_lendvai
can you imagine a situation where two scrapers are needed, one for bclasp, and a smarter one for cclasp?
19:46:18
attila_lendvai
I didn't say that. this setup would have git branches or tags that capture the stages. binaries would not be stored in the repo (except as an optimization perhaps)
19:47:08
Shinmera
Ok, let me revise: I don't see how stashing away any part of the build would help with anything.
19:47:35
attila_lendvai
doesn't the scraper go through the C++ code using LLVM and generates stuff that bridges the C++ universe with the clasp universe?
19:48:21
Shinmera
It uses the C preprocessor and scans for tags to extract info about function signatures.
19:50:17
attila_lendvai
so, what if the bclasp and the cclasp universe are substantially different regarding what the scraper needs to output. e.g. let's say bclasp doesn't GC the C++ objects and they just accumulate until exit (a random simplification). it's good enough to build cclasp, but not as final production code.
19:51:55
attila_lendvai
* would like to know whether drmeister is also this baffled by my above thoughts
19:54:01
Shinmera
Anyway, again, the static analyzer is needed at the very beginning of the build. In fact, its output is never generated on your machine since it would take way too long.
19:54:47
Shinmera
The static analyzer is manually run by drmeister whenever new C++ functions/classes/whatever are exposed, and it takes forever.
19:55:42
Shinmera
It produces a file (I forget which) that describes every pointer in the system, which is necessary for the MPS based builds to run at all, and I believe it's used on the boehm side if it's without dynamic-cast.
19:56:00
attila_lendvai
my examples are not going to be precise until I better understand the actual headaches in the clasp build. but I thought about bootstrapping issues in general, and I have hands-on experience with setting up dual bootstrapping for maru. it can be bootstrapped either from a hand written C interpreter and a new one I wrote in CL.
19:56:52
Shinmera
We have to bootstrap from C++ because we need a C++ runtime from somewhere. So you either bootstrap from C++ or you write a C++ compiler in CL. Your choice.
19:57:49
attila_lendvai
isn't it trivial to invoke clang and LLVM from clasp? isn't it part of business as usual?
19:59:10
attila_lendvai
I guess this: "because we need a C++ runtime from somewhere". but your attitude is rather dismissive, so I guess I should stop brainstorming until I can respond in more concrete terms...
20:00:03
Shinmera
Well it's dismissive because we've had this discussion in here very often, and every time it ends up with the same conclusion. You either write a C++ bootstrapper and get the C++ runtime for free, or you get the CL bootstrap for free and write the C++ runtime.
20:00:49
Shinmera
We have two components that we need to interact with each other fluently, so we need to get them from somewhere into the same process in such a way that they can both speak the same language.
20:03:12
Shinmera
It might be feasible to write something in CL that can generate proper C++ function calls and entry points like we have for C, but I'm not holding my breath for that.
20:04:13
attila_lendvai
clasp can invoke the LLVM compiler and linker on the static C++ files to "bring in the C++ runtime", or even let the waf script do that. what I'm saying is really not that different than what happens currently, but I think with more flexibility while extending cclasp. I think the crux of the issue is that the CL side needs to be grown smart enough to run cleavir, and with the least amount of headaches.
20:08:27
Shinmera
Anyway, luckily for you I'm just the local grump and tester, so I don't really get to say what gets done on the code side of things, unless I scold people very heavily :)
20:08:58
attila_lendvai
another issue: let's say something cleavir version 2 requires tail call elimination. the straightforward way is to smarten up bclasp to do tail call elimination, then bring in cleavir v2. what I'm trying to get to is whether there is an easier way to introduce cleavir v2, that doesn't require making bclasp smarter, only cclasp (based off of cleavir V1)
20:09:48
Shinmera
I... /think/ the build system can do this already with a target called something_dangerzone
20:10:23
attila_lendvai
Shinmera: ok, here's another analogy: what I'm trying to do is make 'dangerzone' always work with the help of using git as a time machine
20:11:53
Shinmera
My point is this: if your change /only/ works when bootstrapped from a prior cclasp then you have run into the image based build hell and have made it impossible to properly bootstrap. If it does not, then what's the point of the time machine?
20:11:56
attila_lendvai
it's already in the repo. bclasp is used to build cclasp stage 0 (running cleavir v1). then cclasp stage 1 (running cleavir v2) is built using cclasp stage 0 (instead of a smart bclasp).
20:12:49
attila_lendvai
Shinmera: the build script can be written so that it automatically builds through all stages.
20:13:16
Shinmera
Plus it would just result in having to build hundreds of versions of cclasp if you have to start from scratch
20:13:49
attila_lendvai
it's probably more complex, but I think it also brings in much more freedom and efficiency for clasp developers.
20:14:05
Shinmera
The target is already called "dangerzone" because you might be inflicting changes that could affect the stability of your build. If possible you should build from scratch to make sure everything is sound. It is the only properly reproducible thing.
20:14:31
attila_lendvai
Shinmera: you only start from scratch when you bootstrap a new platform where you cannot cross-compile for a reason. this is not unlike gcc or any other compiler...
20:14:56
Shinmera
And I think it is exactly the image based hell that I was talking about. I am very much against this.
20:15:49
attila_lendvai
I still don't know what is an image based hell. do you mean 'image' as in sbcl.exe --image heap.dump?
20:15:53
Shinmera
We don't even have a deployable clasp yet, trying to make something that would require you to download previous versions of clasp that are prebuilt is madness.
20:17:22
attila_lendvai
why would it require downloading? it would build through the stages. currently aclasp=stage0, bclasp=stage1, cclasp=stage2, and this is exactly what happens, except they are called different, instead of differentiating them using git branches...
20:18:58
attila_lendvai
but I think I really need drmeister for this discussion, I hope he'll read the log and say that it's a useless/hopeless idea, or that it's worth entertaining it
20:19:39
attila_lendvai
Shinmera: because cleavir version n+1 may require something that is much harder to implement in bclasp than in cclasp (e.g. possibly proper tail call elimination)
20:20:26
attila_lendvai
or even simpler: it needs to be implemented both in bclasp and in cclasp, whereas with a stage3 it would only need to be implemented in cclasp.
20:20:55
attila_lendvai
and then you can just cherry-pick git patches into the separate stage branches
20:23:13
Shinmera
Right. Ok. So you're suggesting to keep three branches for the individual sources needed for each stage. One branch can be updated while a previous stage branch stays behind, if implementing the functionality is not feasible.
20:23:48
Shinmera
Now this may seem ok but you can still run into problems quick when a bug in something like cleavir suddenly prevents the next stage from building.
20:24:20
Shinmera
Backporting the fix may then become infeasible due to cross-dependencies on other features that since have also been missing on that branch.
20:24:59
attila_lendvai
actually I also understand it better now, so thanks for cornering me with questions! :)
20:25:19
Shinmera
I think this may seem like a neat thing, but if anything I'd want the number of stages to go /down/, not up.
20:26:00
Shinmera
And with each stage the plausibility of someone understanding what the hell is even going on anymore decreases. People are already very much afraid of clasp for its build system alone.
20:26:23
attila_lendvai
Shinmera: it doesn't happen in normal circumstances, because you only create the new stage when everything builds. the issue with LLVM is different... e.g. different stages need different LLVM versions due to random bugs/incompatibilities. only then you need to go back to an old stage to fix things up, if you want the whole thing to build with the same new LLVM version
20:27:20
attila_lendvai
bootstrapping is a complex issue. having aclasp thrown into the dir structure with bclasp already doesn't help
20:27:48
Shinmera
Okey, but the code structure is a separate issue that can be resolved without messing with the stages.
20:29:39
attila_lendvai
what I'm trying to get to is that bootstrapping is inherently hard, and what I'm proposing is (hopefully) not accidental complexity, but the formal encoding of the inherent complexity of bootstrapping
20:31:44
Shinmera
I think the stages as they are are easy enough to understand. C++ sources build an interpreter => interpereter loads CL code => Fully loaded loads Cleavir => compiles everything with Cleavir to produce optimised final version.
20:33:28
attila_lendvai
Shinmera: it took me a lot of time to get an idea WTF is a/b/c-clasp... having it called stage0,1,2, and even better separated into directories or branches, would have made it trivial to connect them to the bootstrap stages in my mind
20:34:23
Shinmera
Initially there were iclasp (interpreter), bclasp (bootstrapper), and then came cclasp (cleavir) and iclasp was renamed to aclasp to form a b c.
20:36:59
attila_lendvai
I prefer sticking to using well-known names of well-known concepts to using local names that need project-specific documentation
20:37:39
Shinmera
Lots of things are idiosyncratic because it was and still very much is a one-man project.
20:37:54
attila_lendvai
reminds me of this: “Programs must be written for people to read, and only incidentally for machines to execute.” — Abelson & Sussman, SICP, preface to the first edition
20:38:26
Shinmera
Well, when you don't have much time and are as eager as drmeister, then that often gets thrown to the wind in the name of progress.
20:40:03
attila_lendvai
I don't blame him. a balance must be found between polishing and making progress, because you may end up polishing stuff that will be thrown away, or you may end up making a lot of progress where the whole thing falls apart at 90%, when only the last 90% of the work is left... :)
20:40:33
Shinmera
Which is why I wish for more documentation. Then things could at least be figured out by other people and polished while he's still pushing the project ahead.
20:41:33
Shinmera
I'm doing what I can for that with my constraints, but motivating others to contribute has been hard.
20:43:07
attila_lendvai
I'm rather motivated to experiment with what I described above, but now I need to revive a server that requires windoze, so I need to reboot...