freenode/clasp - IRC Chatlog

13:02:15 drmeister Hello

13:02:30 drmeister slime doesn't compile with the new cst-to-ast code - argh - still problems.

13:05:41 drmeister For some reason (CONVERT-ELEMENTARY-SETQ" #<ATOM-CST raw: SWANK:*LOG-EVENTS*> #<ATOM-CST raw: T> NIL #<CLASP-CLEAVIR::CLASP-64BIT>") fails because its getting no-variable-info on swank:*log-events* in clasp.lisp even though in swank.lisp it's declared with (defvar *log-events* nil)

13:07:17 drmeister (cleavir-env:variable-info nil 'swank::*log-events*) -> NIL

13:08:02 drmeister (core:specialp 'swank::*log-events*) -> NIL

13:08:04 drmeister Huh

13:23:36 drmeister Gotta flush the caches

13:50:01 Bike i realized i can't just throw more slots into function because that'll change the offsets of the slots in funcallable instance, which are set up to match those in standard instance

13:50:21 Bike so i'll just use virtual functions or whatever, since it's not like getting documentation needs to be fast

14:03:15 attila_lendvai drmeister: small merge error in cst branch: in wscript the git entry for mps is duplicated

14:03:27 attila_lendvai doesn't cause any ill effects, though

14:04:40 Bike ah, was wondering why i got towo notes for that

14:04:42 Bike two

14:38:39 Bike okay so the hierarchy we have for functions is ClosureWithSlots < FunctionClosure < Closure < NamedFunction < Function

14:38:44 Bike this seems kind of unnecessary

14:40:56 Bike relatedly, we ALREADY HAVE functions with a docstring slot

14:40:57 Bike unbelievable

14:42:23 Bike of course, calling it on a normal function just dies immediately

14:42:37 Bike drmeister: when i say i don't understand the runtime, this is the kind of thing i mean

14:47:59 Bike we should really get rid of hard errors

14:48:05 Bike or at least not use it for all this stuff

14:48:19 Bike somehow i got a hard error from no applicable method for a single dispatch gf

14:50:33 attila_lendvai ACTION is experimenting with splitting the iclasp, aclasp, b, c stages into git branches. currently stripping down the repo into the bare minimum needed to build iclasp and call it stage 1 (no plans for committing any of this into the official repo)

14:50:54 Bike there's also a CompiledFunction_O, which compiled functions are not?

14:51:19 Bike oh but bclasp compile returns one

14:51:25 Bike that's, we shouldn't have that happening

14:59:01 Bike we have a class called MacroClosure which as far as i can tell is exclusively used for backquote and quasiquote which are different things?

15:00:32 Shinmera I'm going to bet they're different rewrites of the backquote.

15:01:23 Bike no, they at least seem to go through the same code

15:01:29 Bike they're defined right next to each other

15:01:38 Bike why.

15:01:45 Shinmera Yeah but are they both used?

15:01:48 Bike you know what's making the build slow is having all this nonsense in it

15:01:50 Bike yes, yes they are

15:01:57 Shinmera :u alrighty then

15:01:59 Bike the reader puts in backquote, but one point in clos puts in quasiquote

15:02:06 Bike fuck that, i'm cutting this down

15:02:10 Bike clhs `

15:02:11 specbot http://www.lispworks.com/reference/HyperSpec/Body/02_df.htm

15:02:40 Bike what's the macro in sicl? quasiquote, right? i'll make that the canonical one

15:05:09 spiaggia Having explicit offsets for the documentation string, instead of having the offset derived from the location of the slot seems like a bad idea.

15:05:25 Bike good thing it's not on the table

15:06:01 spiaggia I thought that's what you said before.

15:06:21 Bike it's not

15:07:00 Bike The thing is that there's a basic multiple inheritance in funcallable_instance inheriting from both function and instance, and we'd like all of them to be pretty fast

15:07:17 Bike having the function entry point and instance slots at the same locations in everything is convenient

15:10:10 spiaggia What is preventing you?

15:10:50 Bike from what?

15:11:35 spiaggia having the function entry point and instance slot at the same location in everything.

15:13:49 Bike If I add a slot for the docstring to the function class, the instance slots in funcallable instance will move down.

15:15:37 spiaggia How about making a common superclass then, documentable-class.

15:16:17 spiaggia documentable-instance rather.\

15:16:28 spiaggia Sorry, tiny keyboard.

15:16:57 Bike instances don't need docstrings

15:18:02 spiaggia Some do.

15:18:54 Bike so have funcallable instance inherit from documentable instance but have instance not. right. that doesn't change the issue of coordinating instance and funcallable instance.

15:21:13 spiaggia That's not what I meant, but it's not important I guess.

15:22:28 Bike why does backquote have all these special variables in it

15:22:30 Bike ggh

15:23:13 spiaggia In Eclector?

15:23:20 spiaggia Or the SICL reader?

15:25:13 attila_lendvai quasiquote is hell of a rabbit hole. something that human brains are not trained for... or at least not mine.

15:25:31 attila_lendvai I mean their implementation

15:26:32 Bike in clasp's C++ reader

15:26:45 spiaggia Ah, OK.

15:26:46 scymtym spiaggia: they are still mostly the same at this point (except w.r.t. tests)

15:27:10 scymtym bug fixes have been ported in both directions afaict

15:27:46 Bike we indicate whether a function is a macro by setting a slot _in the function_

15:27:52 spiaggia scymtym: Yes, I didn't mean to distinguish between them. I meant to ask whether it was in Eclector/SICL on the one hand or in the Clasp reader on the other hand. Sorry for the misleading formulation.

15:28:03 Bike the C++ part of clasp is conceptually as messed up as possible

15:28:16 spiaggia Sounds like it.

15:28:45 scymtym spiaggia: i see

15:31:36 Bike forget docstrings, i'm just going to mess things up until i can build it without bootstrapping, and then just burn out all these old foundations

15:34:12 spiaggia Hmm, to me "boostrapping" and "building" are almost synonymous. What is "bootstrapping" to you?

15:35:06 Bike having a C++ program that evaluates code for a lisp compiler which then compiles lisp code for another compiler which then compiles itself

15:35:41 spiaggia Heh, got it.

15:36:12 spiaggia What do you plan to do instead?

15:36:54 Bike compile everything and then have a small C++ program load it later

15:37:26 spiaggia Cross compile on some Common Lisp implementation?

15:37:32 Bike yes

15:38:23 spiaggia Music to my ears.

15:39:14 spiaggia How do you count on interfacing LLVM to the host for cross compilation?

15:39:41 spiaggia Sorry for the nosy questions. This is very interesting to me.

15:39:55 spiaggia Don't feel obliged to take the time to answer.

15:42:03 Bike well that's the annoying part

15:42:18 Bike currently it's looking like the cross compiler will have to output some intermediate, and then some other program does the last llvm part

15:42:39 spiaggia Hmm, yes, I see.

15:50:02 spiaggia Speaking of which, how does Clasp handle C++-specific things like vectors of structs that don't make sense in Common Lisp?

15:52:55 attila_lendvai Bike: would it help you if there were separate build stages for iclasp, aclasp, etc... and where the build script would understand these stages randing from 1 to n, and build them one after the other? my impression is that kludges were much less of a headache to contain with such reified stages

15:53:45 Bike Dunno. In this case it's the C++ code (iclasp) just being bad.

15:53:46 Shinmera I thought that was already the case?

15:54:14 Bike spiaggia: C++ arrays can't be used as lisp arrays.

15:54:26 spiaggia Ah, OK.

15:54:31 attila_lendvai Shinmera: the major difference would be: the codebase is isolated in branches. and adding a new stage would be cheap in programmer effort

15:56:03 Bike i don't understand the branch thing. like, during build it has to switch branches a bunch of time to get all the code?

15:56:08 attila_lendvai another huge benefit would be that you can freely rework e.g. the wscript file in stage-n without all the complexity of the prior stages being in your way flattened into the same file.

15:56:17 attila_lendvai Bike: yes

15:56:42 Shinmera I feel like segregating things into directories per stage would be fine, but not branches.

15:57:15 attila_lendvai and there would be an "API" for the build scripts between the stages. or we could just rely on stage-4 knowing some details about the build output of stage-3...

15:58:01 attila_lendvai eventually stage-42 could evolve to the point of needing any CL implementation to bootstrap from, including clasp stage-41.

15:58:56 Bike i remember you put this in the issues before, but we're pretty much going to lose track of things if we need a whole great stage of being

15:59:01 Bike chain

15:59:44 Bike bclasp and cclasp also aren't actually separable

16:00:03 attila_lendvai my impression is that it will be easier on the brain, because dealing with stage-n only requires knowledge/context of that stage, and a bit of the previous one

16:01:07 Shinmera that's assuming that the connections between stages are few.

16:02:55 attila_lendvai IIUC, the connection would be limited to only that part of the codebase that implements compilation, i.e. Cleavir. Cleavir stage-5 can only use features that the stage-4 clasp binary understands

16:05:46 spiaggia I for one am very lost.

16:06:02 Bike i just want to not bootstrap

16:06:24 Bike i want all this stupid code to go away

16:06:34 attila_lendvai Bike: bclasp and cclasp could be one stage in this scheme (if I understand them correctly). stages would only be needed when you want to use/rely on a new feature *in the compiler*.

16:06:49 Bike that's what i'm doing all the time

16:07:20 attila_lendvai not sure you can get rid of bootstrapping. I'm hoping to be able to make bootstrap issues much more explicit and contained

16:08:10 attila_lendvai but I'm very early understanding the clasp codebase, and besides pondering on bootstrapping for a long time, I never really got my hands dirty before

16:10:20 Bike we need to get rid of bootstrapping. it makes most serious improvements impossible.

16:10:51 drmeister Howdy

16:11:22 Bike it'll be hard, we'll basically be refactoring the entire system, but this is not sustainable

16:11:52 Bike it's subprime mortgage crisis levels of technical debt

16:11:58 attila_lendvai I think 'bootstrapping' means different things to us, because I think getting rid of it is not possible

16:12:50 drmeister attila_lendvai: Bike, stassats, beach and I have a plan in motion to redo the build system by cross-compiling in sbcl or another lisp, generating an intermediate representation and then passing that intermediate representation through iclasp to generate the bitcode for cclasp.

16:13:59 drmeister The build system we have now is complicated and messy but we don't want to make any major changes to it because it's going to be superseded by this new system eventually.

16:16:10 drmeister It's going to take work to get the new build system working - you might be interested in helping out with it. Basically we need to develop an environment in standard Common Lisp that replicates the implementation specific environment that clasp bootstraps within.

16:17:01 attila_lendvai drmeister: so, basically the plan is to assume that there's a functional lisp binary (e.g. SBCL), have it load Cleavir + the clasp codebase, then run it to compile clasp sources to something simple and low-level (?) which then gets processed/compiled/finished by a hand-written simple interpreter (iclasp)?

16:18:13 drmeister Yup

16:18:18 attila_lendvai I'd love to. is there anything on it I can look at? docs, irc logs, a branch, etc...

16:18:57 drmeister It's in the logs - but it's pretty scattered. Ask Bike about it - it's mostly stored between his ears.

16:19:50 drmeister But you got the gist of it.

16:20:36 attila_lendvai nah, it would hold him up... Bike should instead write up a short outline of the plan in the wiki, so that newcomers like me can develop at least the same vocabulary

16:20:38 drmeister It will be way more fun to develop an environment within sbcl with all the tools available than to muck around with clasp's bootstrap code.

16:21:24 attila_lendvai drmeister: but on the other hand it's rather cool to be able to bootstrap off of a C++ compiler. even better if you can bootstrap the same codebase off of C++ and CL at the same time

16:22:31 drmeister Yeah - but stassats pointed out correctly that maintaining two very different build systems will be a huge maintenance headache.

16:23:03 Shinmera attila_lendvai: You still can. Build ECL with your C compiler, built SBCL from that, then build Clasp from that.

16:23:05 drmeister So, right now we have what we have and when we develop this other approach we will have two - at that point we can decide what we want to do next.

16:23:28 drmeister There you go.

16:23:37 drmeister Or build ECL with your C compiler and build Clasp from that.

16:23:51 Shinmera Trying to be implementation-independent is probably more headache than it's worth

16:23:54 drmeister ECL is a lot closer to Clasp than SBCL is.

16:23:58 Shinmera Just ask the SBCL guys.

16:24:25 Shinmera And, really, I don't see that much worth in being able to run from a C compiler.

16:25:24 attila_lendvai Shinmera: I don't just want the sheer ability through kludges, I'd like to have something elegant.

16:25:25 drmeister ACTION is feeling the pull of dozens of fields in half a dozen government forms that need to be filled out...

16:25:35 Shinmera attila_lendvai: This is not the project for you, then

16:25:51 drmeister ACTION wanders off to tend his fields.

16:26:35 drmeister C'mon guys - be good to each other.

16:27:09 Shinmera I'm not being hostile, Clasp just really isn't the project to be idealistic with. The constraints are too tight and time too little to be anything but practical.

16:27:43 Shinmera Trying to broaden things to be host-independent is already a step too far into the idealistic world.

16:27:46 attila_lendvai Shinmera: not sure how to react to that. I can always do whatever pleases my call for hacking, and then keep it in a separate repo...

16:28:15 attila_lendvai Shinmera: you don't know how practical it is until it's done at least to the level of proof-of-concept. (me neither, I just have a vision)

16:29:06 Shinmera More ways to fuck up the code is less practical. More implementations means more ways to fuck up the code due to minute or big differences.

16:30:23 Shinmera For now, just getting things to work at all being restricted to SBCL as host is going to be tough enough.

16:30:57 attila_lendvai Shinmera: well, my vision is that it would make things much simpler. if it ends up otherwise, then it can always be abandoned and thrown away...

16:31:24 Shinmera How would having to mind multiple implementations and their quirks while writing code be simpler?

16:31:38 Shinmera Writing any library or application has never been simpler to write portably than not.

16:34:12 Shinmera I agree with you in the sense that, if the code could be made purer thanks to being portable, that would be a good thing. But such a thing needs to be proven first, whereas if you just pick an implementation you can just get started doing your thing.

16:38:01 attila_lendvai the reason I think it would make things simpler is because the source for the various stages would be isolated from everything else, and git would be used as a time-warp-machine to "teleport" to various stages of the codebase. and then in those stages you can make as big kludges as you want, as long as it manages to produce a binary that can give life to stage-n+1. in that sense there would be much more freedom to kludge ahea

16:38:02 attila_lendvai cherry-pick commits between stages

16:38:14 Shinmera attila_lendvai: Just to clear up any potential misunderstanding, I'm not trying to tell you what to spend your time on or anything, and definitely don't mean you any ill will.

16:39:06 Shinmera Ah- well, I was talking about the idea of making the cross-compiler host-lisp independent.

16:39:32 Shinmera Your stage separation idea I can't really estimate, so I have no opinion aside from being wary.

16:49:52 Bike pretty much anything i do towards letting clasp build from sbcl which also let it build from other implementations

16:50:05 Bike at this point i'm basically cutting clasp-specific stuff out of compile time

16:51:39 Bike and i have a head start in that cleavir is already portable

16:52:41 Shinmera Well, it's portable in theory, I don't think it's been proven in practise yet.

16:53:48 Bike sure it has. it loads in clasp, it loads in sbcl, i've loaded it in ccl a couple times.

16:54:03 Bike and generated ir and such.

16:54:22 Shinmera My fear is that nasty things like host-specifics might be sneaking into the cross-compilation that make the process more delicate than expected.

16:54:37 Bike it's a major concern, yes.

16:55:27 Bike but the way cleavir works you totally specify the environment.

16:56:40 Bike cst build. astounding.

16:58:48 Bike beach has been pretty conscientious of cleavir only going through the proper environment accessors. every time i've tried using cleavir without a correct environment it hasn't worked, even when the host is fine.

16:58:59 Bike so i'm not too worried about cleavir.

16:59:56 Bike the bigger isssue is that clasp is written to be loaded rather than compiled. it's pretty much a refactor.

17:04:42 Bike drmeister: i'm getting a special variable problem loading slime, like you did.

17:06:26 Bike with no cache, i mean

17:06:29 attila_lendvai what is the cst branch?

17:06:52 attila_lendvai I mean, what does it stand for, or what's cooking there?

17:08:22 Shinmera concrete-syntax-tree

17:09:18 Shinmera iiuc switching the reader to beach's new thing.

17:10:54 Bike and the compiler.

17:11:15 Bike so that we can have source tracking and debugging our 999999 problems will be a little more tolerable

17:15:27 attila_lendvai yeah, whichever environment I ever had to work in, I never continued until debugging/logging was tolerable

17:26:04 kpoeck Assuming that the build system will be redone one day using sbcl or another lisp, are there part of the codebase that will go away? I'd rather find errors in the part that will stay than in the part that will go away

17:26:38 attila_lendvai ACTION swears a little at waf and wscript for being full of non-obvious dwim magic

17:39:51 Bike kpoeck: mainly the stuff that's only used "early"

17:40:15 Bike so like lots of bits in C++, and some of the "early" definitions in evalmacros and such.

17:44:35 kpoeck Well c++ is not my strength, so I will than try to go for the "late" lisp part

17:45:08 selwyn hello everyone, am building clasp and look forward to using it

17:47:10 selwyn i got a few errors that went away when i swapped in a new version of `concurrentqueue' in /include/clasp/external which has an appropriate patch

17:47:38 Bike kpoeck: like, if you look at build_files.py, everything after, say, evalmacros.lsp is probably going to stick around. excepting some of clos.

17:47:53 Bike selwyn: what branch?

17:47:55 frgo selwyn: You seem to be at the wrong branch!

17:48:00 selwyn i am happy to file a detailed issue report on github, but since it is fixed by upgrading 'concurrentqueue' idk if you would be interested

17:48:26 frgo concurrentqueue isn't used anymore ...

17:48:44 selwyn i am on 'testing' clasp and 'master' externals-clasp

17:49:25 selwyn which branches should i use?

17:50:24 stassats dev?

17:51:04 Bike dev. and externals clasp is gone now

17:53:02 selwyn will do thanks guys

17:53:26 frgo Bike: externals-clasp is gone? Really?

17:54:51 Bike that's what drmeister said last time i asked

17:55:22 Shinmera drmeister: I don't think you're still using my CI machines, right?

17:55:28 Shinmera They haven't been building successfully in a long time.

19:42:13 Bike drmeister: yeah the backend is loaded before swank is.

19:42:30 Bike drmeister: looking at it, i don't think we should refer to the log stuff there anyway. i know clasp is experimental but we can put that in swankrc or whatever.

19:45:10 drmeister Shinmera: No - I haven't been using your CI machines for a while.

19:45:31 drmeister I'm not sure that we will - I think we are going to move CI to here.

19:45:47 drmeister externals-clasp is still needed on MacOS

19:46:38 drmeister Linux doesn't need externals-clasp

19:47:09 Bike ugh

19:47:15 Bike still gotta wait then

19:47:22 Bike my mistake

19:47:51 drmeister For provisioning a linux machine check out the cando-build Docker file in clasp/tools/dockerfiles/cando-build/Dockerfile

19:49:38 drmeister Bike: So just remove the setq for swank::*log-events*?

19:49:44 drmeister from clasp.lisp?

19:50:21 Bike i'm gonna take out the block yeah

19:51:45 drmeister Building slime

19:52:52 Shinmera drmeister: Okey, saves me some energy.

19:53:47 drmeister I would have liked to get it working one more time - because it gave us a record of how/if the build times were improving.

19:54:04 drmeister But it's probably more trouble than it's worth.

19:54:49 drmeister That's why I've been hemming and hawing about getting it working again - but - go ahead - cut the chord.

19:58:25 Shinmera Well I don't know what's wrong with the build. Likely a missing library or something which should be easy to fix, no?

20:08:20 Bike now the error is somewhere in gray

20:10:16 Bike and might be an actual problem. there's a variable 'package' apparently bound to nil, but if it is so it ought to signal an error earlier

20:44:52 Bike why do we load direct-calls.lsp before init

20:45:14 Bike there's also a comment saying we don't load it at all, which is false

20:45:19 Bike hate

20:46:11 Bike and i can't look at the generated/cl-wrappers that apparently exists

20:46:40 drmeister Shinmera: It should be easy to fix if you want to hold off for a few days.

20:47:46 drmeister Bike: the scraper generated wrappers are in clasp/build/boehm/generated/*

20:48:19 drmeister I put direct-calls.lsp and cl-wrappers.lisp before anything else so that everything takes advantage of them.

20:48:34 Bike what i'm trying to do is use (setf macro-function) instead of fset

20:49:07 Bike i defined a (setf macro-function) function in evalmacros, but that wasn't early enough. so i put it in defmacro, but that wasn't early enough. so i put it in init, because i thought that was the first thing loaded, but nope

20:49:21 Bike we should just define it in c++

20:49:29 Bike except we have to change the scraper or something for that