freenode/#clasp - IRC Chatlog

12:57:47 beach Bike: So if I understand you correctly, just have a single AREF instruction with not type annotation, and use TYPEQ to determine the type of the array input and THE to for the type of the output. I think that's a simplification, except perhaps for the parson writing the HIR->MIR translator who must then also know the type(s) in order to translate the AREF instruction to a memory reference.

13:00:29 beach I need to think more about having UNBOX and BOX ASTs. That would mean that ASTs can manipulate unboxed Lisp objects, and I haven't thought of the consequences of such a thing, but I guess it is no worse than having HIR manipulate such objects.

13:44:44 Bike morning. my idea was that if the hir->mir translator writer wants to do different memory references based on the type of the array, they'd subclass the array instruction and put the element type or whatever in it.

13:45:20 Bike aref involves typeq choosing array instructions either way

13:45:44 beach Not at the moment, I think.

13:46:07 beach At the moment, it is chosen by the person implementing the AREF function.

13:46:12 drmeister Hello. I ran the static analyzer on clasp last night. I just pushed clasp_gc.cc. It should be good to go with shared-mutex.

13:46:13 Bike that's what i meant, sorry

13:46:18 beach OK.

13:46:39 drmeister ACTION dives back into quaternions

13:46:52 beach Bike: Since that subclass of the AREF instruction is something that every implementation likely wants, we might as well put it in Cleavir.

13:48:09 beach There is no point in removing some feature from Cleavir that then has to be implemented in every client.

13:53:33 Bike you could have an implementation where array upgrading only controls what's stored in array-element-type, and all arrays are accessed identically. or, say there's an implementation that has characters as immediate UTF32, and also has immediate 32-bit integers. both have an upgraded array type. at MIR level they could be identical, and the implementation doesn't need to distinguish them in hir.

13:54:17 Bike maybe you're right though. and in clasp i could just subclass array to keep track of rank.

13:55:54 beach For such implementations, there is very little harm done to have a type field in the AREF instruction.

13:56:03 Bike yeah.

13:56:08 Bike actually that reminds me, what about complex arrays? it seems doubtful to me that an implementation could compile much down for a non-simple-array.

13:56:22 Bike maybe it could if there were like, arrays that aren't adjustable but do have fill pointers.

14:00:16 Bike in clasp i think an array access would ideally be (cond ((typeq array (not simple-array)) (full-call)) (t (let ((temp (if (typep array vector) array (md-array-underlying array)))) ...access into vector based on element type...)

14:00:34 Bike typeq, not typep, whatever

14:00:35 beach "complex arrays" as in arrays specialized for storing complex numbers?

14:00:41 Bike no, non-simple-arrays.

14:00:52 beach ACTION re-reads...

14:02:20 beach It depends on how you represent those arrays. SICL, for instance has a header and a "rack" for every object. The rack of a non-simple array contains the rank, so in a loop, for instance it is possible to have consistent access even if some other thread adjusts the array.

14:03:03 beach Yes, some implementations may not be able to do much with non-simple arrays, but that's their choice.

14:04:27 beach My plan for SICL is to implement adjust-array by allocating a new rack and then use CAS to replace the old rack.

14:04:46 Bike does sicl have simple arrays (in the sense of arrays that are actually not adjustable, etc)

14:04:55 beach Not really, no.

14:05:13 beach With the header/rack thing, basically all arrays are adjustable.

14:05:19 Bike thoughth so.

14:05:21 Bike *thought

14:05:37 beach I seem to recall *some* restrictions, but I can't remember them now.

14:05:59 Bike so right now cleavir aref tracks simple-p, which is inadequate for clasp and irrelevant to sicl.

14:06:06 beach The other advantage of the header/rack thing is that no locking is required between the time the size is checked and the time the access is made.

14:06:48 beach OK, the AREF instruction should definitely contain more complete type information.

14:06:58 beach Simplicity might have to be included.

14:07:51 beach We used to have a host of aref instructions, and I believe you simplified that by putting the type into a slot.

14:08:06 beach But I guess that type slot is not fully used yet.

14:08:19 Bike right on both counts.

14:08:41 Bike currently aref instructions have three slots: the upgraded element type, simple-p, and whether the elements are boxed.

14:09:01 Bike at least i think that last one is there. it's in the AST, where it's used in AST->HIR

14:09:18 Bike yeah, it is.

14:09:21 beach OK.

14:09:57 beach All those three slots could potentially be part of a single extended type descriptor.

14:11:28 Bike SBCL has "specialized array element type properties" structures. each one has an element-type, along with low level stuff like padding and the stride for memory reference, and probably stuff about boxing

14:12:33 beach I see. Some of that stuff is definitely implementation specific.

14:13:31 Bike let me check. the specifier, the sbcl internal type specifier, "fixnum-p", the default initial-element, n-bits, "typecode", "complex typecode", "primitive type name", and pad

14:14:05 beach OK.

14:15:38 Bike anyway, what i can do for now is have clasp subclass the array asts/instructions/etc to store the rank, and cleavir doesn't have to be rewritten at all

14:15:50 beach OK, sounds good.

14:16:30 Bike these decisions would probably be more straightforward if there were more clients.

14:16:56 beach Yeah, I need to get back to SICL so that there are at least 2.

14:17:11 beach ... and encourage jackdaniel to work on his compiler for ECL.

14:18:13 Bike CSTs going well? I mean you have the editor to deal with too.

14:18:24 beach I know :(

14:18:45 beach But yeah, CST is going well. Working on the documentation for lambda-list parsing at the moment.

14:19:18 beach About the rank in Clasp. Like I said before, in general, my hunch is that it is not worth making special cases for a single access in a function, so it is worth thinking about how to simplify the code. I think, for instance, that making NIL both a CONS and a symbol (so to speak) is not worth the trouble.

14:19:19 beach In a loop over the elements of a list, CONSP could be checked for first, making the SBCL optimization valid only for the last element in a list. Similarly, making a special case for a vector in order to make access to the size faster is probably not worth it. But, I advice against changing anything until we have loop-invariant code motion in Cleavir.

14:21:02 Bike oh, it's not to get to the size faster, it's to get to the data. which i suppose is also loop invariant, yes

14:21:18 beach And in fact, what SBCL does makes it necessary to do two tests in each iteration of a loop over a list. But we have been through this. stassats measured the performance and the additional test is not costing much. But, I am also concerned about simplifications of code maintenance.

14:21:59 beach Yes, well, in general, I would think twice about optimizing cases for a single access.

14:23:01 Bike that makes sense, but right now it just calls aref, so there's no chance of cleavir doing any code motion even if it did code motion.

14:23:10 beach Sure.

14:23:26 beach Just saying while we are on the subject.

14:23:32 Bike right. gotcha.

14:24:11 Bike and i don't think anyone was planning on nil = cons, as i remember that implies a lot of weird stuff in sbcl anyway

14:27:02 beach I am pretty sure it influences many aspects of the system, yes.

14:27:18 beach You mean nobody is planning a thing like that for Clasp?

14:27:24 Bike not that i'm aware of

14:27:54 beach Sure, but I often hear drmeister "justifying" a choice in Clasp by "that's how SBCL does it" as if that automatically meant it's the right choice.

14:27:57 Bike a few days ago i fixed it so that calling nil didn't cause segfaults, so i don't want to add any more maintenance like that :p

14:28:16 beach Heh, yeah.

14:28:26 Bike drmeister: when you're not busy - why again are there separate C++ types for not simple vectors (as in simple array and vector, not simple-vector)? I mean, there could just be complex-mdarray, simple-mdarray, and then a constellation of simple vectors with element types.

14:30:31 Bike sbcl is, you know, good, but i'd rather emulate the compiler than the runtime. the runtime is kind of scary.

14:30:46 Bike i mean, so's the compiler, but that's mostly in terms of how it's written, not what it's doing.

14:31:12 beach The result, seen by the application programmer, is very good indeed. I am not sure that it is as good for the maintainers.

14:31:54 Bike doesn't seem so, no

14:32:20 beach I do not want to plan to put myself (or anyone else for that matter) in a situation of having to maintain way too many special cases, especially if they can reasonably be avoided.

14:32:28 beach ... without loss of performance.

14:33:06 beach Bike: Somewhat related: I think vtomole needs some additional training and experience before we can put him on the type stuff. For now it is probably better to stick to simpler tasks.

14:33:23 beach But I'll deal with that.

14:34:09 Bike tell me if you need any help with that, i'd appreciate more hands

14:34:20 beach Oh, OK. Will do.

14:35:39 beach About the maintainability issue, I think it's an age thing. I feel my time is running out. drmeister has considerably more time left and you even more, so it is natural that the issue is not as important to you as it is to me.

14:36:37 Bike i don't mind a little moderation from the old and wizened, anyway.

14:37:02 beach Good. And thanks (assuming I was the reference).

14:38:02 Bike :p

14:40:02 beach About CST, I realized the other day that the work is not done when I finish the lambda-list parsers. I also need to implement PARSE-MACRO in terms of CST.

14:40:12 beach But that should be straightforward.

14:40:49 beach Anyway, it's shaping up. Once I have the documentation in place, I'll ask you to have a look.

14:40:52 Bike You did the macroexpansion guesser thing already, I think you said? and parse macro is closely tied to lambda list parsing anyway.

14:40:54 Bike Sounds good.

14:41:04 beach Yes, most of it is done.

14:47:48 drmeister Bike: There are separate C++ types because I wanted template function accessors in C++.

14:51:57 drmeister I figured out how to use quaterions in non-linear optimization.

14:51:59 drmeister http://i.imgur.com/6vEgoOB.png

14:52:11 drmeister The green is the DNA before optimization and the red is after.

14:53:09 drmeister This is a double helix that has been nicked twice to form segments: long-short-shortest

14:59:53 Bike template function accessors?

15:00:09 Bike that's a pretty sharp corner, it looks like.

15:03:25 drmeister Checking...

15:06:20 drmeister I'm not arguing strongly for the way I implemented things - but this...

15:06:36 drmeister I wanted a C++ operator[] for each specialized array.

15:07:03 drmeister One that would return values of that specialized array

15:07:37 drmeister Actually, one that returned references to values of that specialized array - so that I can use (*array)[i] = <value>;

15:08:14 drmeister So I created template_Array and template_SimpleArray that provides those methods

15:09:14 drmeister Then for each specialized array I subclass template_Array and template_SimpleArray and I have a non-simple and a simple specialized array.

15:10:04 drmeister Also there is template_SimpleVector that provides the behavior for the simple vector for that type.

15:12:00 Bike I'm thinking in terms of what should be done in llvm. say you have a simple mdarray or whatever it's called. then a read is just (mdarray->underlying)[row_major_index] regardless of the element type (because underlying has that). right?

15:12:02 drmeister So really, there are just three C++ classes that need to be dealt with: template_SimpleVector, template_SimpleArray, template_Array - it is only those that layout structures in memory.

15:12:17 Bike maybe bit arrays are weird...

15:12:23 drmeister bit arrays are weird

15:12:39 drmeister So are nibble and crumb (2bits)

15:13:04 drmeister bit, crumb and nibble arrays use the same underlying machinery

15:14:24 drmeister But from LLVM - for the arrays that work with everything else it's not too bad. The layout is defined by the template_SimpleVector, template_SimpleArray, template_Array template classes.

15:14:37 drmeister template_SimpleArray and template_Array have identical layout.

15:16:32 drmeister It's even simpler. There are only two memory layouts - for multi-dimensional/simple/non-simple arrays it is defined in MDArray_O and...

15:29:10 drmeister This summarizes the layout - there are just two

15:29:11 drmeister http://i.imgur.com/FwoEjnw.png

15:32:59 Bike well, right, i guess i was just confused why there are so many classes for teh two layouts, since the XXXX is only in the latter.

16:20:31 drmeister The only difference between a simple-array and a non-simple array is that the simple-array guarantees that the array it contains is not a displaced array

16:22:13 Bike right.

16:23:53 drmeister So - yeah - it's a lot of code for that guarantee...