freenode/#clasp - IRC Chatlog
Search
14:27:35
drmeister
I hit a bit of a roadblock in image save/load with Boehm - it's still possible in MPS.
14:29:16
drmeister
The problem with Boehm is 1. I need to put data and code close together for RIP relative addressing 2. So I need to put literal vectors into GC managed memory along with lots of code 3. I want the GC to ignore the code - it contains no pointers and I expect it will slow things down 4. So with Boehm I need to write my own marking procedure.
14:30:55
drmeister
5. There's a comment in the boehm header file that appears to have huge implications for this - I don't know how to get around it. They say that the marking function should only mark about 100 bytes of pointers before returning. I have no idea how I do that with things like simple-vector. I need to investigate what ECL does in this case.
14:34:13
drmeister
"do it in smaller pieces"???? How would I save the how much marking work I've done for any particular object and then pick up later?
14:37:08
drmeister
I don't see how it can work - other than for objects that it doesn't handle everything is scanned conservatively.
14:42:11
drmeister
What I need is something where I can say "for this object - only scan up to this point".
14:45:36
drmeister
ECL marks fields of well defined objects in a simple way in its marking function - ECL doesn't worry about overflowing the mark stack because the well defined objects only have a few fields to mark.
14:46:02
drmeister
I'm assuming for the moment that when it doesn't handle an object that boehm marks the entire object conservatively.
14:46:57
drmeister
If Code_O objects had small literal vectors (not something I can guarantee) then I could handle them like a well defined object.
14:47:36
drmeister
I should check the size of literal vectors. They are unbounded and there is no easy way to bound them.
14:49:44
drmeister
I've emailed Ivan Maidanski - the maintainer of boehm to see if he can give me pointers on this. https://github.com/ivmai
14:50:30
drmeister
Could you check the log for the last 30 min? I posted some stuff that I'd love to have your input on.
14:55:23
karlosz
drmeister: i'm not too familiar with how boehm scanning works, but even though vectors are unbounded in size, the size is still stored in a known position in the object, so couldn't the marking procedure just use that? or is the problem that you'd need to write a custom procedure in that case to parse the objects?
14:56:51
drmeister
The problem is they say I can only do a certain amount of marking work and if I don't fully mark everything that I push the object back onto the marking stack.
14:57:12
drmeister
How do I keep track of how much marking work is done if the marking function returns after doing a certain amount of marking work.
14:57:44
drmeister
Or does it not return and it calls a function after doing a certain amount of marking and then it continues?
15:01:36
karlosz
well, since you're supplying the mark procedure, can't you can keep track of the marking work there?
15:02:08
Bike
the marking procedure you define is supposed to do only a small amount of work and is called rpeeatedly by boehm while it's marking
15:02:09
drmeister
jackdaniel: In the precise GC mode ECL doesn't appear to mark things like the contents of vectors or hash-tables. Where does that happen?
15:04:27
karlosz
hm, i guess otherwise you'd have to add a field to each object for the gc to keep track of the state
15:05:59
karlosz
the comment there also seems to suggest that it's not actually necessary to break it up into smaller chunks
15:07:44
karlosz
in which case only varyobj sized objects would need to keep track of how much marking has been done, assuming that does incur space overhead of some kind
15:10:14
jackdaniel
drmeister: I'm not looking very carefully at the code, but cl_object_mark_proc seems to mark the contents of vectors
15:10:41
jackdaniel
self.t contains vector data, so the array pointer is marked, similar with hash.data
15:17:00
drmeister
What does a hash table entry look like? Is it a key/value pair? Does hash.data point to a vector of key/value pairs that are GC managed pointers?
15:18:18
drmeister
karlosz: I have the same feeling that it might be an optimization and not actually necessary - but I'm anxious about that.
15:18:45
beach
Does "precise mode" mean what I think it does? And if so, how much stack-scanning code does the client need to supply?
15:19:03
drmeister
If the marking function is called repeatedly on the same object (I can check that) then I could keep the state of marking in thread local storage.
15:21:39
drmeister
You are supposed to push the object back on the marking stack if it isn't finished. Maybe that ensures that it is the next object to be passed to the marking function. Then this wouldn't be so much trouble.
15:23:04
drmeister
I haven't yet gotten boehm to call my marking function - so I'm missing something.
16:17:17
Bike
the stuff i had to speed up typecase probably isn't immediately suitable for subtypep because it's simplifying in different ways
16:19:00
Bike
and of course for type inference we can lose a lot of precision. karlosz, what kind of types do you think will come up most? i mean we can probably lose stuff like satisfies, right?
16:21:11
Bike
optimizing just cl:subtypep generally might be something to avoid. we need it early which severely limits our options, and it has a lot of distinct use cases
16:24:55
Bike
handling numbers, arrays, functions, and values types and collapsing everything else to classes might be good. maybe conses, not sure
16:32:05
Bike
although i'm not sure of the imprecision semantics. if a function is defined as taking a (satisfies foo) argument and we collapse that into T and eliminate the test that seems unfortunate
16:44:46
karlosz
Bike: from instrumenting subtypep the main problems right now are that it doesn't handle cons types at all
16:45:12
karlosz
so everytime subtypep (canoncialize-type) encounters a cons type it does a C++ unwind tanking compiler performance
16:46:30
karlosz
i think right now subtypep unwinding on cons types contributes to 10-20% compiler overhead at the moment on certain system
16:47:42
karlosz
the next thing that subtypep encounters a lot are unknown types coming from type declarations in defclass's and stuff like that
16:48:11
karlosz
about 90% of the types we encounter in self build that we can't handle right now in subtypep are CONS types
16:50:39
karlosz
to implement cons types properly we'll need to throw the bit vector implementation out and replace it with a normalizing interpreter like the one in sbcl i think
16:51:20
Bike
well what i mean here is we don't need to use cl:subtypep in the compiler, we can have the ctype methods do something more specifically tailored to our needs
16:55:01
karlosz
Bike: it seems hard to make subtypep in cleavir handle cons types without invoking cl:subtypep
16:58:13
karlosz
yeah, that's why i thought it might be difficult to handle cons types outside cl:subtypep
16:59:07
karlosz
so essentially this restrictred subtypep will just handle compund and specialized types and punt to cl:subtypep for the more rpimtiive sutff
17:00:17
Bike
yeah i meant like, specializing the methods rather than using the default implementation
17:01:49
karlosz
ironclad dumps out a lot of bignums, i'm guessing for all the unsigned-byte 64 declarations
17:04:32
Bike
if you look at core__next_primitive_string in bignum.cc you can see how to get at the raw data. it's pretty simple. i guess we'd want functions to read and write it from a stream, or something?
17:05:51
Bike
next-primitive-string prints out the words for the bignum. you can try like (core:n-p-s (* m-p-fixnum m-p-fixnum))
17:09:15
karlosz
well i was just thinking of dumping the bignum by dumping the bytes directly into the stream with a header for how many bytes
17:10:19
Bike
the data is just a size (which is sometimes negative to indicate that the bignum is) and an array of mp_limb_t, which is uint64_t or so
17:21:55
Bike
ok it looks like sbcl actually does do what i was thinking of doing but i was worried about the performance of
17:42:50
Bike
the definition is in src/core/byte-code-interpreter.cc. it's generated by that stuff at the end of cmpliteral
18:00:38
karlosz
i guess the way the mp interface is written it would be better to dump out the limbs and then load them in
18:09:10
kpoeck
(mp:process-run-function :foo #'(lambda() (mod 1 0))) does not hang, but cpu is at 100%
18:10:31
kpoeck
after evaluating (mp:all-processes) 3 times, I finally get the expected DIVISION-BY-ZERO exception and cpu lod is back down
18:18:16
Bike
well, maybe it is doing an FPE and it ends up in our signal handler and that hangs for whatever reason
18:35:19
kpoeck
If i put the following in the beginning of clasp_truncate, the error goes away (not surprising)
20:24:03
karlosz
i changed ltvc_make_next_bignum to take a length argument instead of a string and now i get ("Mismatch of ltvc read types read '7' expected 's'")
21:16:51
drmeister
A light is starting to come on above my head. This mysterious stuff in the boehm gc_mark.h file is starting to look less mysterious.
21:17:56
drmeister
It's a tagging scheme for objects in Boehm. It's got tag bits and something like a stamp (proc_index) and something they call (env).