freenode/#clasp - IRC Chatlog
Search
10:36:50
drmeister
::notify kpoek There are instructions about a symbolic link that you may need to use on OS X to get rid of those missing header file messages: https://github.com/drmeister/clasp/wiki/Clasp-0.5-Build-Instructions
12:42:08
Colleen
Bike: drmeister said 8 hours, 41 minutes ago: cclasp works with slime in :spawn mode - we are back in business.
12:42:08
Colleen
Bike: drmeister said 8 hours, 16 minutes ago: We aren't taking full advantage of these new fastgf optimized slot accessors. ECL's std-class-optimized-accessors are still using hash tables. This is revealed by profiling allocation of CLOS objects.
12:52:30
beach
I would like to make the following changes to Generate-AST, CST-to-AST, and AST-to-HIR:
12:52:43
beach
First, Generate-AST and CST-to-AST should not enter the AST corresponding to a variable into the environment. Instead, it should enter an instance of a new class, that we may call VARIABLE and which is not an AST (so it has no source information).
12:52:49
beach
Next, Generate-AST and CST-to-AST should create a new AST for each variable reference, and instead of the name of the variable, the AST should contain a reference to the unique VARIABLE instance. That way, each reference to a variable has its own source location.
12:52:50
beach
To make that work, some changes need to be made to AST-to-HIR, but I think they are fairly limited. It looks like FIND-OR-CREATE-LOCATION needs to be altered so that it uses the VARIABLE to look up the location, rather than using the AST directly.
12:54:51
beach
I should have said: Generate-AST and CST-to-AST should create a new instance of a LEXICAL-AST for each lexical variable reference.
12:57:25
beach
Unfortunately, this change will break all the CST-to-AST tests that I so painstakingly created. :(
12:58:08
beach
But the change is necessary so that we can keep correct source location for each variable reference.
12:58:32
Bike
I think right now we just put... symbols? cleavir env info? in lexical asts, so having dedicated variables makes more sense to me
12:59:43
Bike
so, in the paper i didn't see reference to when fastgf could be applied. can it only be used on direct instances of standard-generic-function, or is it just that for indirect instances we can only make a dispatching tree out of what we get from calls to compute-applicable-methods-using-classes?
13:02:33
beach
The MOP does not allow the result of compute-applicable-methods to be cached as I recall.
13:03:03
Bike
but we can do something like that for standard-generic-function since it's standard and methods can't be defined etc.
13:05:28
Bike
because it would just be caching on the actual objects, which wouldn't recur so much, i guess. a user could define a compute-applicable-methods method that uses identities different from classes or eql specializers.
13:06:42
Bike
anyway, the other thought i had was that it might make sense to compile effective methods directly into the discriminating function. i think that's /allowed/ anytime since nothing seems to state effective methods have to be functions and they aren't accessible, and i think it could be a good idea because most of the time they're pretty simple. any thoughts? i looked through sicl and it doesn't seem to do this.
13:08:20
beach
That ought to be possible, yes. There are many optimizations that I have not yet done in SICL.
13:08:31
drmeister
FYI Clasp's fastgf implementation does test for EQL specializers - and does check if the effective method function can be memoized here:
13:08:31
drmeister
https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/clos/closfastgf.lsp#L527
13:10:37
drmeister
https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/clos/closfastgf.lsp#L522
13:17:23
beach
That's not a good reason in my book. It distracts the maintainer and makes him or her look for a more profound reason, such as the existence of special tools.
13:18:49
Shinmera
I'm sure he cares, but is often too deep in things and busy with stuff to think about making thins well readable.
13:19:58
drmeister
What Shinmera said - I'm usually in a headlong rush to get things working because I'm working under extreme pressure - largely self imposed.
13:19:59
beach
Well, a large part of programming is being meticulous and pedantic. When I see messy comments or messy code, I don't get the impression of good quality code.
13:21:06
Shinmera
Some people tend to try and push cleaning things up to a later point in favour of quick progress in the moment. That doesn't mean they don't care about other people reading their code.
13:23:34
Shinmera
That's beside the point. I'm merely arguing that unclean code does not imply that they don't care.
13:27:04
Bike
well, if nothing else, i'm going to look over the fastgf code in the next few days to make adjustments, i can rewrite comments as i do
13:27:30
Bike
i tend to go through erasing #+(or) blocks we're not going to need and so on whenever i make changes in a file
13:28:17
Bike
but as far as not cleaning up though, i just noticed this yesterday and it's probably older than i am https://github.com/drmeister/clasp/blob/master/src/lisp/kernel/lsp/loop2.lsp#L127-L137
13:33:29
Bike
i think it was like, you'd go through the list normally, and then when you hit a cdr-coded end you go through like an array
13:35:20
Bike
i think the idea of copylist* was that if you copied a list you were going to rplacd the end of later, cdr coding didn't make sense, so you'd want to use a normal list for the copy
13:37:26
drmeister
Anyway Bike - I noticed that there are still slot accesses that are going through hash-tables - probably because the code I wrote doesn't apply the optimization to anything other than standard-reader-method.
13:38:48
drmeister
Also, the optimized accessors aren't fully inlined yet - I've been rearranging code to achieve that.
13:39:03
beach
I think CDR was defined to look what kind of cell the pointer points do. If it points to a CDR coded cell, then instead of making a memory read, it just adds 1 to the address.
13:41:08
drmeister
Also... quicklisp fails to build because I think one of the other optimized accessors is being generated - a CONS instead of an index for the slot. We talked about this one. I think we are about to find out what they are for.
13:41:55
beach
You can't use that technique for slots with allocation :CLASS if that is what you mean.
13:42:33
drmeister
Yeah - the ECL CLOS code is involved here as well, it has it's own way to optimize accessors. I don't want to break it because then I can't use ECL's dispatch during startup.
13:42:43
Shinmera
You theoretically could if you make sure to copy the value to all instances when it's updated.
13:43:34
drmeister
Got it - ok - I have code to deal with it but it's not fully operational yet because I didn't understand what it was supposed to do.
13:44:21
Shinmera
Speaking of that, how does the make-instances-obsolete/update-instance-for-redefined-class mechanism works? Does the instance itself check if the class has been updated?
13:44:54
drmeister
There's a code path for it that ends in a construction site and a sign that says "Figure out what you are supposed to do here".
13:45:01
Shinmera
If I recall what the CLHS says it's not specific about /when/ u-i-f-r-c is called.
13:45:30
beach
Shinmera: The class gets a new unique number, and all generic functions that dispatch on that class are invalidated.
13:46:02
beach
Shinmera: That way, as soon as an object is being used in a non-trivial way, it fails the dispatch and is updated accordingly.
13:47:40
drmeister
Shinmera: That's one of the really nice and only recently appreciated features (by me) of fastgf.
13:48:07
Shinmera
beach: Could the cache not miss for another reason? If so, wouldn't that incur unnecessary calls to u-i-f-r-c?
13:49:11
beach
Shinmera: This trick relies on the fact that, in order to access the contents of an object, some generic function must be called.
13:49:32
Shinmera
I used to keep explicit lists of weak pointers from the class to its instances before I realised I could use u-i-f-r-c for most of what I had to do.
13:51:08
specbot
standard-instance-access: http://metamodular.com/CLOS-MOP/standard-instance-access.html
13:54:15
Bike
i think we might still not support it on clasp, since someone in ECL there's an exported function with the effect
13:57:19
Bike
the first actual thing i should do once dev rebuilds is figure out source tracking for generic functions and methods
14:14:51
Bike
yeah. clasp tracks function source locations with, i think, a part of the macroexpansion
14:26:14
beach
OK, I think the best way is to eliminate Generate-AST first, so as to avoid redundant changes. For that, I need to retrofit some of the code I omitted in CST-to-AST, like inserting THE-ASTs and such.
14:26:51
beach
Also, I still have a call to EVAL in CST-to-AST, and that one still uses Generate-AST as I recall.
14:27:55
beach
I also need to provide some temporary compatibility in the form of a version of GENERATE-AST that calls CST-FROM-EXPRESSION and then CST-to-AST.
14:31:37
beach
So, then I can either go ahead and do the redundant change, or I can work on inlining local functions first.
14:32:48
Bike
just to make sure we're on the same page. i think your conception that LET should basically be a lambda call is correct. but as a practical matter, clasp function calls are slow and i'm pretty sure having every let be a function call would destroy our performance. this isn't a consideration for you, so since the bug is pretty rare, i don't mind staying a bit behind while cleavir's inlining is changed.
14:33:39
Bike
and i'll help with that if i'm able, but i'm still kind of confused about how to do it.
14:33:41
Shinmera
beach: Unfortunately so, yes. I'd be interested in working on Cleavir once I'm done with my Bsc, though.
14:35:49
Bike
ok. do tell me if there's something i can do though. i'd prefer clasp to not be buggy as well.
14:38:58
specbot
compute-class-precedence-list: http://metamodular.com/CLOS-MOP/compute-class-precedence-list.html
14:45:02
beach
I do think it's good that Clasp is adopting several techniques from SICL. That way, more people will be familiar with those techniques. It would be even better if we could figure out how to avoid code duplication and use the exact same code for both implementations.
14:46:36
Bike
for the standard library, you mean? for most of it clasp's is how it is in large part for bootstrapping reasons, which i can hopefully make inroads against
14:47:29
beach
Right then I was thinking of the code for computing the discriminating automaton, but yeah, library functions as well would be good.
14:54:16
beach
I think there are two reasons. For one thing, I think I have observed that drmeister wants to understand how everything works, and his way of doing that is to write it himself, even when there is existing code. The other thing is that there is some variation between how things are done in Clasp and in SICL, and it is more work to figure out how to abstract out the commonalities than it is to just write a special version.
14:57:57
beach
I could also have a look at my SICL code, see what is missing from it, check whether it contains any SICL-specific code, and try to separate the general part from the specific part.
15:06:43
Bike
right. it would also be kind of neat if we could have a specialized compiler for the kind of stereotypical code the automaton is made of. not a huge deal, probably...
15:12:04
beach
Yes, a lot of optimization passes could probably be skipped since the automaton does not contain any old code.
15:16:43
Bike
(segue) clasp has C++ functions cl:find-class and core:lookup-class. the latter takes the same arguments as find-class, and find-class just calls it immediately
16:06:39
Bike
it seems like it does from the description, but then it needs the possible user call-next-method, which they can define with make-method-lambda
16:06:42
beach
I have read the Common Lisp HyperSpec page for CALL-METHOD and MAKE-METHOD several times, and never was able to figure them out.
16:07:26
Bike
hardly anyone uses custom method combinations, so it's not very worrying, but still vexing
16:13:19
beach
I think if I read that page again after having ingested my morning coffee, I might be able to understand it this time. But not after a long day of work.
16:23:33
Bike
ok, so clhs says specifically "The form used with make-method is evaluated in the null lexical environment augmented with a local macro definition for call-method and with bindings named by symbols not accessible from the COMMON-LISP-USER package.", so no call-next-method
16:23:57
Bike
but then (call-method (make-method form) ...), which is explicitly allowed, is just form
16:35:16
Bike
oh hey, and the paper mentions that thing i was worried about yesterday with compile time ensure-generic-function
16:37:08
Bike
defmethod needs a generic function at compile time in order to call make-method-lambda, so if you put a defgeneric in a file followed by a defmethod, the gf will actually be created at compile time by defmethod, and then reinitialized later at load.
16:44:35
Bike
hm, but done this way naive call-next-method uses apply. i'd like more optimizations before doing that
16:46:11
Bike
or... actually i guess all that's needed is a local compiler macro for the common case. or a global one that refers to variables magically
16:47:25
drmeister
Turning on the llvm-ir rewriting optimization in aclasp does slow it down - by about 3min.
16:48:45
drmeister
Just for yucks I profiled a loop in the interpreter - it spends more time doing GO than anything else. That's because GO uses stack unwinding - even local ones.
16:51:16
Bike
i think we could probably have make-method-lambda work like this instead of like in the MOP. i suspect it wouldn't work anyway since we use valists. there's something for later
16:58:03
drmeister
Hmmm, the profiling 'do-flame' command depends on the FlameGraph.git repo being installed in a hard-wired place - I need to change that still
17:13:47
Bike
drmeister: just to confirm, all this stuff in kernel.lsp about discriminating functions written in C is only used during startup?
17:41:14
drmeister
Other things that you may need are also stored in the Atom_O object. Only some of it is copied into the AtomTable_O structure.
18:05:51
Bike
i should make a script that finds functions that aren't exported or called because god do we have a lot of them
21:13:22
Bike
hey. shiho ran into some kind of problem on boot. and i have a question, where are C++ files in the build system? I want to try removing some
21:15:56
Bike
i'm removing the standard-optimized-reader/writer-method things as a prelude to removing the ECL dispatch system
21:16:08
drmeister
It really should just build - when I stopped by the CL code was building. The C++ is done at that point.
21:18:16
drmeister
Crap - it's not like I haven't removed a hundred C++ files. I'm kind of drawing a blank. Ask me tomorrow - I'll probably remember when I stop trying.
21:21:52
shiho
then I got Condition of type: SIMPLE-ERROR Constructing call to intrinsic cc_dispatch_slot_reader - mismatch of arg#0 value[((PREPARE-OP COMPILE-OP))], expected type #<POINTER-TYPE {}*> - received type COMMON-LISP::CONS Available restarts: (use :r1 to invoke restart 1)
21:23:29
drmeister
Bike: So you are removing ECL dispatch? There are a few functions in genericFunction.cc that are needed by fastgf.
21:24:59
Bike
the one in particular that i'm worried about is that there's a "CompiledDispatchFunction_sp" which is treated differently from a general function
21:26:09
Bike
I was hoping I could have set-funcallable-instance-function just work the same for all functions, but i suppose it's okay if it just differentiates that
21:26:25
Bike
hopefully i can just pass invalidated-dispatch-function as a function instead of a symbol, though
21:27:13
drmeister
Here's why - the first entry point takes the arguments from the calling convention and creates a Vaslist and rewinds it - it passes that to the second entry point.
21:29:12
Bike
I think I can remove ECL independently, and just leavea test for the compiled dispatch function type
21:37:10
drmeister
I already have the instance (and the value for the slot-writer) from the dispatch function - I don't need to read them out of the va_list again.