freenode/#clasp - IRC Chatlog
Search
14:36:09
drmeister
The problem I fixed last night I found by listening to the compiliation warnings.
14:39:53
drmeister
I'm noticing something weird though. It's not clear yet, but when I run clasp under the debugger it's like linking doesn't work properly. I get .fasl files with zero length and that's a show stopper.
14:41:14
drmeister
I can't even fathom why that would make a difference. If you do try to run clasp when it is connected to the debugger and you see that - can you confirm it?
14:42:55
drmeister
Let me put it this way - if you are tempted to run under the debugger because clasp backtraces are nonexistant and you start seeing fasl files with zero length - don't panic - I've seen the same thing. When I don't run under the debugger - I haven't seen it.
15:58:57
drmeister
Clasp is working pretty well - cl-jupyter builds and runs - that means Clasp compiles and compiles a whole mess quicklisp systems.
15:59:39
drmeister
I want to get up to the latest sicl - this means figuring out what is going on with that cc_writeCell call that is failing.
16:01:05
drmeister
If we could get a framework for adding inlined code for AREF then I can keep adding to it.
16:03:22
drmeister
What I'm thinking is if you could help me get started on the inlining and code generation stuff then I can keep working on that alongside other things that I need to implement.
16:03:57
Bike
like, mapcar and friends can be inlined and that's not even anything to do with cleavir, for one
16:04:32
drmeister
Meanwhile you could continue working on kildall and other compiler optimization while taking a look at the cl-jupyter notebooks and jupyter widgets stuff.
16:05:17
drmeister
Kevin and Hana will be working on that full time - but they don't know much Common Lisp. So if you could help them there that would be great.
16:05:33
drmeister
I think we can get some starter code put together for them and let them expand on it.
16:06:29
drmeister
I have a rough idea of what is involved to implement widgets in Common Lisp. I know where the Python whos functionality we are copying is and I have hacked it to dump logging information of the communication that we can work to replicate.
16:07:27
drmeister
mapcar can be inlined? Neat. I recently fixed them so that they wouldn't CONS - inlining would be even better.
16:08:17
drmeister
Right. My old implementation was before I figured out how to do more dynamic stuff on the C++ stack.
16:08:20
Bike
yeah, that sounds fine. i like teaching. and rapidly learning jupyter while having to explain it sounds like a good way to make it solid
16:09:58
Bike
just basically (mapcar #'foo bar baz) = (loop for x in bar for y in baz collect (funcall #'foo x y)).
16:12:23
Bike
and if loop doesn't work you can just write it out with dotimes or tagbody or whatever. loop's just the intuitive way to do it.
16:13:15
drmeister
It should be fine - mapcar is only used three times in loop2.lisp and not in any way that would cause problems.
16:21:58
drmeister
I think of it like auto-compile.lisp turns on Cleavir and that compiles inline.lisp and generates lots of inline definitions and compiler-macros and then it recompiles EVERYTHING with those definitions.
16:23:52
drmeister
General announcement: Folks - it's no big secret but now that it's pretty much finalized... Bike is coming to Philadelphia to work over the summer on Clasp/Cleavir. I have two other undergraduates starting this week and next who will start by incorporating widgets into cl-jupyter.
16:26:46
Bike
i was looking at ccl's macros too. but a lot of them use implementation guts cleavir doesn't have yet. or won't, like all that bit bashing
16:27:53
Bike
funcall might be better as (cond ((typeq ,fsym function) ,fsym) ((typeq ,fsym symbol) (symbol-function ,fsym)) (t (type-error ...)))
16:28:44
Bike
there are about a dozen functions that should expand to typeq, but it won't do any good until the thing with booleans is rsolved, so i'll skip those
16:29:14
drmeister
That funcall - you mean instead of the (if (cleavir-primop:typeq ,fsym function) ...
16:31:22
Bike
well, you'd have e.g. (defun vectorp (o) (if (typeq o vector) t nil)), and then if you had (if (vectorp x) ...) it would expand to (if (if (typeq x vector) t nil) ...), but cleavir doesn't know how to collapse that into (if (typeq x vector) ...) like it should be
16:41:12
drmeister
I have a feeling that there is a lot of array manipulation going on. Cando needs a lot of array manipulation.
16:41:36
drmeister
Cando now has lots of specialized arrays. You've implemented a better cleavir-primop:aref.
16:41:51
drmeister
I'd like to put it all together and start generating inlined code for array operations.
16:44:57
drmeister
https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/cleavir/translate.lisp#L530
16:46:50
drmeister
That is this: https://github.com/drmeister/clasp/blob/dev/src/llvmo/intrinsics.cc#L269
16:49:25
Bike
there are some others that should work in implementation ways. e.g., apply should work like funcall in that the coerce-fdesignator bit is inlined, but then it calls some kind of "%apply" instead of a cleavir primop.
16:50:50
drmeister
I turned backtraces on for bclasp - but this is cclasp compiled by bclasp compiling cclasp.
16:52:32
drmeister
But the very first order of business for me is to get backtrace frames back into cclasp.
16:55:38
Bike
What is (funcall (compiler-macro-function 'mapcar) '(mapcar (lambda (x) x) '(1 2 3 4)) nil)?
16:56:09
drmeister
(LOOP CORE::NAMED #:UNUSED-BLOCK14396 CORE::WITH #:MAPCAR-FUNCTION14394 = (LAMBDA (X) X) CORE::FOR #:MAPCAR-ARGUMENT14395 CORE::IN '(1 2 3 4) CORE::COLLECT (FUNCALL #:MAPCAR-FUNCTION14394 #:MAPCAR-ARGUMENT14395))
16:56:42
Bike
does (loop with f = (lambda (x) x) for arg in '(1 2 3 4) collect (funcall f arg)) work?
17:00:59
drmeister
I changed (define-compiler-macro funcall ... to (define-compiler-macro funcall-foo ...
17:05:40
Bike
Uh... well I don't know exactly where in the depths it is. But try macroexpanding (define-compiler-macro identity (form) form)
17:06:10
Bike
there's something like (if (and (eq (car #:g) 'funcall) (eq (caadr #:g) 'function)) ...)
17:08:53
Bike
compiler macro functions are supposed to deal specially with (funcall ...) forms so defining a compiler macro for defun is a bit harrowing, i guess
17:10:21
Bike
i'm not actually sure how to fix it since the name the macro is for doesn't seem to be available
17:12:43
drmeister
Rather - I've spent a lot of time debugging the compiler when it hits 'destructure'
17:13:13
Bike
basically you want that to be (if (and (eq (car ,whole) 'cl:funcall) (consp (cadr ,whole)) (eq (caadr ,whole) 'cl:function) (consp (cdadr ,whole)) (eq (cadadr ,whole) whatever-function-this-is-for))
17:22:50
drmeister
So... a third, maybe &optional, argument to sys::destructure and fix all of the callers up to define-compiler-macro - if indeed it calls this.
17:23:47
drmeister
define-compiler-macro is here - it calls sys::expand-defmacro... https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/lsp/evalmacros.lsp#L172
17:27:47
drmeister
So I could add a second optional argument to expand-defmacro to pass the macro name and pass that to destructure.
17:42:59
drmeister
I know a guy who wrote a compiler - he has more than 100 employees. He's been looking for years for someone who can take over writing compiler code.
17:43:46
drmeister
Ok, this is going to build for a while - I'm going to go back to working on my proposal.
17:45:08
drmeister
I had commented out the (define-compiler-macro funcall ... ) and tried to rebuild - it choked on something to do with returning multiple values - so there may be more rough edges in there.
17:46:12
drmeister
The way I see it, this should work smoothly to incorporate compiler macros and inlined functions. I've been working towards this for a long time.
17:47:09
drmeister
This code gets compiled/tested at the end of a long build cycle - so it's slow work to fix these bugs. Thats why I keep vacillating back and forth between incorporating these things and trying to speed up cclasp compilation.
17:48:00
drmeister
I think we've barely scratched the surface in terms of speeding up clasp using these features.
17:48:52
drmeister
I want to use inlining and custom code generation a lot in Cando. I think there are lots of places where the processor is underutilized in scientific computing because of code that doesn't take advantage of registers/cache the way it could.
18:14:00
drmeister
Bike: For this to work: (define-compiler-macro values (&rest values) `(cleavir-primop:values ,@values))
18:17:01
drmeister
I've got the mapcar and friend and funcall compiler macro installed at the moment and I'll let that build through
18:18:17
drmeister
https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/cleavir/translate.lisp#L378
18:20:53
drmeister
Not sure. I wiped out the terminal buffer before I started the latest build - so I can't go back until this one is done.
18:21:52
Bike
Maybe it doesn't work right if there are multiple inputs. I don't think an ftm with multiple inputs is generated without the values primop.
18:22:16
drmeister
On the plus-side - the (define-compiler-macro funcall ...) seems to be working fine.
18:23:09
drmeister
On this "Maybe it doesn't work right if there are multiple inputs. I don't think an ftm with multiple inputs is generated without the values primop."
18:24:14
Bike
fixed-to-multiple takes any number of lexical inputs and has one values output. but AST-to-HIR doesn't actually produce any fixed-to-multiple instructions with more than one input, except with the values primop.
18:26:42
drmeister
Ok, well, I know a little bit more about what I'm looking for when it hits that error again.
18:27:35
Bike
i don't know how multiple values work in clasp. how do you handle something like (multiple-value-call foo (values a b) (values c d)) where you have two values locations at once?
18:29:14
drmeister
IIRC It evaluates (values a b), copies out the values, evaluates (values c d), copies out the values ...
18:30:07
Bike
hm, looks like multiple-value-call on clasp only knows how to deal with having one argument form. if there's more than one it does core:multiple-value-funcall
18:30:16
drmeister
Beach and I had a long discussion about this a long time ago - we decided it was workable.
18:30:52
drmeister
Right - it's optimized for one and calls multiple-value-funcall for all other cases.
18:32:00
Bike
and this probably wouldn't run into that. it should just convert "FDEFINITION 'values -> FUNCALL 'values" into "F->M"
18:41:22
drmeister
The whole if statement https://gist.github.com/drmeister/be0164350c965938c5aac52308d6115e#file-gistfile1-txt-L25
18:41:56
Bike
well, basically, if you have a compiler macro on foo, you're supposed to be able to call it both on forms like (foo ...) and like (funcall #'foo ...)
18:42:20
Bike
in the latter case the arguments to the compiler macro function, the ..., have to be the cddr of the form instead of the cdr
18:42:56
Bike
since we're allowing funcall, it has to work on (funcall ...) and (funcall #'funcall ...), not that that's likely to happen
18:43:27
Bike
If the cmf on funcall gets (funcall #'foo ...) it should just take the cdr, not the cddr
18:43:27
drmeister
So is this time when the error occurred - is this the first time that compiler macro was activated?
18:45:33
Bike
in the latter case (consp (cadr ,whole)) would be false so it wouldn't get to the ,macro-name.
19:00:44
drmeister
(error "Finish implementing return-value-elt - you need to use gep to index into the array")
19:03:49
drmeister
Two registers are returned from a function. I use one for the first return value and the second for the number of return values.
19:05:33
drmeister
Six integer/pointer arguments are passed in through registers and only two out? And they don't even overlap. I think it's AX and DX that are returned.
19:06:18
drmeister
This will take more quiet brain power than I have to dedicate at the moment to write. So I'll comment it out with a note and pick it up in the week.
19:07:29
drmeister
This should be short - but it needs time to generate code, look at it carefully, test it...
19:09:03
drmeister
I should write a simple %GEP instruction that verifies the type of the argument and provides a list of indices into the argument.
19:11:56
drmeister
In llvm-ir I have to ensure that the IR does the same thing as the C++ code. I also have to avoid the nightmare of the llvm-IR thinking the data structure is organized one way when in C++ it's organized another way.
19:13:14
drmeister
I have sanity checks to ensure that the size of key data structures are the same in C++ and CL.
19:14:21
drmeister
What I need is a way to more facile way of specifying the size and shape of these data structures in C++ to communicate them to the CL code.
19:14:47
drmeister
Here is the multiple-values structure in CL https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/cmp/cmpintrinsics.lsp#L291
19:15:23
drmeister
https://github.com/drmeister/clasp/blob/dev/include/clasp/core/multipleValues.h#L35
19:16:40
drmeister
The way they are expressed in the two languages are very different. It's easy to make a mistake - that would be disastrous.
19:17:11
drmeister
That's what the intrinsics.cc intrinsics take care of - they access data structures and return results.
19:19:24
drmeister
If I generate IR directly in the CL code then it works with COMPILE and COMPILE-FILE and it isn't subject to llvm inlining - which I don't really understand - but seems to work pretty well.
19:27:54
drmeister
Rather than continuing to yack about how hard it would be - I just implemented it.
19:34:08
drmeister
It's using the return value from this: (primitive-nounwind module "cc_multipleValuesArrayAddress" %t*[0]*% nil)
20:19:50
drmeister
I wrapped a handler-bind around the build process - so now I get error messages. Now it's compiling with the values compiler-macro