freenode/#clasp - IRC Chatlog
Search
17:08:52
stassats
have you noticed the RAM prices recently? it's all the clasp users buying more memory
18:43:32
pfdietz
I doubt quicklisp loads all the installed systems, as they are not all mutually compatible.
19:40:47
drmeister
I moved inline.lisp and fli.lsp up to compile them early in the parallel build I'm building it now.
19:41:28
drmeister
The real problem was inline.lisp. It takes the longest and it was started last - so the build system spends all the time building inline.lisp on one core.
19:43:26
drmeister
Well, when inline.lisp is last it means that everything else finishes while inline.lisp is compiling and then it spends a lot of time running on a single core.
19:43:56
drmeister
It's not an issue for fli.lsp because it was being started fairly early. Hang on...
19:45:13
drmeister
Compiling [112 of 439 (child-pid: 84660)] /Users/meister/Development/dev-clasp/src/lisp/kernel/lsp/fli.lsp
19:45:30
drmeister
So fli.lsp is the 112th file to start compiling but it's the second last to finish.
19:46:36
frgo
Yes, sure - so move the ones taking longer up the chain so that some cores are busy while other cores can take on "smaller" compiles.
19:46:39
drmeister
It's a load balancing thing. Ideally it builds on all cores right up to the end and then all cores finish at the same time.
19:53:48
drmeister
Hmm, even when I start inline.lisp as the first compilation - everything else finishes on all the other cores before inline.lisp finishes.
20:13:01
drmeister
It's way more balanced when I move inline.lisp to be the first file to start compiling
20:13:57
drmeister
I didn't do anything fancy - in clasp-builder.lisp I added a move-to-front command and then explicitly move fli.lsp and then inline.lisp to the front of the queue.
20:14:09
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/clasp-builder.lsp#L663
22:37:35
drmeister
Compile-file seconds real(1399.9) run(1399.8) llvm(494.9) link(396.4) (llvm+link)/(real+link)(50%)
22:38:32
stassats
i would guess llvm does involve some exponential algorithms, everybody is complaining about its speed
22:45:46
drmeister
I just looked at the inline.ll file - it has many, many, many calls to INITIALIZE-INSTANCE.
22:56:33
drmeister
Those are the functions that build the AST nodes for CADR and at the bottom (line 251) is the code for CADR
23:03:45
drmeister
These inline.lisp^242^TOP-COMPILE-FILE.xxx functions are invoking ALLOCATE-INSTANCE and INITIALIZE-INSTANCE in a really inefficient way.
23:11:37
stassats
i'm not that into the details, but it's 2018, you can be as inefficient as you want and still not approach 10 minutes
23:12:35
drmeister
It's not the time to run the result - that happens every time at startup and it's a fraction of a second. It's the time to generate the llvm-ir, optimize it, lower it to native code.
23:14:57
drmeister
That's llvm - it's not designed to be fast (as the folks at the llvm-dev meeting told me over and over and over again).
23:18:22
stassats
i imagine it's like expanding a macro into a lot of direct calls? can it be a bit loopy and do things at runtime?
23:20:40
drmeister
I don't think so. For instance - this is what you get for a (initialize-instance obj ...)
23:24:49
stassats
are you expanding into a code that contains lots of (initialize-instance obj ...)?
23:25:16
drmeister
Not at this point. Line 43 gets the INITIALIZE-INSTANCE symbol, line 44 looks up the fdefinition, lines 45 to 60 get the arguments, 61-64 get the function pointer, 65 makes the call.
23:26:23
drmeister
This is the code that is run at startup - it draws everything from the CONTAB vector, which contains GC roots.
23:28:46
drmeister
I think I can substitute this mess with something like ltvc_indexed_call(@CONTAB17270,155,162620,157,162621,180,162624...)
23:31:18
stassats
so, you're dumping clos objects, but do you really need to call initialize-instance on startup? can you reconstitute by just dumping whatever is in its slot vector?
23:32:27
stassats
can you dump a call to allocate-instance + dump a slot vector + set-instance-slot-vector?
23:33:49
drmeister
That's essentially what I'm going to do. The arguments are all in this massive vector. So I'll just reference them.
23:35:41
stassats
or maybe instead of dumping (list (make-instance ... (make-instance) )) dump '((some-kind-of-marker (some-kind-of-marker))) and then walk that tree at startup time and call allocate-instance in place of some-kind-of-marker
23:37:38
drmeister
That would require a bit more thought. I'm gonna try this ltvc_indexed_call thing - it's more inline with what I've already got.
23:38:51
drmeister
It will, we will only do the work to generate one llvm instruction and llvm will only have one instruction to deal with rather than the 50 or so it currently does.
23:39:35
stassats
but you are expanding into code that does (list (make-instance ... :x (make-instance)))?
23:44:11
stassats
i'd uses some binary format for dumping constants, but even just printing into a string and reading from it would be better for starters
23:47:31
stassats
but 2.6 M lines of ast-creating compiled code, yeah, no compiler can chew through that
0:51:19
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cmp/cmpliteral.lsp#L270
0:52:06
drmeister
I was thinking that I could check the head and if it's 'INITIALIZE-INSTANCE I could convert it to a special case call.
0:57:46
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cleavir/inline-prep.lisp#L71
0:58:43
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cleavir/inline-prep.lisp#L71
0:58:52
Bike
we could do make-load-form-saving-slots, in which case it does a bunch of (setf slot-value) and slot-makunbound, the latter of which is maybe not necessary
1:00:08
drmeister
It's still going to generate a function for the initialize part of make-load-form - no?
2:08:12
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cmp/cmpliteral.lsp#L278
2:08:30
drmeister
Here: https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cmp/cmpliteral.lsp#L269
2:13:20
drmeister
There is an entire function being created for each allocate-instance and again for initialize-instance from this...
2:13:32
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/cleavir/inline-prep.lisp#L70
2:16:16
Bike
Could we reasonably special case the situation of a creation or initialization form being only a function call?
2:17:22
drmeister
(with-open-file (*debug-io* "/tmp/output.log" :direction :output) (compile-file "sys:kernel;cleavir;inline.lisp"))
2:20:27
drmeister
make-load-form - you pass it an object as the first argument and it gets substituted in the create and initialization forms that are returned - correct?
2:40:38
drmeister
Yeah - that's still not enough - you need to compile the make-load-form create and initialize forms so that they compile other make-load-form forms.
2:42:02
drmeister
A simple thing I could do is suppress the spilling of register arguments - that's not needed at all.