freenode/#clasp - IRC Chatlog
Search
13:04:16
kpoeck
I am getting spurious errors with find-class returning nil although the class is clearly defined
13:06:03
kpoeck
I have tried to defined a minimal test case in https://github.com/clasp-developers/clasp/issues/636
13:09:53
drmeister
kpoeck: Interesting about the spurious find-class issues - and you have a test - excellent!
13:10:51
drmeister
You may be used to using 'clasp', which is a symlink to 'cclasp-boehm', which is a single executable.
13:11:16
drmeister
'iclasp-boehm' contains only the linked C++ code and it loads the cclasp image as a fasl file.
13:11:27
kpoeck
The seg fault is not urgen, I have a second installation with older code that works fine
13:13:39
drmeister
Huh, I built the latest dev last night and I just counted the number of lines of llvm-ir using: wc `find . -name '*.ll' -print` --> 9,985,761
13:23:25
drmeister
That's a substantial reduction in code! Yet it says that it took more than 4 hours to build.
13:25:13
Bike
kpoeck: WITH_READ_LOCK will end at the end of the block it's in (so just after the "foundp =" there). it's raii stuff.
13:25:44
Bike
kpoeck: this particular function is organized like that because signaling an error while holding a lock is problematic
13:29:35
drmeister
Looking at the code I don't see how it could fail, even in a multithreaded situation, to report classes that are present.
13:30:05
Bike
i'm pretty willing to believe i'm missing something, when it comes to multithreading, though
13:31:11
drmeister
Ugh - actually I'm rebuilding cclasp - so I don't have anything to test with for a bit.
13:32:59
drmeister
If we have really knocked the amount of llvm-ir down by more than half then the build should be noticeably faster.
13:38:44
drmeister
I've seen wildly inconsistent build times before. I think it is something to do with macOS throttling processes that are in windows in the background.
14:26:44
drmeister
The build isn't any slower - it must have been power management issue that caused that 4h build.
14:27:26
drmeister
But inline.lsp and fli.lsp are taking a long time (707 and 866 seconds respectively).
14:27:52
drmeister
So it sits there doing serial compilation for a long time. I'm going to pull those two forward to start them first.
14:31:55
drmeister
But the overall build time was 1h7m - that's longer than the 27 min I saw before.
17:03:13
beach
I guess it could be. I am assuming Quicklisp loads all the installed systems, so the total amount of space could be significant.
17:08:52
stassats
have you noticed the RAM prices recently? it's all the clasp users buying more memory
18:43:32
pfdietz
I doubt quicklisp loads all the installed systems, as they are not all mutually compatible.
19:40:47
drmeister
I moved inline.lisp and fli.lsp up to compile them early in the parallel build I'm building it now.
19:41:28
drmeister
The real problem was inline.lisp. It takes the longest and it was started last - so the build system spends all the time building inline.lisp on one core.
19:43:26
drmeister
Well, when inline.lisp is last it means that everything else finishes while inline.lisp is compiling and then it spends a lot of time running on a single core.
19:43:56
drmeister
It's not an issue for fli.lsp because it was being started fairly early. Hang on...
19:45:13
drmeister
Compiling [112 of 439 (child-pid: 84660)] /Users/meister/Development/dev-clasp/src/lisp/kernel/lsp/fli.lsp
19:45:30
drmeister
So fli.lsp is the 112th file to start compiling but it's the second last to finish.
19:46:36
frgo
Yes, sure - so move the ones taking longer up the chain so that some cores are busy while other cores can take on "smaller" compiles.
19:46:39
drmeister
It's a load balancing thing. Ideally it builds on all cores right up to the end and then all cores finish at the same time.
19:53:48
drmeister
Hmm, even when I start inline.lisp as the first compilation - everything else finishes on all the other cores before inline.lisp finishes.
20:13:01
drmeister
It's way more balanced when I move inline.lisp to be the first file to start compiling
20:13:57
drmeister
I didn't do anything fancy - in clasp-builder.lisp I added a move-to-front command and then explicitly move fli.lsp and then inline.lisp to the front of the queue.
20:14:09
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/clasp-builder.lsp#L663
22:37:35
drmeister
Compile-file seconds real(1399.9) run(1399.8) llvm(494.9) link(396.4) (llvm+link)/(real+link)(50%)
22:38:32
stassats
i would guess llvm does involve some exponential algorithms, everybody is complaining about its speed
22:45:46
drmeister
I just looked at the inline.ll file - it has many, many, many calls to INITIALIZE-INSTANCE.
22:56:33
drmeister
Those are the functions that build the AST nodes for CADR and at the bottom (line 251) is the code for CADR
23:03:45
drmeister
These inline.lisp^242^TOP-COMPILE-FILE.xxx functions are invoking ALLOCATE-INSTANCE and INITIALIZE-INSTANCE in a really inefficient way.
23:11:37
stassats
i'm not that into the details, but it's 2018, you can be as inefficient as you want and still not approach 10 minutes
23:12:35
drmeister
It's not the time to run the result - that happens every time at startup and it's a fraction of a second. It's the time to generate the llvm-ir, optimize it, lower it to native code.
23:14:57
drmeister
That's llvm - it's not designed to be fast (as the folks at the llvm-dev meeting told me over and over and over again).
23:18:22
stassats
i imagine it's like expanding a macro into a lot of direct calls? can it be a bit loopy and do things at runtime?
23:20:40
drmeister
I don't think so. For instance - this is what you get for a (initialize-instance obj ...)
23:24:49
stassats
are you expanding into a code that contains lots of (initialize-instance obj ...)?
23:25:16
drmeister
Not at this point. Line 43 gets the INITIALIZE-INSTANCE symbol, line 44 looks up the fdefinition, lines 45 to 60 get the arguments, 61-64 get the function pointer, 65 makes the call.
23:26:23
drmeister
This is the code that is run at startup - it draws everything from the CONTAB vector, which contains GC roots.
23:28:46
drmeister
I think I can substitute this mess with something like ltvc_indexed_call(@CONTAB17270,155,162620,157,162621,180,162624...)
23:31:18
stassats
so, you're dumping clos objects, but do you really need to call initialize-instance on startup? can you reconstitute by just dumping whatever is in its slot vector?
23:32:27
stassats
can you dump a call to allocate-instance + dump a slot vector + set-instance-slot-vector?
23:33:49
drmeister
That's essentially what I'm going to do. The arguments are all in this massive vector. So I'll just reference them.
23:35:41
stassats
or maybe instead of dumping (list (make-instance ... (make-instance) )) dump '((some-kind-of-marker (some-kind-of-marker))) and then walk that tree at startup time and call allocate-instance in place of some-kind-of-marker
23:37:38
drmeister
That would require a bit more thought. I'm gonna try this ltvc_indexed_call thing - it's more inline with what I've already got.
23:38:51
drmeister
It will, we will only do the work to generate one llvm instruction and llvm will only have one instruction to deal with rather than the 50 or so it currently does.
23:39:35
stassats
but you are expanding into code that does (list (make-instance ... :x (make-instance)))?
23:44:11
stassats
i'd uses some binary format for dumping constants, but even just printing into a string and reading from it would be better for starters
23:47:31
stassats
but 2.6 M lines of ast-creating compiled code, yeah, no compiler can chew through that