freenode/#clasp - IRC Chatlog
Search
15:26:19
beach
At least I am now off the hook, since drmeister has decided that the compilation slowness is largely the fault of LLVM.
15:30:05
kpoeck_
So since drmeister nicely logs where time is spent, i might be able to support or contradict his intuition
15:32:32
beach
I remember thinking that LLVM could be slow because we are giving it huge chunks of code to compile, but drmeister was able to convince me otherwise. I forget the details of that argument, though.
15:40:00
kpoeck_
Now that disassemble works i can compare size of generated code between sbcl and clasp
15:42:38
kpoeck_
But giving positive feedback, responsiveness of clx/mcclim is a lot better than it used to be
15:59:42
drmeister
I built the MPS version of clasp and tried compile-file-parallel - it fails spectacularily - I'm looking at it.
16:17:49
jackdaniel
kpoeck_: I've meant that you should replace your define-constant-non-eql changes with shadow in pr
16:41:22
kpoeck_
In the clx.asd there is code for sbcl regarding the uneql problem, will comment that out
17:03:50
drmeister
With an EQ hash table gethash sets a read lock on the hash-table but when it doesn't find a key it has to check location-dependency on the key and if the location-dependency is stale then it rehashes the hash table - BUT! it doesn't set a write-lock when it rehashes.
18:02:02
drmeister
With MPS objects move in memory and so EQ hash table searches can fail simply because the location of the object moved since it was added to the hash table. It's called "location dependency"
18:03:06
drmeister
If a hash table search fails you have check if the location dependency has gotten stale and if it has - then you rehash the hash table.
18:03:35
drmeister
I wasn't properly upgrading the lock on the hash table from a shared lock to a write lock under those circumstances.
18:05:31
drmeister
Well, I hope I havent' spoken too soon. A parallel ASDF compilation I was running to test it might be deadlocked right at the end.
18:08:10
drmeister
Yeah - it's deadlocked - but right at the end. It gets all the way through and then locks at the end.
18:08:58
drmeister
So I'm certain this is the issue but my UpgradableSharedMutex may not be correct or I may be using it incorrectly.
18:08:59
drmeister
https://github.com/clasp-developers/clasp/blob/dev/include/clasp/core/mpPackage.fwd.h#L195
18:16:30
drmeister
Have you used upgradable shared mutexes before? One that you can put multiple read locks on and then upgrade one of them to a write lock?
18:17:25
drmeister
I'm reading this - it looks like it describes the theoretical underpinnings: https://shlomisteinberg.com/2018/06/24/fast-shared-upgradeable-mutex/
18:23:02
drmeister
I've got four threads deadlocked. Two are trying to upgrade a read lock and two are trying to get a read lock. Bleh.
18:24:09
drmeister
I'm certain this is the problem though - I just need to figure out the UpgradeableSharedMutex situation.
18:25:50
frgo
ah - where is UpgradableSharedMutex used? I tried to find uses in clasp code but somehow I can't ...
18:26:43
drmeister
I'm going to make sure that boehm still works with these improvements and then push it.
18:27:15
drmeister
I'm certain that this is the problem. I'll have to relearn some stuff about mutexes before I can fix this though and I have no more time for this today.
18:31:09
frgo
I don't know ... It's the statement "Written in modern cross-platform c++17." that prompted my remark.
18:56:13
drmeister
The std C++ threads only allow a fixed stack size and on macOS it's 512K - pathetic.
18:56:21
drmeister
https://www.google.com/search?q=c%2B%2B+threads+stack+size&oq=c%2B%2B+threads+stack+size&aqs=chrome..69i57j0l3.4392j0j4&sourceid=chrome&ie=UTF-8
20:05:43
drmeister
https://oroboro.com/upgradable-read-write-locks/ . Section: Upgradable Write Locks and Deadlock
20:06:17
drmeister
I'm pretty sure I wrote the UpgradableSharedMutex based on this webpage and I'm running into the deadlock they are describing.
21:01:04
drmeister
That's the hashtable problem that you pointed out with MPS that shows up when you use compile-file-parallel.
21:02:21
drmeister
Ok - with MPS objects move around. If you lookup a key in an EQ hashtable you may get a miss because the object has been moved in memory since it was added to the hashtable.
21:02:55
drmeister
So for MPS we have to check this thing called the location_dependency to see if the key has become stale. If it does you have to rehash the hashtable.
21:03:12
drmeister
This means upgrading the lock from a read lock (for gethash) to a write lock (for rehash).
21:04:54
drmeister
Parallel compilation of asdf.lisp is Time real(178.025 secs) run(178.023 secs) consed(2123097968 bytes)
21:05:51
drmeister
Parallel... Compile-file-parallel seconds real(178.0) run(178.0) llvm(78.9) link(3.6) (llvm+link)/real(46%)
21:06:05
drmeister
Serial... Compile-file seconds real(208.2) run(208.2) llvm(76.7) link(100.4) (llvm+link)/real(85%)
21:11:17
drmeister
kpoeck_: There may still be a problem. I tried your test code and it hangs after output of a few lines. I can control-c it though.
22:17:48
jackdaniel
I've mentioned in the comment, that in jd's ideal world you'd also test with ecl, but I took care of that myself ,)
22:19:04
drmeister
kpoeck_: I'm rebuilding MPS now as well. I can't rule out that the current problem running your test code isn't due to an unclean build.
22:19:38
drmeister
I'm surprised that you were able to get it to work with (stress-2 300) - I can't do anything over (stress-2 18) without a problem.
22:29:34
kpoeck_
And there is one (declare (fixnum ...)) for the result of a sxhash, but i didn‘t had much success with pr #628 yet
22:56:07
Bike
hashes are fixnums. i mean, from sxhash. it's not defined for internal hashes but we probably want fixnums there too.
22:56:33
drmeister
Ok, so this will limit the hash to 61bits - this will fit into a positive fixnum.
23:17:26
drmeister
Yeah - but you have to do that conditionally. I thought a bitwise and would simplify the code.