freenode/#clasp - IRC Chatlog
Search
13:24:12
frgo_
8 GB RAM is probably not enough ... 16 GB and above is fine. And you better don't need to swap - or you are easily going to have build times beyond 2 hours.
13:28:30
drmeister
refpga: master and testing are the same - I'm surprised testing didn't work - what did you see?
13:30:00
drmeister
This is Keychron K2 to Keychron K2 communication here! It's like telepathy - but clickier.
13:32:08
drmeister
I know right? It supports bluetooth and cable, windows and Mac, it'compact with mechanical keys - why did it take until 2019 for such a thing to exist?
13:33:44
drmeister
But it's further evidence that we truly live in the best of all worlds. (Roiling fascism through democratic political systems notwithstanding).
13:38:11
drmeister
Yesterday I was in Washington pitching an idea that could fix lots of problems. Even if they don't fund it I will keep searching for some way to make it happen. It's very important.
13:39:11
drmeister
Hey, Bike may have some questions about process control - he is trying to improve it in Clasp. Are you available to answer some questions when they come up?
13:40:26
drmeister
We are proposing to search for artificial antibodies based on molecular Lego. I think we have everything in place to do it now. This will mean better diagnostic tools and better therapeutics down the road.
13:46:14
selwyn_
do you have some time to chat? i've been thinking about making a new hash table in clasp
13:48:12
selwyn_
but i wonder if its enough to expose a few atomic primitives and write most of it in lisp
13:49:03
selwyn_
Shinmera has a portable atomics library https://github.com/Shinmera/atomics that could be implemented in Clasp
13:49:15
drmeister
Sure. There are no fundamental differences between C++ code and Lisp code in Clasp.
13:50:01
Shinmera
drmeister: selwyn_ Way ahead of you guys https://github.com/clasp-developers/clasp/issues/672
13:57:14
Bike
well, starting with implementing and exposing atomics to lisp would be good. we ought to do that regardless.
13:58:16
drmeister
I've also been thinking about using read-copy-update to implement hash-tables that require no synchronization between threads. But my thinking on read-copy-update may be incorrect. I've been trying to get hold of some of our computer science faculty to ask a few questions about it. But no time.
14:00:00
Shinmera
drmeister: It's easy to make something that requires no synchronisation, but you're gonna be spinning a lot, and thus blocking progress, unless you do complicated stuff such as what my attempted port did.
14:00:08
drmeister
selwyn_: We would be happy to provide support for your efforts there. It's important but not yet one of our main concerns - so you could work on it at your pace.
14:00:35
Bike
i can try making some hir instructions and whatnot for CAS, shouldn't be anything conceptually difficult given llvm's instruction
14:01:37
drmeister
I think the read-copy-update RCU works like this. You have a hash-table with an alist in front of it. The hash-table stores the entire association but when threads write to the data-structure they update the alist atomically. Reads of the data structure first check the alist and then the hash table.
14:02:13
drmeister
Then you need some mechanism to guarantee that all threads are not reading the data structure and at that time you copy the contents of the alist to the hash-table.
14:02:45
drmeister
The alist is more than just key/value pairs. It also keeps track of keys that were removed AND rehashes (that's the fussy part).
14:03:49
Shinmera
Anyway, here's some slides by the guy who actually did this stuff. https://web.stanford.edu/class/ee380/Abstracts/070221_LockFreeHash.pdf
14:05:51
Shinmera
Luckless does implement an actually working luckless list. That part I'm in the very least sure works. https://github.com/Shinmera/luckless/blob/master/list.lisp
14:06:28
drmeister
Ok, in my thinking you don't update the list that way, you just keep pushing instructions that are interpreted as changes to the hash table into it. Doesn't that bypass the ABA problem?
14:08:21
drmeister
And then when the threads synchronize the instructions are used to update the hash table.
14:09:27
Shinmera
If you know when such synchronisations make sense, and how to resolve conflicts of such a synchronisation, and can afford blocking all threads until the synchronisation is done, sure.
14:10:05
drmeister
There is one worst case with copying GC where location dependency is detected and the hash table has to be rehashed. After that happens and until the threads are synchronized you have to treat the entire hash table like an alist.
14:11:15
Shinmera
Essentially what you're proposing is what distributed databases with eventual consistency do.
14:14:41
drmeister
selwyn_: As you can see - there is great interest and technical knowhow here to draw from. Have at it!
14:20:16
drmeister
I do want to do a lot more parallel programming and hash-tables that can handle this efficiently will be very valuable.
14:22:07
drmeister
We already ran into performance problems with the find-class database hash-table.
14:25:05
drmeister
I tend to use maphash because I'm a knuckle dragging neanderthal. (apologies to those of us with neanderthal heritage).
14:28:19
selwyn_
lparallel provides parallel map variants https://lparallel.org/pmap-family/ but no lparallel:pmaphash
14:31:42
drmeister
For example, at this very moment I'm writing code to compare N molecules to each other. This would be so much better to do in parallel and I need a thread-safe hash-table. We have one - but it's stupid and uses locks.
14:36:23
refpga
drmeister: Let me run it again and put the log here. The initial C++ code was built successfully and I kept falling back to lisp debugger. I don't remember the exact error.
14:44:36
drmeister
I just want to say that I love, LOVE programming in Cando. Between jupyterlab, slime, Common Lisp, source code tracking and the chemistry libraries I've built up - it is an absolute joy. The debugging experience could be filled out a bit more and we always need more speed in the compiler and generated code. But we really have something here.
14:48:54
drmeister
Is there a way to shut down this message? "Compilation failed. Load fasl file anyway?"
14:50:01
refpga`
drmeister:Here is the log: https://bin.disroot.org/?efd5a3f9038138f2#PeJ1F6pFL0tDvCGTrU3b8wubDHL+7AphHTmTahi3eII=
14:50:26
refpga`
I used the command `nice -n 19 ionice --class 3 ./waf --jobs 2 build_fboehm | tee compile.log`
14:51:16
drmeister
refpga`: build_fboehm? It should be build_cboehm. Are you following the instructions here...
14:51:43
refpga`
https://github.com/clasp-developers/clasp/wiki/Build-Instructions#building-on-machines-with-limited-resources
14:52:46
refpga`
Yes. Building on machines with limited resources builds in a two step process, linking step is run on a single thread.
14:53:43
drmeister
In the wscript file it says... ./waf build_fboehm # will build most of clasp,/ except the most memory hungry linking tasks at the end
14:54:54
drmeister
It looks like it's going completely off the rails. What version of llvm/clang are you using?
14:55:51
Bike
it's complaining about va_tooManyArgumentsException, but we removed that symbol months ago, right?
14:56:38
drmeister
Yeah - but he is building testing - I don't know on what side of the time divide testing is at.
14:57:38
refpga`
I'll be back, I have to kill X and all processes each time to bring ram usage to 150MB.
14:58:22
drmeister
It's funny that it would signal an error on one thing that we possibly removed around the time that I pushed things up to 'testing'. It's possible that I pushed broken code.
15:00:02
drmeister
refpga: Yes - we use packaged llvm on linux (debian, apt-get) and macOS (homebrew)
15:01:10
drmeister
selwyn_: Quite sensitive. Clasp uses bleeding edge features. When I upgrade it's because I need features from that latest version.
15:01:33
drmeister
I've held back for a while because I've been waiting for distros to catch up with package manager versions of llvm.
15:07:01
drmeister
refpga: I use this... LLVM_CONFIG_BINARY = '/usr/local/Cellar/llvm/6.0.1/bin/llvm-config'
15:07:26
drmeister
The llvm-config program tells the waf build system where to find everything related to llvm and clang.
15:07:45
drmeister
If you run ./waf configure it should tell you at the top where it's getting clang from.
15:08:18
drmeister
On macOS we have the stock clang and clasp avoids that and uses the one pointed to by LLVM_CONFIG_BINARY
15:13:48
refpga
https://bin.disroot.org/?d7a22bcb75dfa3f7#3msw9ZRVipsaP1tafaZzPVAYaf5cUq20zvo/JztZ8CE=
15:15:15
drmeister
Try it with 'dev' but start from scratch... ./waf distclean configure build_cboehm
20:03:55
refpga`
Do I need to do something other than set inferior-lisp-program to make it work with sly/slime?
20:05:18
drmeister
refpga`: I don't think so - others may have other suggestions but I use C-u M-x and then I provide the path to the clasp executable.
20:07:19
refpga`
Seems like sly waits forever to set up a connection with it. I can run the repl in terminal fine.
20:18:10
drmeister
C-u prefix means "that thing that emacs does when you use M-x whatever? Don't do that - do the thing that I want you to do".
20:51:35
drmeister
I haven't tried sly with Clasp. I don't think it can work unless someone writes the sly implementation dependent code.
20:58:54
drmeister
It depends on how far sly has drifted from slime. The clasp.lisp implementation file in slime would be a good starting point.