freenode/#clasp - IRC Chatlog
Search
14:16:49
drmeister
So - I've been toying with the idea of generating webassembly from Clasp. Sadly - it won't work. While C++ can be compiled to webassembly, garbage collection isn't easy to achieve within the webassembly runtime.
14:17:46
stassats
i wanted to do something with webassembly, but it's poorly documented and requires a ton of support javascript code
14:20:14
drmeister
The machine registers aren't pushed onto the C++ stack. They are on a shadow stack that is inaccessible to the C++ code.
14:21:26
drmeister
So I couldn't ship MPS with the code because it can't scan the stack conservatively and find roots in registers
14:22:00
drmeister
webassembly doesn't have registers - but the machine running webassembly has registers.
14:24:08
drmeister
If the compiler optimizes some variable out of the stack frame and into a register - then the GC won't see it. I was hoping you would challenge this.
14:25:14
drmeister
You don't see a problem with that? I worried that roots could disappear into registers.
14:27:04
drmeister
Here's my thinking. I thought I could compile C++ and CL generated bitcode to webassembly. This would include the MPS garbage collector - which is written in C and compiled into Clasp.
14:27:43
drmeister
Bike: I talked with a couple developers yesterday on the #webassembly IRC channel. I posted something this morning to the webassembly/gc discussion board on github.
14:28:21
drmeister
The webassembly gc page goes into a lot of detail of what they want - everything that I already have with MPS.
14:28:23
stassats
drmeister: if the code is optimized which breaks the semantics of a stack machine, then there would be problems
14:29:39
drmeister
(a) There would be problems with conservative scanning of the stack for garbage collection - or (b) there would be problems in general that guarantees that they won't do optimizations that break the semantics of a stack machine?
14:32:56
drmeister
Shinmera: If it's easy - I think it would be a good way to disseminate it - it would also allow me to do things that require me to work with Javascript now.
14:33:20
Shinmera
you don't seriously think you can run Clasp, a 50mb binary or what, implemented in basically JS, in your browser?
14:35:39
drmeister
Shinmera: It's a bit of a lark at the moment - what is a reasonable sized binary for running in a browser?
14:36:06
stassats
my cpu has 168 registers, with only 16 exposed to the user, so it waits for all the in flight operations to complete when you access all of them, wasm could have something like that too
14:36:18
Shinmera
If you want to just run it, I don't know. I can't imagine anything wasm to be really fast if it's that fucking huge because browsers are garbage at memory.
14:39:52
drmeister
Shinmera: It's part of my (unwritten) job description not to be reasonable at times.
14:41:29
drmeister
I was thinking - wasm looks like a great and up and coming way to deliver web applications. It's getting more capable (exception handling, threads, memory page protection, garbage collection). It's llvm based.
14:42:28
drmeister
We have a Common Lisp that has a C++ runtime and uses llvm and could generate wasm.
14:43:05
Shinmera
drmeister: I just don't see this working now, or in the next three years. Don't waste your breath on this at the moment.
14:43:06
drmeister
I'm not wasting time on it. I've been finding people to ask questions while things compile for the last two days.
14:43:23
Bike
if i clicked on a webpage and it started downloading 250 MB of binary i'd probably close the tab, or more likely try to close the tab while my browser freezes
14:44:03
drmeister
The 250MB binary is the current state of Clasp - we have spent zero time optimizing size.
14:44:07
Shinmera
Bike: Even if my browser just downloaded 1mb of JS but then maxed my CPU trying to run "(print :hello)" I'd still close it.
14:44:34
Bike
clasp's present design is a big monolithic system, like most lisps. moving away from that would be interesting, but "nontrivial" doesn't begin to cover it
14:46:27
Bike
anyway, i've been sketching a manual. is the C++ integration documented anywhere already or do we just refer people to the bullet demo?
14:47:31
Bike
i can document the mop and environment extensions (also, i should export the latter a bit), but i don't know gray streams, sockets, or c++
14:47:38
frgo
Well, having a small MAIN module being the executable and separating the rest of clasp into a shared library is not that hard. I'd love to see C++ Binding, GC interfacing, CLOS, etc being in separate shared libs. We could then use the loader to handle init at load.
14:49:20
Bike
but i'd rather not copy a manual that doesn't actually apply, and i don't know how much we've changed from ecl in those modules (though i'd guess not much). lying manuals are bad
14:50:25
frgo
stassats: Yes - but only as long as you are doing clasp core development. Whenb done, create an optimized image and load this.
14:52:58
drmeister
I used luabind as the template when I developed clbind - I think I also stole code.
14:53:19
drmeister
The developers of luabind were fine with us taking their documentation and modifying it to describe clbind.
14:54:31
drmeister
Understood. The clbind-doc is the closest thing. The luabind documentation is more like my springboard to fill in the gaps of the clbind-doc.
14:55:20
Bike
this is like, the main feature. if we want users we should document it so if somebody goes "i want to use opencv" or whatever we can just point them at it rather than walking through the whole thing based on oral tradition
15:01:15
Bike
i know it's yet more to do, but we should probably act like clasp is actually useful rather than perpetually coming soon. then we can fake it til we make it
15:05:31
drmeister
The build system has been a major barrier for me for releasing it. I also kinda wanted to get a minimum set of features that Common Lisp users expect (CFFI, threading, Slime, quicklisp) and MPS. We've pretty much ticked off all the boxes.
15:06:32
Bike
sure, there's lots of problems. but even if we don't release we should adopt a kind of release mindset, i think.
15:12:13
frgo
Nope. Too many changes in clasp and I am unable to follow at that pace with changes. Or I just don't understand.
15:12:23
drmeister
Those may still be broken - I don't recall the fleeting feeling of satisfaction that comes with fixing something related to callbacks.
15:14:40
drmeister
I have an important DOD meeting in just over a week and then I can push for a release.
15:26:05
Shinmera
It's so that the kids in this channel are saved from being exposed to swear words when they work on clasp.
15:28:17
Shinmera
Ahahaha, it's all me https://www.google.ch/search?q=site%3Airclog.tymoon.eu+clasp+"fucking"&oq=site%3Airclog.tymoon.eu+clasp+"fucking"
15:28:49
Bike
https://www.google.ch/search?q=site%3Airclog.tymoon.eu+clasp+"fucking"&oq=site%3Airclog.tymoon.eu+clasp+"fucking"
15:43:00
drmeister
ACTION is glad to know that as far as google is concerned - he is a pure as the driven snow.
15:44:04
Bike
on the topic of binaries, what is needed to actually run clas? just iclasp boehm and an image, or more?
15:44:05
jackdaniel
I'm sure they have already profiled you as a cursing person who obfuscates these words. They wouldn't be able to make many if they didn't know such things ;-)
15:45:33
jackdaniel
I'm going to give some rest to my brain (that is – going to watch a movie which doesn't require a single thought [as I've been told] – justice league ;)
15:46:06
drmeister
Bike: minimally - I'd start with the executable and a couple of bitcode files: builitins.* fastgf.* intrinsics.*
15:46:56
drmeister
They are linked and inlined with JITted code, dispatchers and compile-file'd code.
15:47:43
drmeister
They are incorporated into the image - but as native code - these are bitcode - for llvm.
15:51:26
drmeister
I'd rather embed them in the executable. I've been thinking of base64 encoding them in an array.
15:52:19
drmeister
Note: We still have a problem with backtraces on linux. That's kind of a show stopper.
15:52:59
drmeister
I tried to get libunwind to work - but there is a complicated JIT registration system that has thwarted me until now.
16:00:02
drmeister
ACTION goes back to debugging thread local variables - which he appears to have broken this morning.
16:03:13
frgo
When doing coding for clasp I often hear https://www.youtube.com/watch?v=OpnmLRKktU4 and find that kind of very matching ...
16:05:57
frgo
And - I have a real use case for Clasp: Binding RTI Connext DDS' C++ library to Lisp. AND I need to have that running in January 2018. I very much would like to be able to have PoC ready.
16:08:35
frgo
The title / interpret I mentioned helps in pushing the cpu fan noise into the background
16:11:51
drmeister
stassats: A lot of Clasp's problems are inherent to the way it builds itself up from C++ and the way that it starts up - there are a lot of optimizations that we can't take advantage of.
16:12:15
drmeister
Bike is working on it - he got cclasp to build cclasp a new way yesterday that is a stepping stone to getting sbcl to build it.
16:12:16
frgo
drmeister: Re clbind: I have a class dds::domain::DomainParticipant. How would I expose that using clbind? Like so:
16:13:26
Bike
though on the note of speeding things up, i asked this a few days ago but: i realize parallel jit is broken, but what about parallel compile file?
16:15:27
frgo
drmeister: Sure: Example here: https://community.rti.com/static/documentation/connext-dds/5.3.0/doc/api/connext_dds/api_cpp2/classdds_1_1domain_1_1DomainParticipant.html
16:20:48
drmeister
I am being pulled out the door to get some brunch - I'll be back in an hour or so.
16:21:39
drmeister
That example needs more classes exposed to flesh it out because many of the functions take instances of other classes that need to be exposed as arguments.
16:22:16
frgo
Yeah. Thx. And the DDS C++ shared lib? Do I link that just to clasp during compile or do I do a dlopen() and load it at runtime?
17:09:12
drmeister
The shared lib can be loaded with dlopen or linked and the wrapper code can be loaded as bitcode. If you compile the shared lib as bitcode then everything can be linked as bitcode and LTO will inline (almost) everything into everything.
18:21:13
drmeister
Yes - the extension modules give you something in addition - they let you define GC managed C++ classes.
18:28:24
drmeister
The way I see it is an asynchronous signal handler should get the SIGINT signal and should set a flag that some thread (which?) will occasionally check and then unwind the stack.
18:33:17
frgo
E.g.: Do we want to stop clasp on sigkill? Certainly. But in a controlled way, Now: What is the "controlled way"? Send sigkill to all threads - meaning; Calling a function per thread that kills the thread? Would be one way to do it.
18:34:13
drmeister
But the underlying mechanism - how does a thread recognize that it got a sigkill?
18:36:11
frgo
we need to have a function registered in the thread struct that gets called when the process' signal handler is invoked. So:
18:37:20
frgo
Signal -> Caught by signal handler in process -> That signal handler does for all threads: Check if function (generally or a separate one for each signal)
18:38:01
frgo
The called function my be set to something thread specific - but normally would just exit the thread .
18:40:21
frgo
For applicatoions that really need to shutdown gracefully no matter what signal is received it is required to have an alternate stack in place (via function sigaltstack) that is used to handle signal handler function execution.
18:42:46
frgo
Yes. That is the question. For C++ only that's magic (for me) - for clasp: I don't know.
18:44:41
Bike
if signals are process level they're hard to use. say we get a floating point exception, how could we know what thread triggered it?
18:44:44
drmeister
The only way I know how is for C++ to occasionally check a flag (global or thread local) and proceed based on the results.
18:45:52
drmeister
Bike: Martin Cracauer brought that up - he said we shouldn't involve signals at all - they are too expensive, and bring problems like the one you are raising - they are the wrong mechanism.
18:45:55
frgo
Well, a thread that does cause a non-recoverabele error wll be in a defunct state that we can check for.
18:47:42
drmeister
The only way I see is for every thread to occasionally check a flag to see if it should check further for directions.
18:49:01
frgo
I've never done it, but: I would like to test if we can setup signal handlers per thread. Hang on ...
18:50:27
drmeister
Even if you can - they are fundamentally asynchronous and the stack will be in a dirty state and shouldn't be touched, let alone unwound.
18:52:42
drmeister
Unless someone has a better idea or understanding - I would say we have a thread-local variable that each thread checks as it leaves a function or at the bottom of loops (and could be generated or not using a (DECLARE (OPTIMIZE ...))) and take the hit in performance.
18:55:17
frgo
So we have pthread_sigqueue to send a signal to a specific thread. will read up on that again ...
19:02:39
drmeister
Hmmph - I just figured out what I broke this morning - I moved GC managed pointers out of GC managed memory.
19:07:12
drmeister
I managed it in a special way - I defined a ThreadLocalState struct that contained all of the thread local data structures directly.
19:08:01
drmeister
I allocated that at the top of the stack. With it on the stack it is managed, its roots are identified by the conservative stack scanner.
19:09:40
drmeister
I ran into an issue with header file order now that I want to define the allocation points within the ThreadLocalState structure - so I defined ThreadLocalState to contain pointers to some of the thread local data structures - BRRRRAAAAPPPP! Wrong - don't do that.
19:12:11
drmeister
As long as the thread is live the stack is active and the thread local roots are roots.
19:14:23
drmeister
Bike: By the way - David Lovemore (Ravenbrook) says (re: Saving and restoring MPS memory) "It's not possible at the moment. But it ought to be possible."
19:46:25
jackdaniel
just as I thought, watching this movie was a great rest for the brain – I had enough visual effects to keep it at some low futile gear yet it has too little meaning to engage higher parts
20:59:47
drmeister
Bike: Is there a way to identify the bottom of loops - to insert a check for notifications from the system to threads?
21:00:47
Bike
we could detect obvious ones. that would cover most of it but obviously we can't identify all possibly nonterminating regions
21:01:46
drmeister
Could we insert an invocation of an inlined test controlled by a (declare (optimize ...)) policy?
0:48:10
drmeister
It's a little slow - I'm not sure why - it took 1h45m on my system here when a couple of days ago it was 1h30m
0:51:01
Bike
for the not found, you have to set the variable in wscript.config to point to the externals config llvm-config.
0:52:14
drmeister
stassats: Put this in your wscript.config: LLVM_CONFIG_BINARY = '/Users/meister/Development/externals-clasp/build/release/bin/llvm-config'
0:52:43
stassats
this is what make says Checking for program 'llvm-config' : /usr/lib/llvm-5.0/bin/llvm-config
1:02:22
stassats
and please, don't remove makefile altogether, i wouldn't remember the right build incantation
1:07:20
Bike
yeah, looks like some arm cpus have a "performance monitoring unit" that has a few counters
1:16:30
stassats
export LIBRARY_PATH=/usr/local/lib/ works (and why do i have it in /usr/local? because i had to build it by hand for clasp to get linked by clang)
1:16:31
drmeister
Bike: There's a guy at Temple who could help us set up some virtual servers to do continuous integration.
1:17:38
drmeister
stassats: You had to build boost recently by hand? I've been using package managers to install boost for a couple of months.
1:19:46
frgo
-> missing file: '/opt/common-lisp/lang/clasp/src/clasp/build/mps/src/gctools/interrupt.sif'
1:22:15
drmeister
interrupt.sif should have been built by the scraper - I don't understand how it could be missing. It's automatic.
1:25:21
drmeister
When it was building clasp - did it report something like: [ 19/356] Scraping with preproc.scan src/gctools/interrupt.cc
1:26:37
drmeister
It's using sbcl and running the preprocessor on each source file - I'm worried that it failed somehow for interrupt.cc
1:27:59
frgo
Yes, it did fail for interrupt.cc - see: https://gist.github.com/dg1sbg/5538a17e93b32543d3fa4bd33cc0569f#file-gistfile1-txt-L53
1:36:57
frgo
drmeister: For a later date: Signal handling the safe way - that's what we need to do in clasp: https://gist.github.com/dg1sbg/3b608bf1c880148cc06faffd641c8809
1:38:11
drmeister
stassats: It's a macro that generates a wrapper around a C++ function so that it compiles the lambda list/argument handling with Cleavir rather than using the C++ LambdaListHandler_O class.
1:49:26
frgo
To get rid of that FRGO... tag I need to assign a new one. How should be name the current state of affairs? which version of clasp do we currently have? like pre_0.5 ?
1:51:23
stassats
all uses of GENERATE-DIRECT-CALL-DEFUN use MAGIC-INTERN, maybe it should just be moved into generate-direct-call-defun?
2:01:42
drmeister
There's a chance that it might not use magic-intern - but they are all using it at the present time. I think I'll leave that this way.