freenode/#clasp - IRC Chatlog
Search
15:55:22
drmeister
Hello - I wasn't able to work on compiler issues this weekend. I'm tied up in application land.
15:59:31
drmeister
What I wanted to do was either 1. set things up again so that clasp's serial and parallel compilers both dump llvm-ir and clasp can load it directly; or 2. figure out how to generate library files (dylib or faso) from llvm-ir files.
17:04:40
karlosz
Bike: you can try inserting reverse or something in the list implementation in an attempt to get a "bad order"
17:36:30
karlosz
but theres like a miniscule chance its a problem with the hash table implementaiton
18:11:08
karlosz
i solved that by fixpointing the algorithm so you'd get deterministic results from that
18:11:52
karlosz
i bet there's a similar issue lurking somewhere where the order in which we operate on functions over a module is important
18:19:50
judson_
ok dumb question - I thought any llvm IR could be turned into wasm? or are certain things not allowed
18:20:34
Bike
https://webassembly.org/roadmap/ as you can see, exception handling, simd, tail calls, and atomics/threads are all merely proposals
18:21:24
Bike
this also doesn't cover llvm intrinsics we might use. i don't know if things like llvm.frameaddress are implemented
18:22:57
Bike
also, practically speaking, even if wasm did have all the components, you'd end up with a gigabyte sized web page
18:23:50
phoe
I'll be giving a presentation of control flow in Lisp for the WASM people at the end of this month
18:25:06
Bike
i am one of the world's foremost experts in implementing lisp conditions in terms of C++ exceptions
18:25:16
Bike
the other expert is drmeister, and nobody else even knows what the hell we're on about
18:26:32
Bike
i'm not seeing The Problem with shuffled lists. it does segfault sometimes, though. that's super
18:28:52
phoe
Bike: my talk will be exactly about not needing to implement non-local jumps with throws
18:30:07
Bike
phoe: we non local jumps with C++ exceptions. maybe that's more or less what you mean by throws? anyway, it's horrible and i'm happy to rant at you
18:32:59
Bike
C++ implementations sometimes provide a setjmp/longjmp implementation of exceptions, which is more or less handler-case in terms of return-from, but nobody uses it
18:33:41
Bike
llvm doesn't provide any primitives for nonlocal exits other than the handler-case thing and setjmp/longjump :(
18:34:44
Bike
not sure what you mean by respect unwinds. it does allow nonlocal control transfers, but there's no unwind-protect
18:36:37
Bike
well, not as such, no. libunwind lets you implement setjmp/longjmp, more like. lets you do stuff like manipulate stack frames as data structures
18:37:09
Bike
learning how to implement unwind-protect has been interesting. it's rather more involved than merely exiting
20:22:17
princesspancake
hello. i'm making a CL binding for a game engine by using ECL. i recently heard about clasp and i wanted to know how embeding a shared library would work with it but i failed to find any documentation. do you have any info or a link toward some docs ?
20:22:53
princesspancake
(by embedding a shared library, i mean creating a shared library form CL source code, and use it in a c/c++ program)
22:15:55
drmeister
Reading. Hi princesspancake - ok thinking about creating a shared library from CL source code. Can you do this with ECL? How do you deal with garbage collection?
22:54:34
princesspancake
drmeister: garbage collecting works as usual. foreign datatypes aren't GC'd so there is no issue
22:55:35
princesspancake
in ECL i can do `(asdf:make-build :package-name :type :shared-library)` and i get a .so file
22:56:32
princesspancake
and since it's for a game engine i want as little as possible overhead for interfacing C <> CL
22:57:38
princesspancake
in those i just do `cl_boot(); ecl_init_module(NULL, module_init_function); ecl_funcall(1, ecl_make_symbol("entry-point", "mypackage"));` and the control is given to the CL code
22:58:36
princesspancake
even if it has ways to add C declarations and inline C code, it has issues regarding FFI types handling
22:59:27
princesspancake
if you want i can upload my project to github/gitlab and show you the source if you want to see how it's done
22:59:49
drmeister
The C <> CL interfacing is much easier in Clasp I believe. But could we talk about this a bit more? I'm trying to wrap my head around how we would turn Clasp into a library.
23:00:36
princesspancake
is there some runtime features that can only be in the clasp .so and not embeded directly in the output dll ?
23:01:53
drmeister
I'm not sure. Clasp supports two garbage collectors. When I was sorting out how to deal with the two of them I came up with an approach that is kind of troublesome to put into a library. I'm not sure though.
23:03:27
drmeister
Hmm, the syntax is a little out of date - but it's close enough to get the idea across.
23:03:44
drmeister
https://github.com/clasp-developers/demo-clasp-cxx-interoperation/blob/master/hello-world/hello-world-cxx.cc#L56
23:04:19
drmeister
https://github.com/clasp-developers/demo-clasp-cxx-interoperation/blob/master/hello-world/hello-world-cxx.cc#L7
23:05:22
princesspancake
binding from the C/C++ side looks p good yes, very easy to do. in my project i'm binding from CL side because the whole API is in .json files and need generation, and it's way easier and cleaner to generate CL code than C/C++
23:05:34
drmeister
https://github.com/clasp-developers/demo-clasp-cxx-interoperation/blob/master/hello-world/hello-world-cxx.cc#L13
23:06:19
drmeister
Again - a string that defines the name in Common Lisp and a pointer to the function.
23:08:05
drmeister
https://github.com/clasp-developers/demo-clasp-cxx-interoperation/blob/master/double-vector/double-vector-cxx.cc#L18
23:08:43
drmeister
https://github.com/clasp-developers/demo-clasp-cxx-interoperation/blob/master/double-vector/double-vector-cxx.cc#L133
23:10:56
drmeister
I've recently revamped this stuff so that it looks more like PyBind11 - that is a C++ template library that exposes C++ code to Python. I did that so I could take PyBind11 code and use it with very little modification in Clasp.
23:11:14
drmeister
If you have lots of classes and methods and functions - you expose them like the following.
23:11:15
drmeister
https://github.com/clasp-developers/demo-clasp-cxx-interoperation/blob/master/double-vector/double-vector-cxx.cc#L133
23:11:48
drmeister
https://github.com/clasp-developers/clasp/blob/master/src/asttooling/clangTooling.cc#L726
23:13:11
drmeister
But note - there are no argument types or return types mentioned here - that is all handled by the C++ template library that generates the wrappers (mostly).
23:13:45
princesspancake
all of that is impressive. i never understood template-based metaprogramming
23:16:16
drmeister
The strength of this approach though is that the C++ template library runs inside of the C++ compiler - and it knows everything about your types.
23:17:29
princesspancake
for this i do something called grovel. it is generating a .lisp file with some data and inlined C code, compiling it, running it and reading back the data, all of that during macro expension so the requested data becomes constants in the compiled file
23:19:31
drmeister
In Cando we expose over 500 classes and over 1000 functions and methods to Common Lisp.
23:20:27
princesspancake
https://github.com/linkpy/godot-ecl if you want to what i did for my project. currently i have issues because FFI doesn't support arrays in function return types and i need to make an ugly workaround for that, that's why i checked clasp
23:21:06
drmeister
It's different ways to achieve the same thing. I use this to expose much of LLVM and the Clang compiler. When those libraries change (and they do a lot) it's relatively easy to keep up - because the C++ compiler complains when there is breakage.
23:21:48
princesspancake
it is true that i could generate nearly everything in C from CL but that needs customizing the build system
23:22:21
princesspancake
gdcl/ffi contains all of my custom FFI code, and there is gdcl/binding/entry for the library loading
23:22:40
princesspancake
https://github.com/linkpy/godot-ecl/blob/master/gdcl/binding/entry.lisp this contains the code used by the game engine to load the .so file
23:23:25
princesspancake
and build.lisp is loaded in the repl to build the library, using ECL's ASDF extension
23:25:16
drmeister
Is your main loop in C and then you call the Common Lisp code - or do you call a Common Lisp function and that has a loop that calls out to C code?
23:27:48
princesspancake
but the engine doesn't export any C functions. in the file gdnative_api.lisp you have the binding description (all available functions) that the engine gives to the .so throught the godot_gdnative_init_options pointer
23:29:01
drmeister
Where is the array in function return types that you need to use the ugly workaround?
23:30:09
princesspancake
look at types.lisp in the root of the repo. it is all of the native types of the engine (it's macroexpanded from %define-core-godot-types in gdcl/binding/macros.lisp)
23:30:19
drmeister
As you know - under the hood, C can only return two values in registers (one really) - so we pass pointers or references to structs to return more complicated things.
23:30:53
princesspancake
and the work around would be to generate X fields of padding, of type uint8_t to have the structure of the correct size
23:31:41
princesspancake
the structures that are returned are usually p small and so it works returning them on the stack
23:35:22
princesspancake
for example : `(ffi:def-struct foo (field (:array :int 1)))` defines a new structure
23:35:22
princesspancake
`(defun test () (ffi:c-inline () () 'foo ""))` uses that structure (c-inline is lisp-values value-types return-type c-expr)
23:35:22
princesspancake
and you get the output "unknown representation type 'FOO". tho switching to use 'foo as an argument works
23:35:40
drmeister
Just to be clear - this is the godot graphics engine - yes? https://godotengine.org/
23:36:10
princesspancake
https://github.com/godotengine/godot_headers/tree/3.2 this is the native interface (poorly documented, sadly)
23:39:21
princesspancake
https://github.com/godotengine/godot_headers/blob/f2122198d51f230d903f9585527248f6cf411494/gdnative/variant.h#L209
23:39:53
princesspancake
these headers are just for reference, the functions declared in them are not exported by the engine (you need to get them from the init_options struct)
23:40:38
princesspancake
all of the godot_* types has that issue (except _bool, _int, _real which are just typedefs to bool, int and float)
23:41:01
princesspancake
(all of the others are structs with uint8 arrays in them so they keep the same size as the internal engine ABI)
23:41:34
princesspancake
https://github.com/godotengine/godot_headers/blob/f2122198d51f230d903f9585527248f6cf411494/gdnative/string.h#L48
23:59:52
drmeister
I suppose we could set Cando up so that it could work as a library. I don't see any reason why not.
0:00:42
drmeister
I set things up so that Cando provides a main(...) function and that calls a function called gctools::startupGarbageCollectorAndSystem(...) here...
0:02:38
drmeister
Clasp is a standalone Common Lisp implementation and it runs a REPL. Cando is Clasp extended with a whole bunch of chemistry code written in C++ and Common Lisp.
0:03:27
drmeister
The idea was that we add C++ libraries to Clasp and Cando to extend them. Rather than adding Clasp as library to other C++ code.
0:04:35
drmeister
But if we took everything up to the REPL starting and run that in one function - called say cl_startup(...). and have another that does everything to shut the system down - called say cl_shutdown(...) - then it would do everything that you need there.
0:07:24
drmeister
Compiling arbitrary code isn't a problem - you just compile it and load it with LOAD.
0:08:37
drmeister
We also do stack unwinding using C++ exception handling - so RAII works and you can have C++ frames and Common Lisp frames on the stack in any order to any depth.
0:08:54
princesspancake
ye but if i write the code in an external c++ file i need to modify (or make my own) build system
0:09:12
drmeister
No - we don't have FFI:CLINES or FFI:C-INLINE. I never got the point of those - just write it in C++ and call it?
0:09:51
drmeister
ECL FFI can do that because they compile everything to C and use the C compiler to generate machine code.
0:10:09
drmeister
We don't do that. We generate the LLVM-IR intermediate representation that the Clang C compiler generates.
0:11:41
princesspancake
yes, but in your case you can make some kind of system to allow generating LLVM-IR code to have arbitrary C-compatible machine-code in the compiled file, which would be way better than inlining a string into the outputed C file
0:12:09
princesspancake
i need to invoke the C++ compiler and then manually links all of the compiled lisp & cpp files to make the .so
0:12:38
princesspancake
ECL has that asdf:make-build extension which basically compiles a whole system into an exe/dll/static lib