freenode/#clasp - IRC Chatlog
Search
9:56:38
attila_lendvai
hi! drmeister, saw your mail, sent a response. i'll be around here for the next 10 or so hours.
11:37:31
drmeister
You did some work on the waf build system and reached a deeper understanding of it than I have.
11:38:21
attila_lendvai
keep in mind though, that it was a long time ago, and clasp was my only only exposure to waf
11:39:21
drmeister
I implemented snapshot save/load - I use dlsym at load time to lookup addresses of functions/vtable pointers at load time. Everything works on linux.
11:40:13
drmeister
On macOS however, vtable symbols for template classes are hidden/local so I have to go through a complicated song and dance to export them.
11:41:00
attila_lendvai
(re hidden vtable symbols: that's surprising, although i have abandoned my c++ knowledge long ago)
11:42:54
drmeister
It's basically (1) build iclasp-boehmprecise and compile the common lisp code for cclasp (2) run iclasp-boehmprecise loading cclasp common lisp image with a special flag and build a symbol list and save it to a file (3) LINK an executable called "cclasp-boehmprecise" that uses the new symbol list. (4) new build stuff that saves snapshots and links new executables that have the snapshots embedded within them.
11:43:26
drmeister
I've been screwing with the waf build system for days to add step 3 and I have something working - but it's a bit broken.
11:43:52
drmeister
I haven't figured out how to fix it and the clock is ticking loudly that I have to work on other things.
11:48:19
attila_lendvai
i never used a mac, but i compiled maru on it in a VM under linux, but the bugs drove me mad...
11:50:07
drmeister
The end goal is to get a working clasp - on macos and linux. It's probably broken on both operating systems - I'm not writing macos specific code here. We can use the same symbol list generating code on linux and get a slightly smaller executable.
11:51:09
drmeister
This clasp builds using a very recent version of llvm - you can use our new 'deploy' script that builds the specific version of linux.
11:54:32
drmeister
Hmm, at least I think it is - but I might be wrong about that - I thought we were waiting for new changes to llvm but we may have come up with a solution that would work on llvm11.1
11:57:50
drmeister
You are the closest person I know to a waf expert other than ita - and he's been... unresponsive.
11:58:12
drmeister
But I have a lot of sympathy - constantly having to deal with people who don't understand your creation.
11:59:40
drmeister
Right now - when I go: ./waf distclean configure build_cboehmprecise it crashes very quickly with the following error message/backtrace...
12:00:57
drmeister
No such file or directory: '/Users/su-chris/Development/cando-future/build/boehmprecise/cclasp-boehmprecise'
12:01:35
drmeister
I have been screwing with the wscript file a lot to try and get it working - I probably put in some bugs that I don't know how to find.
12:05:04
attila_lendvai
if you already don't care about having a large external dependency (waf) for the build, then maybe it's not crazy to consider writing the build "script" in CL.
12:12:38
drmeister
Cando has been progressing nicely - the compiler is faster than it has ever been before and the code it generates is better and better.
12:13:33
drmeister
But the startup time for our most complex application - a jupyterlab environment has become unacceptable - 45 seconds on macOS. So this winter I implemented memory save/load for the boehm GC. We are about to switch to a new GC called MMTk.
12:13:54
drmeister
The snapshot save/load works - and last week I got the startup time down to 3.5 seconds. But loading the snapshot uses dlsym to lookup symbols to get addresses of functions and vtables at load time.
12:14:20
drmeister
That's when I ran into this problem that macOS doesn't export vtable symbols. It's a known problem - and within Apple there was discussion a few years ago to add C++ syntax to solve the problem - but that was never implemented.
12:16:32
drmeister
So I am implementing this wonky approach where I take control of the exported symbol list.
12:16:51
drmeister
I built a symbol list by hand last week - about 12,000 symbols that worked - but it was painful.
12:18:03
drmeister
So I implemented a feature in clasp that runs dladdr on every symbol we need (except vtables) and generates a list of symbols and concatenates it with symbols generated by searching the clasp C++ object files for patterns "__ZTV" (vtables) and "_wrapped_" wrapper functions.
12:19:22
attila_lendvai
drmeister, do you mind if i'll create a github issue for this, collecting/recording there what's relevant?
12:20:01
drmeister
When it's just me hammering on the keyboard I don't tend to bother - I can barely explain the problem as it is.
12:21:14
drmeister
I think I can then do ./waf build_bboehmprecise I'm kind of creeping up on the problem.
12:21:58
drmeister
I think I have the cclasp-boehmprecise executable as a dependency somewhere - or a cyclic dependency - but damned if I can find it.
12:25:13
drmeister
I want to put out a new version of clasp - with memory/snapshot save/load that lets you build everything to a single executable - we are a grown up Common Lisp implementation.
12:26:54
drmeister
Following that with ./waf build_bboehmprecise --- I'm currently trying this - but it's building bclasp now.
12:27:59
attila_lendvai
drmeister, i'll try to reproduce it, and record the relevant parts on the issue.
12:33:37
drmeister
Also - I just realized I'm building cando - and you are probably just building clasp.
12:39:22
drmeister
Hookay - the problem doesn't show up with clasp alone - of course not. It's something I'm screwing up in the wscript files in cando.
12:39:29
attila_lendvai
drmeister, i have created an issue: https://github.com/clasp-developers/clasp/issues/1161
12:40:21
drmeister
attila_lendvai: I've been looking in the wrong place for the problem. It's in the cando wscript files.
12:41:10
drmeister
I'm testing it now - but I pulled a fresh clasp and I'm running ./waf build_cboehmprecise
12:41:57
drmeister
I still want to talk with you about building clasp and what we can do to improve it.
12:42:26
drmeister
If we stick with waf - I'd like to understand it better. Or I understand it well enough and I've just been looking at the wrong wscript files for the problem.
12:46:42
drmeister
I'll update the issue as soon as I confirm that ./waf build_cboehmprecise works on clasp future branch