freenode/#clasp - IRC Chatlog
Search
9:56:38
attila_lendvai
hi! drmeister, saw your mail, sent a response. i'll be around here for the next 10 or so hours.
11:37:31
drmeister
You did some work on the waf build system and reached a deeper understanding of it than I have.
11:38:21
attila_lendvai
keep in mind though, that it was a long time ago, and clasp was my only only exposure to waf
11:39:21
drmeister
I implemented snapshot save/load - I use dlsym at load time to lookup addresses of functions/vtable pointers at load time. Everything works on linux.
11:40:13
drmeister
On macOS however, vtable symbols for template classes are hidden/local so I have to go through a complicated song and dance to export them.
11:41:00
attila_lendvai
(re hidden vtable symbols: that's surprising, although i have abandoned my c++ knowledge long ago)
11:42:54
drmeister
It's basically (1) build iclasp-boehmprecise and compile the common lisp code for cclasp (2) run iclasp-boehmprecise loading cclasp common lisp image with a special flag and build a symbol list and save it to a file (3) LINK an executable called "cclasp-boehmprecise" that uses the new symbol list. (4) new build stuff that saves snapshots and links new executables that have the snapshots embedded within them.
11:43:26
drmeister
I've been screwing with the waf build system for days to add step 3 and I have something working - but it's a bit broken.
11:43:52
drmeister
I haven't figured out how to fix it and the clock is ticking loudly that I have to work on other things.
11:48:19
attila_lendvai
i never used a mac, but i compiled maru on it in a VM under linux, but the bugs drove me mad...
11:50:07
drmeister
The end goal is to get a working clasp - on macos and linux. It's probably broken on both operating systems - I'm not writing macos specific code here. We can use the same symbol list generating code on linux and get a slightly smaller executable.
11:51:09
drmeister
This clasp builds using a very recent version of llvm - you can use our new 'deploy' script that builds the specific version of linux.
11:54:32
drmeister
Hmm, at least I think it is - but I might be wrong about that - I thought we were waiting for new changes to llvm but we may have come up with a solution that would work on llvm11.1
11:57:50
drmeister
You are the closest person I know to a waf expert other than ita - and he's been... unresponsive.
11:58:12
drmeister
But I have a lot of sympathy - constantly having to deal with people who don't understand your creation.
11:59:40
drmeister
Right now - when I go: ./waf distclean configure build_cboehmprecise it crashes very quickly with the following error message/backtrace...
12:00:57
drmeister
No such file or directory: '/Users/su-chris/Development/cando-future/build/boehmprecise/cclasp-boehmprecise'
12:01:35
drmeister
I have been screwing with the wscript file a lot to try and get it working - I probably put in some bugs that I don't know how to find.
12:05:04
attila_lendvai
if you already don't care about having a large external dependency (waf) for the build, then maybe it's not crazy to consider writing the build "script" in CL.
12:12:38
drmeister
Cando has been progressing nicely - the compiler is faster than it has ever been before and the code it generates is better and better.
12:13:33
drmeister
But the startup time for our most complex application - a jupyterlab environment has become unacceptable - 45 seconds on macOS. So this winter I implemented memory save/load for the boehm GC. We are about to switch to a new GC called MMTk.
12:13:54
drmeister
The snapshot save/load works - and last week I got the startup time down to 3.5 seconds. But loading the snapshot uses dlsym to lookup symbols to get addresses of functions and vtables at load time.
12:14:20
drmeister
That's when I ran into this problem that macOS doesn't export vtable symbols. It's a known problem - and within Apple there was discussion a few years ago to add C++ syntax to solve the problem - but that was never implemented.
12:16:32
drmeister
So I am implementing this wonky approach where I take control of the exported symbol list.
12:16:51
drmeister
I built a symbol list by hand last week - about 12,000 symbols that worked - but it was painful.
12:18:03
drmeister
So I implemented a feature in clasp that runs dladdr on every symbol we need (except vtables) and generates a list of symbols and concatenates it with symbols generated by searching the clasp C++ object files for patterns "__ZTV" (vtables) and "_wrapped_" wrapper functions.
12:19:22
attila_lendvai
drmeister, do you mind if i'll create a github issue for this, collecting/recording there what's relevant?
12:20:01
drmeister
When it's just me hammering on the keyboard I don't tend to bother - I can barely explain the problem as it is.
12:21:14
drmeister
I think I can then do ./waf build_bboehmprecise I'm kind of creeping up on the problem.
12:21:58
drmeister
I think I have the cclasp-boehmprecise executable as a dependency somewhere - or a cyclic dependency - but damned if I can find it.
12:25:13
drmeister
I want to put out a new version of clasp - with memory/snapshot save/load that lets you build everything to a single executable - we are a grown up Common Lisp implementation.
12:26:54
drmeister
Following that with ./waf build_bboehmprecise --- I'm currently trying this - but it's building bclasp now.
12:27:59
attila_lendvai
drmeister, i'll try to reproduce it, and record the relevant parts on the issue.
12:33:37
drmeister
Also - I just realized I'm building cando - and you are probably just building clasp.
12:39:22
drmeister
Hookay - the problem doesn't show up with clasp alone - of course not. It's something I'm screwing up in the wscript files in cando.
12:39:29
attila_lendvai
drmeister, i have created an issue: https://github.com/clasp-developers/clasp/issues/1161
12:40:21
drmeister
attila_lendvai: I've been looking in the wrong place for the problem. It's in the cando wscript files.
12:41:10
drmeister
I'm testing it now - but I pulled a fresh clasp and I'm running ./waf build_cboehmprecise
12:41:57
drmeister
I still want to talk with you about building clasp and what we can do to improve it.
12:42:26
drmeister
If we stick with waf - I'd like to understand it better. Or I understand it well enough and I've just been looking at the wrong wscript files for the problem.
12:46:42
drmeister
I'll update the issue as soon as I confirm that ./waf build_cboehmprecise works on clasp future branch
12:54:03
drmeister
In this case there isn't much difference between starting up cclasp-boehm using the image or using the snapshot - but there is a difference
13:01:29
attila_lendvai
drmeister, hrm... it means that the issue only affects when building cando, and you have been looking for it in the wrong place, which means that you may be that much closer to resolving it for good?
13:20:13
drmeister
I completely forgot I was building cando - which has it's own wscript files and it's own problem.
13:20:32
drmeister
Could you do that thing though and contact Fare and ask him what he thinks is a good build solution these days?
13:21:05
drmeister
We have to (1) scrape C++ code (2) compile C++ code (3) build increasingly powerful versions of clasp/cando
13:22:24
drmeister
And we need to get to the end of the build development cycle with a build system that is easier to extend/work with than waf.
13:38:30
drmeister
So - yeah - I may have jumped the gun here in contacting you - I'm sorry about that.
13:38:51
SAL9000
drmeister: please let me know too if you find a better extensible build system in lieu of waf :-)
13:38:51
drmeister
While I was trying to get you up and running though - that's when I realized I was working with cando and not clasp.
13:40:59
drmeister
I didn't mean to sound like I was challenging you. I mean 'How are things going? What are you up to that also requires a better build system"?
13:41:43
SAL9000
We don't "need" a better build system per se -- the current one works -- but it'd be nice to clean up the mess
13:49:09
SAL9000
drmeister: if you meant what we do that is "non-standard"; we have a custom C++ preprocessor which has to be run multiple times (generated header files, then source files) on whole subdirectories ("packages") at a time.
13:51:50
attila_lendvai
drmeister, no worries, i also find it myself that explaining helps to understand what's going on. glad it works! i'll write a mail though, askin Fare
17:44:54
attila_lendvai
re cando bug: a rename_executable() in a build script? whatever problem it solves, it's a very bad idea... (and sorry for my misuse of IRC, it's slowly fading from my memory, too)