libera/#clasp - IRC Chatlog
Search
15:26:47
yitzi
drmeister: I am around also if needed. I don't exactly understand what not working though....since it is building on my machine.
15:28:05
Bike
i can get you a log later, but it's just saying that these libraries aren't on my system, which is true https://github.com/clasp-developers/clasp/blob/main/wscript#L119-L147
15:28:05
Colleen
Bike: drmeister said 23 minutes, 20 seconds ago: Is there a log of the build failure that you can show me?
15:35:14
drmeister
Bike: Can you paste a directory listing of the directory that contains any of these libraries?
15:42:54
drmeister
Bike: Oh - sorry - I thought you were trying to install on an Arch system. I thought that was a bit unexpected.
15:43:29
Bike
I am trying to build clasp from the repo on an arch system, and it is failing because I don't have these libraries
15:43:52
Bike
If I built from the aur package, it would build as yitzi patched it to not require these libraries, but my system would hang
15:44:01
drmeister
Right - ok - and it doesn't have any clang libraries - you mentioned you saw libclang.so - where was that?
15:44:55
yitzi
The hanging in the AUR build can probably be fixed by setting a limit on the thread count.
15:46:14
drmeister
I think there are two problems (1) Bikes problem - missing clang tooling libraries??? (2) This issue - which looks like too many threads (https://github.com/clasp-developers/clasp/issues/1243)
15:47:36
yitzi
Bike: if there are changes to the wscript.config I can update the AUR repo also. If someone wants comaintainer on that repo I can do that also.
15:47:43
drmeister
yitzi: Presumably you have all of the clang libraries listed here (https://github.com/clasp-developers/clasp/blob/main/wscript#L119-L147) somewhere on your system - is that correct?
15:49:00
yitzi
drmeister: No. The PKGBUILD does a sed and uses `1CLANG_LIBRARIES = [ 'clang-cpp' ]` instead. I've never been able to get those libraries to work with the stock clang on Arch.
15:49:44
yitzi
When I was building the llvm13 package myself using AUR those libraries where present.
15:51:17
drmeister
What about using `strace` to get a list of directories that your (yitzi) clasp loads at startup. Then we look for those on Bike's machine.
15:52:55
Bike
drmeister: i think that on both my and yitzi's machines, those libraries do not exist anywhere and clasp does not load them.
15:53:35
drmeister
Ok, I'm trying to debug build problems on operating systems I don't run and on machines I can't access with no logs.
15:56:51
Bike
later when i get on my computer i can try patching the wscript like the pkgbuild does, and build from repo with -j4 or something, and hopefully it will be successful and not hang
15:57:06
drmeister
Ok. So is it that (1) whatever mechanism the aur package for clasp is using to discover the clang libraries is not working on Bike's machine or (2) those libraries are still missing on Bike's machine (3) something else.
15:58:29
Bike
well like you said there's two problems that are basically unrelated. the first problem is that arch does not, as far as i can tell, provide the ast matcher libraries and the clang configure will not proceed unless they exist; this can be worked around by altering clang's configuration, since clasp doesn't seem to actually need these libraries,
15:58:30
Bike
except maybe to run the static analyzer. this workaround is what yitzi did in the arch pkgbuild, which is why that works
15:58:53
drmeister
In good news, on the llvm14 branch of clasp/cando on zeus the startup is under 2 seconds and everything works fine on linux and macOS. I also have the /tmp/perf-<pid>.map files working on linux but I don't yet have a consistent way of getting profiling information with call stacks.
16:01:03
drmeister
The ast matcher stuff can be pulled out of clasp into a separate extension - with some work. I don't have time to do that work right now.
16:02:07
drmeister
I'm writing a statement of work and budget for the DOD to literally save humanity.
16:04:16
yitzi
I did a grep on nm and got piles of stuff like _ZTVN5clang12ast_matchers8internal39matcher_hasExternalFormalLinkageMatcherE
16:18:09
yitzi
Bike: maybe we should just do `./waf build_cboehmprecise -j4` or something in the PKGBUILD? I would think most people would have at least four cores?
16:19:26
drmeister
Maybe we could figure out how much memory they have and number of cores and use that to limit the number of cores?
16:20:13
drmeister
Also, if we knew where it's failing - that would be helpful. We might be able to do something to reduce the memory needed if it's clasp that is causing the problem. There's C++ compilation, C++ linking and Common Lisp compilation that could be causing the problem.
16:20:40
drmeister
The crashes I see on zeus when I build llvm (not clasp) is when linking llvm executables and libraries with debug information on.
16:21:42
drmeister
zeus has 64MB of RAM - I still have to limit it to -j6 when linking or I have to drive 30 min to cycle the power because it locks up.
16:24:19
yitzi
I'll look at the AUR packaging guidelines to see what it says about PKGBUILD on multiple cores.
16:24:37
drmeister
It's the memory that matters. We tried a swap configuration on that machine that is terrible. The swap is on a software RAID system. Don't do that - bad, bad, bad.
16:25:28
yitzi
Yeah, there is a flag the user specifies in their pacman config for the number of cores.
16:25:38
drmeister
We had 32GB on zeus for a while and the problem was REALLY bad. I built with 2 cores during that time.
16:26:57
yitzi
Well, most people do a terrible job configuring swap when they do a Linux install. The "assisted" install that people are using for Arch didn't even set up a swap file the last time I tried it.
16:29:24
yitzi
I have 32gb on my machine and am able to build using all the cores. Then again I have something like 84gb in swap.
16:31:24
yitzi
Without a swapfile Linux has a meltdown when it runs out of memory (in clasp build). I was very surprised by how badly the memory allocation works in low memory situations. Then again, I know nothing about those.
17:02:52
yitzi
Is this really wise? https://github.com/clasp-developers/clasp/blob/d2e22cee78622932b0f42594a4ccb44f197367aa/wscript#L2456
17:03:49
yitzi
I understand the irritation of redoing scraping just because one changes the number of jobs, but is it okay to run the scraper with 8 jobs if the user has a limited number of cores?
17:13:58
yitzi
ok. The default makepkg does not define MAKEFLAGS. So I am thinking in the PKGBUILD to pick some `--jobs 8` with a fallback to whatever the user defines.
17:26:40
yitzi
We could also set `bld.jobs` to `sysctl -n hw.logicalcpu` on the mac and `nproc` on Linux as a default.