libera/#clasp - IRC Chatlog

12:26:00 drmeister The subgraph matching/specialization thing works like a charm

12:37:56 yitzi Supporting llvm17 will probably mean dropping support for llvm15 if it doesn't have new pass builder. Unless we want to maintain two different sif files for the precise builds

12:40:10 bike can we not use the new pass builder in llvm15? i thought the one we've been using has been legacy for several versions

12:42:15 yitzi I don't know. Some headers have been moved around in llvm17, but that isn't too big of a deal. I've started an llvm17 attempt so I'll check to see if at least llvm16 can handle it.

12:44:37 yitzi bike: What do you think about merging the arm stuff? M1 doesn't work but Linux ARM does.

12:46:11 drmeister Yeah - I'm not sure which version of llvm fully implemented the new pass manager. When I gave that talk in 2018? 2019? it wasn't'

12:49:11 drmeister Huh - there's a Ben Schafmeister on youtube - I wonder if we are related. https://www.youtube.com/@BenSchafmeister/featured

12:49:53 drmeister Sorry, I got distracted - 2018 - https://www.youtube.com/watch?v=mbdXeRBbgDM&t=8s

12:51:30 yitzi Then we may be able to support 15 <= LLVM_VERSION_MAJOR <= 17

12:57:12 yitzi BTW, I think the breakage in M1 is in bytecode_vm or clbind::WRAPPER_varaidicFunction...

13:00:54 bike oh, i thought it was working on M1

13:01:07 bike regardless, sure, i can merge

13:02:00 yitzi It is much better on M1, but it breaks during the build in generated-encodings.lisp ... I was a bit too optimistic

13:03:08 bike mergèd

13:04:02 yitzi thx

13:06:02 drmeister yitzi: So, I can install the Apple Silicon homebrew alongside the x86 version and install all the same kegs/breweries/octoberfests?

13:08:36 yitzi Yes. On mine it uses the x86 by default since M1 goes into /opt .... When I build on M1 I do `./koga --llvm-config=/opt/homebrew/... --pkg-config=/opt/homebrew/..."

13:08:59 yitzi I don't remember the specific paths ... I'm on a Linux box right now.

13:09:55 yitzi In the stuff that bike just merged I made it possible to bootstrap from an x86 sbcl in an arm build, btw.

13:10:55 drmeister Ok.

13:14:07 yitzi Its very important that you install everything needed into the M1 homebrew, including pkg-config. Otherwise it will get very confused.

13:16:47 drmeister When I'm in an arm shell and I `brew install pkg-config` it says it's installed already. `which pkg-config` says: `/usr/local/bin/pkg-config`

13:17:04 drmeister How do I force brew to install a keg?

13:18:18 yitzi You have to call the M1 brew. Its in /opt ... not /usr/local

13:18:28 drmeister Ah!

13:19:02 drmeister Got it. `/opt/homebrew/bin/brew install llvm@15`

13:19:37 yitzi I didn't try the M1 build on llvm 15 yet, so you may need 16

13:19:53 drmeister Oh - good point.

13:21:19 drmeister And the M1 stuff is merged right? I'll clone clasp and build it with M1 and poke around.

13:28:01 drmeister Why do you use x86 sbcl? Doesn't the arm version of sbcl work?

14:06:58 yitzi Either SBCL is fine. I just used the x86 one since it is in PATH so I don't have to do `sbcl --script koga ...` everytime. Instead `./koga ...` works.

14:11:50 drmeister Ah - for convenience.

14:13:25 drmeister yitzi: I'm ready to step into the wonderful world of homebrew arm - but this happened - is there something that I'm still missing?

14:13:27 drmeister https://www.irccloud.com/pastebin/RpolK2Up/

14:13:28 yitzi Yes, I want to eventually move more of the logic of "TARGET_OS_bla" out of koga and into claspconfig.h so that we can bootstrap from Lisps which may not have much in common.

14:13:58 drmeister I tried to install arm /opt/homebrew/bin/brew llvm@16 and it says it's already installed (it is in x86)

14:14:28 yitzi That seems fine

14:14:46 drmeister I did the same thing with llvm@15 and it did install llvm@15 into the /opt/homebrew hierarchy?

14:15:02 yitzi hold on...

14:15:10 drmeister Is it fine? I don't see llvm@16 in the output of `/opt/homebrew/bin/brew list`

14:15:27 yitzi its just called llvm

14:15:36 drmeister Ah!

14:15:45 drmeister Smart guy.

14:15:57 yitzi llvm is the "most recent version" When they release 17 it will change

14:16:00 yitzi :)

14:16:15 drmeister Indeed it is...

14:16:19 drmeister https://www.irccloud.com/pastebin/juGRNef0/

14:16:45 drmeister Ok, I'm feeling pretty good about this. I will keep installing things.

14:17:26 drmeister I read on the internet that you can install x86 and arm homebrew but you should not install the same kegs - that random person didn't know what they were talking about.

14:17:59 drmeister yitzi: When you get a chance - can you post the command you use to build clasp with this arm version?

14:18:04 yitzi Yep, so `./koga --llvm-config=/opt/homebrew/Cellar/llvm/16.0.6/bin/llvm-config --pkg-config=/opt/homebrew/Cellar/pkg-config/...." or something

14:18:10 drmeister And it is clasp that fails - right? Not cando?

14:18:20 drmeister Or at the very least, clasp fails.

14:18:23 yitzi Yes ... you just need to find the pkg-config path

14:18:48 drmeister Ok, I can find the pkg-config path.

14:19:08 yitzi You can use the x86 ninja. That shouldn't matter. In other words just `ninja -C build` is fine

14:19:27 drmeister Ok.

14:19:51 yitzi I feel like we are so close to M1 ... but I've probably been staring at it too long.

14:20:40 drmeister Yeah - I'll warm up lldb and poke around. I'll have to try and remember how we debugged things before the days of time travel.

14:20:59 drmeister ACTION thinks If only I had a time machine...

14:26:08 drmeister I'm really interested in getting the M1 native build because when I'm traveling I pull everything back to my M1 laptop and work on there.

14:27:48 yitzi GitHub M1 runners are supposed be here soon...so we can build M1 homebrew packages.

14:32:21 drmeister That would be awesome.

14:34:44 drmeister How much arm homebrew stuff do I need? llvm, pkg-config?

14:35:20 drmeister I'm a little mystified why I need pkg-config - that's a build thing - right? It's not really important though (unless it is).

14:36:40 yitzi pkg-config locates the libraries by it's own root. the x86 one one be able to find the arm libraries

14:37:31 yitzi You should need the same things as in x86

14:37:58 drmeister Locating libraries - right - that could get messy.

14:38:03 drmeister Good thing I asked.

14:43:45 yitzi I'll be back about noon

14:48:14 bike for some reason the recurring problem i have with the fcells is with the :test argument to make-hash-table. it's so silly

14:48:23 bike i've hit this in like three different ways now

14:53:21 bike and in this case it's because cleavir's irc-fdefinition loads the _Function directly instead of going through Symbol_O::symbolFunction like i naively assumed. ah well.

14:54:42 bike so to get this working right i guess i need to teach the compiler about the FunctionCell structure, and to avoid an extra indirection i'll need to futz with the faso stuff also

15:01:37 bike god, everything is just so much easier with bytecode.

15:19:18 drmeister I may go through _Function in more places. If you want to find them change the name of the _Function slot?

15:22:45 bike i actually think that's the only place - it's the only thing in the code referring to info.%symbol% (along with irc-setf-fdefinition of course)

15:23:14 bike and with the extra indirection put in everything mostly works, so now i just need to work on fasos until that's optimized away

15:58:05 yitzi drmeister: any luck with arm?

17:23:34 yitzi I'm a little unclear if the new pass manager supports codegen at all.

17:24:23 bike uh? like we can't use it for actual compilation?

17:26:21 yitzi I haven't found out how to do that. Some documents (maybe outdated) seem to suggest you still have to use the legacy one for that. https://llvm.org/docs/NewPassManager.html

17:26:30 yitzi See the very last sentence

17:27:40 yitzi Granted I find the LLVM docs are super confusing.

17:32:56 yitzi Hmmmm .... may it is now buildCodeGenPipeline vs addPassesToEmitFile

17:35:33 bike if they're actually axing the old pass manager it's hard to imagine it's still necessary.

17:35:43 bike so i think the page must be outta date.

17:36:26 yitzi Yeah, except addPassesToEmitFile is still in TargetMachine in the llvm repo

17:39:29 bike uugh, that the.9 test is causing trouble again

17:39:35 bike what the hell is this unidentifiable entry point thing

17:42:13 yitzi Looking the release notes it says that PassManagerBuilder has been removed, not that PassManagerBase has been removed.

18:33:19 yitzi Compiling with the new pass manager...fingers crossed

18:38:08 yitzi Seems like it works. I had to disable some optimizations b.c. I don't know what the new version is yet or I haven't exposed that stuff to Lisp yet. I replaced the addPasses..andRun lisp function with EMIT-MODULE that handles usage of the old pass manager for codegen, thereby shielding us from its existance.

18:53:26 drmeister yitzi: Does the --pkg-config path need the path to the bin directory?

18:56:18 yitzi It needs the path to pkgconfig binary

18:56:41 yitzi drmeister: ^

19:03:23 drmeister Got it.

19:04:09 drmeister I screwed up and hadn't installed the arm fmt library. I `/opt/homebrew/bin/brew install fmt` and then `ninja -C build` and it failed again linking fmt.

19:04:31 drmeister I'm thinking I need to `koga --deep-clean` each time to reset everything - does that sound right?

19:05:31 yitzi No. Just --clean

19:05:48 drmeister Ok, well it worked, now it's compiling Lisp code.

19:05:54 drmeister I'll do --clean next time.

19:06:02 yitzi deep clean deletes the repos which isn't needed between archs

19:06:27 yitzi Did you see that I got the new pass manager working on llvm16?

19:06:59 drmeister I did not. Yer takin' my jerbs!

19:07:33 yitzi hehe. I had to disable some optimizations and I haven't exposed it to Lisp yet. But it compiles and runs.

19:07:34 drmeister https://www.youtube.com/watch?v=QGmhLtsK2ZQ

19:07:58 yitzi nice

19:08:24 drmeister Is this the failure you see?

19:08:32 drmeister https://www.irccloud.com/pastebin/GIpxmjYh/

19:09:35 yitzi Yep. Although frame 52 is probably erroneous. I think the error is in frame 55 or frame 54

19:09:51 drmeister Thank you. Investigating

19:10:57 yitzi Somewhere in that area (translate-logical-path "real-path") gets turned into (translate-logical-path "random-junk")

19:12:00 yitzi Which made me think it was some kind of problem in bytecode_vm or Wrapper_Variadic...

19:20:56 drmeister How do I turn on compiling with fork?

19:21:12 drmeister use-parallel? parallel-build - searching...

19:22:10 yitzi It should be parallel always unless jobs=1

19:22:28 drmeister I want to disable fork building.

19:23:16 yitzi `./koga --jobs=1` probably

19:43:39 drmeister (:parallel-build nil) works as well.

19:43:44 drmeister In config.sexp

19:44:08 yitzi Gotcha. With koga that would be `--no-parallel-build`

19:44:45 yitzi Looks they have removed the last non-opaque pointer stuff in LLVM17. So there isn't a getPointerElementType anymore.

19:46:28 drmeister Were we using that anymore?

19:47:18 yitzi One usage in cmpliteral.lisp ... not sure if it is dead code

19:47:27 drmeister How would I load all the source-code->bytecode and then drop into a repl?

19:49:51 yitzi ninja -C build load_cclasp-boehmprecise

19:58:23 yitzi Looks like I've got llvm17 compilation also. If so then we can keep 15 & 16 support.

20:01:11 yitzi The new pass builder has a textual representation of the optimization pipeline that it can parse. That might be easier then exposing the classes that represent different passes to Lisp.

20:02:05 yitzi For example, you can do stuff like `module(function(fpass1,fpass2,fpass3)) module(cgscc(cgpass1,cgpass2,cgpass3)) module(function(loop(lpass1,lpass2,lpass3)))`

20:04:21 drmeister In serial build the problem reproduces - so after lunch I'll dig deeper

20:05:51 yitzi ok.

20:09:31 yitzi bike: I am suspicious that this is unused code https://github.com/clasp-developers/clasp/blob/main/src/lisp/kernel/cmp/cmpliteral.lisp#L571-L630

20:10:12 yitzi I can't find any references aside from itself. Do you remember if it is being used?

20:33:36 bike i don't remember, but if it's not referenced outside that file i doubt it's used. all the literal stuff is in its own package, after all

20:34:27 yitzi thx

22:06:29 drmeister The M1 problem is reproducible using: `ninja -C build load_cclasp-boehmprecise` followed by `(compile-file "sys:src;lisp;kernel;lsp;generated-encodings.lisp")`

22:18:41 drmeister bike: Can we get arguments in the bytecode backtraces?

22:18:55 drmeister I don't recall if you answered that when I asked a few days ago.

22:27:42 bike getting the original arguments is tricky since you'd have to look at the lcc_args in the frame, which I do not know how to do programmatically

22:27:59 bike of course everything ought to be available in the saved lexical variables, but that's not apparent in udb

23:09:29 bike i have function cells working with everything now, i think. have to make some cleavir changes for the first time in a while