libera/#clasp - IRC Chatlog
Search
14:41:41
drmeister
I'm running the ansi-tests and watching `perf top -p <pid>`. _Unwind_Find_FDE swings from a few percent to over 80% as we go from test to test.
15:04:14
Bike
https://gitlab.common-lisp.net/ansi-test/ansi-test/-/blob/master/auxiliary/char-aux.lsp#L49-61 and that in turn does catch-type-error...
15:07:19
Bike
if we need to, i could also do some other stuff to make the dynamic environments more informative, e.g. i could have the unknown-marker dynenvs store the name of the function they're from or something
15:07:20
drmeister
I don't know if ansi-tests prints the test before it starts it or after it completes it. So it may have been char-upcase.3 or char-upcase.5
15:11:11
Bike
hrm, for type errors we go through multiple-value-foreign-call, might be a bit of funkiness there
15:14:27
yitzi
drmeister: So I got this mocked out now `(clasp-user-install::install :snapshot '(:cando) :jupyter '(:clasp :cando :scando))`, which makes a cando snapshot and installs jupyter kernels for clasp, cando and scando. It also applies some logic to the kernel names. For example, if the user only installs an scando kernel then the kernel name is just "Cando". If they install both cando and scando then the kernel names are "Cando" and "Cando (snapshot)"
15:15:56
Bike
cppreference says it has a race condition and that it's better to use std::tmpfile or mkstemp
15:16:34
yitzi
Yes, but I was not really able to figure if mkstemp or really std::tmpfile really would work for that application.
15:17:00
Bike
alright, none of those look immediately like unwinding problems. some of the miscs might be
15:18:06
Bike
we fail a lot of the arithmetic because we have problems with SIGFPE, that's been the case for a while
15:18:18
yitzi
If there are tests we don't pass they should be added to expected failures list. Otherwise I can't make the GitHub CI test every give you a success.
15:18:26
Bike
the main thing leaping out at me here is the MAKE-LOAD-FORM.ORDER failures, but yitzi said that was in main too
15:19:06
drmeister
I see - so fix the unexpected failures or punt and move them into expected failures?
15:20:08
Bike
yeah, ok, as usual the misc tests are doing a bunch of obscure stuff that could be causing problems, i'll see what i can do about fixing them
15:21:02
Bike
https://gitlab.common-lisp.net/ansi-test/ansi-test/-/blob/master/misc/misc.lsp#L11082-11095 but like this doesn't look like it's unwinding, for example
15:21:17
yitzi
I can explain the tmpnam issue whenever. Maybe someone has a better solution then I do.
15:21:59
Bike
when debugging a single test i usually just dump the code into clasp without bothering with the test harness
15:22:24
yitzi
I have not added a way to do that via koga. I need to investigate how the test selection works.
15:22:47
Bike
https://gitlab.common-lisp.net/kpoeck/ansi-test/-/wikis/How-to-run-the-tests-for-clasp details here
15:23:45
Bike
so for example to run just the misc tests, you'd pass -f ansi-partial-execution -f ansi-load-misc
15:24:17
yitzi
If that is the case then you should be able to do `CLASP_FEATURES=ansi-partial-execution,ansi-load-misc ninja -C build ansi-test`
15:24:36
Bike
we're failing a bunch of the cl:bit tests actually, there's probably some weirdness there disconnected from the unwinder
15:24:39
drmeister
For later - running the ansi tests on zeus: Time real(3377.787 secs) run(3377.787 secs) consed(189978738168 bytes) interps(176) unwinds(130451)
15:26:13
Bike
oh, alright. i was trying to figure out if they indicated the new unwinder breaking anything, but i guess not
15:27:31
drmeister
`CLASP_FEATURES=ansi-partial-execution,ansi-load-misc ninja -C build ansi-test` works great!
15:30:33
drmeister
Of course all my timing data is shite because Bike is building on zeus as well. The unwinds will be useful.
15:31:36
drmeister
yitzi: Is this kpoeck's PR that you were talking about earlier? That would fix some tests? https://github.com/clasp-developers/clasp/pull/1295
15:34:00
Bike
merged clasp is built now, i'ts just loading cando. so i'll push the merge in a minute
15:34:41
yitzi
drmeister: No, I meant to update the "expected failures" list we would have to do a PR to kpoeck's fork of ansi-tests.
15:35:22
Bike
we might want to examine the need for kpoeck's fork. i think he originally did it because some of the tests would crash clasp, but we're hopefully past that now
15:35:44
Bike
and stuff like the feature input could be possibly be PR'd into the upstream ansi-tests
15:37:21
drmeister
I have MISC.316 failed - I'd like to know what test that corresponds to in the misc.lisp file. I think I'm asking a stupid question.
15:38:20
Bike
https://gitlab.common-lisp.net/ansi-test/ansi-test/-/blob/master/misc/misc.lsp#L6082-6098 here's misc.316. it uses cl:bit again
15:39:28
yitzi
Looks like kpoeck added "implementation specific expected failures" to his fork. Not in the main ansi-test repo.
15:40:26
yitzi
And there is this, which went nowhere. https://gitlab.common-lisp.net/ansi-test/ansi-test/-/merge_requests/22
15:41:49
Bike
maybe we could find a way to run the ansi-tests as a function that just returns the list of failures? and then we can compare that with our list. maybe that's overly exotic.
15:42:10
yitzi
What would really be nice is the ability to specify the expected failures with the call to start the ANSI tests. That way we could keep a local list.
15:42:22
drmeister
I have a list of tests that fail with main. Once you push - I can run ansi-tests again and compare the output.
15:43:24
yitzi
After the merge to main I can rebase and squash the testing PR. Maybe we should merge so we at least have testing on main (even if ANSI fails now)
15:48:39
Bike
it looks like the array type hierarchy is messed up such that a simple bit vector doesn't count as a bit array for some purposes
15:52:07
Bike
https://github.com/clasp-developers/clasp/issues/1253 here's a related bug with some explanation
15:52:29
drmeister
The class hierarchy looks ok in initClassesAndMethods_inc.h - nothing jumps out at me.
15:54:13
Bike
which is because mdarray-bit, rather than being a real abstract class, specifically means non-simple bit mdarrays to C++
15:55:41
Bike
yes, but we conflate the types here https://github.com/clasp-developers/clasp/blob/main/src/lisp/kernel/lsp/predlib.lisp#L1160-L1178
15:56:53
drmeister
udb appears to be working well - there are no huge pauses as it loads object files. I'm not sure why.
16:08:10
Bike
it varies. it's usually sorta possible to replace the runtime without swapping out any lisp, but that can sometimes hide problems that would arise if the lisp was compiled with the new runtime, so a lot of the time i just rebuild everything
16:11:58
Bike
complex vector is a subtype of mdarray, meaning we can't conflate mdarray with non-vectors
16:13:50
Bike
maybe we can just render MDArray an implementation detail instead of trying to treat it like a lisp type
16:19:48
drmeister
In every build we go from .sif files to clasp_gc.cc - right? Yeah - it's generated/clasp_gc.cc
16:21:55
drmeister
I think the deal was we run the static analyzer for clasp alone and then for each isolated extension + clasp.
16:22:23
drmeister
It should be. If there was cando classes - it should go in the cando extension directory.
16:23:30
drmeister
Maybe we are running the static analyzer with clasp+cando+seqan-lisp and it's getting confused and dropping it into the clasp/src directory?
16:24:43
drmeister
There wouldn't be a trace of seqan because it doesn't create any low level clbind classes.
16:26:27
drmeister
IIRC we should run analyze on (1) clasp (2) clasp+cando (3) clasp+seqan-clasp and it should put everything in clasp (1) or the extensions (2 & 3)
16:27:03
drmeister
There will be a lot of redundant information and we need to add some versioning info or hash or something to avoid problems.
16:27:36
Bike
wait, if i screw around with the array types i'll need to run the analyzer again, won't i. crud.
16:46:16
yitzi
I am getting an error in the build on main. Probably because the static analyzer sif file hasn't been updated in the cando repo.
16:47:27
drmeister
Then I'll create a clasp+cando repo and run the static analyzer there and make sure it does what it should - put the .sif file in the cando extension directories.
16:48:02
drmeister
If it does we may need to add something to koga to ensure that nobody runs analyze with more than one extension.
16:51:49
Bike
it is good that the analyzer works with my changes, since i didn't really examine clbind and stuff to make sure they're cooperative
16:52:51
drmeister
Right - because the static analyzer uses derivable classes and they go through lisp-> c++ -> lisp
17:08:42
Bike
if you're not seeing unwind_find_fde, it might mean they're not marked as unsafe properly
17:11:36
drmeister
https://github.com/clasp-developers/clasp/blob/main/include/clasp/asttooling/clangTooling.h#L153
17:14:50
drmeister
https://github.com/clasp-developers/clasp/blob/main/include/clasp/asttooling/clangTooling.h#L153
17:15:35
drmeister
Exposed here... https://github.com/clasp-developers/clasp/blob/main/src/asttooling/clangTooling.cc#L968
17:18:31
yitzi
kpoeck: on the clasp `ci` branch I have made the regression tests and ansi tests return a non zero error code on failure so the github workflow will function correctly. Once we figure what tests are not fixable we'll need to keep the "expected failure" list updated in your fork so the CI testing doesn't just error out all the time.
17:23:08
kpoeck
Yitzi i made more tests, also with your unicode branch. I tried to define 2 further tests, but get some errors both in your and in my branch. Could you please check whether the test in https://gist.github.com/kpoeck/e6cab4a26c65beb4b801e4a1acfa74ea are reasonable? For whats it worth, the work fine in sbcl
17:24:29
kpoeck
and a second question, how did you arrive the the exceptions for darwin in clasp_islower_wide? Same for clasp_isupper_wide
17:44:20
drmeister
Building ASDF with sjlj... Time real(46.737 secs) run(46.737 secs) consed(3895517400 bytes) interps(547) unwinds(0)
17:45:15
drmeister
Previously: Time real(51.441 secs) run(51.441 secs) consed(3907625424 bytes) interps(547) unwinds(6138)
18:02:01
yitzi
kpoeck: I came up with a code snippet, but forgot to save it. Give a sec and I'll reproduce the basics of it.
18:14:04
yitzi
kpoeck: This is essentially what I did. Warning, very kludgy. https://gist.github.com/yitzchak/404ee60f669b1791b035f99733115e97
18:26:17
yitzi
kpoeck: I really don't know what to do here. Seems like even the exceptions that I came up with don't fix stuff on all platforms at the same time. What do you think?
18:28:13
kpoeck
Perhaps we need to fix both what you did and also claspCharacter_downcase and claspCharacter_upcase?
18:29:35
kpoeck
what you did for clasp_isupper_wide and clasp_islower_wide looks fine (and a lot easier than my code)
18:31:10
yitzi
Or instead of comparing Linux to Mac, maybe compare to our downloaded UnicodeData.txt file?
18:32:13
yitzi
I think that would fit in what you are saying. Scan through everything for isupper, islower, toupper, tolower and check them adding exceptions as needed?
18:37:14
kpoeck
in sbcl - perhaps they still use an older unicode standard, this letter is not even recognized as lower-case-p
18:38:59
yitzi
Its kind of confusing. Is `lower-case-p` supposed to report on `Ll` category or if something has an uppercase letter? The title case letters also seem to be `lower-case-p = t`
18:42:13
yitzi
I haven't really thought about this, so I could not understanding the complete issue. But this http://clhs.lisp.se/Body/f_char_u.htm#char-upcase seems imply that a lowercase character has to have an uppercase equivalent.
18:43:27
Bike
13.1.4.3 says "Every character with case is in one-to-one correspondence with some other character with the opposite case." this is not how unicode works, so it's kind of problematic.
18:48:38
drmeister
Bike: the ASTMatcher library invokes the virtual method `run` and that invokes `_sym_run`
18:49:49
Bike
i think i have fixed some of the bit vector problems. one of the other misc ansi test failures is actually caused by an unwinding thing, think i have a fix for that as well
18:51:27
drmeister
Calling into the ASTMatcher library is done after we set up all the files we want to scan and all the matcher objects and then we call one function and it does everything.
18:51:58
Bike
how do we call that one function? do we go through a scraper-generated wrapper? do we do multiple-value-foreign-call?
18:57:06
drmeister
Bike: I think it's here: https://github.com/clasp-developers/clasp/blob/main/src/lisp/modules/clasp-analyzer/clasp-analyzer.lisp#L3947
18:57:49
drmeister
llvm updates are coming that will allow it to work on macOS - it cannot right now.
19:03:22
drmeister
https://github.com/clasp-developers/clasp/blob/main/src/lisp/modules/clang-tool/clang-tool.lisp#L850
19:03:23
kpoeck
but (char-upcase #\LATIN_SMALL_LETTER_B_WITH_STROKE) should be #\LATIN_CAPITAL_LETTER_B_WITH_STROKE and it isn't with std::toupper (tested with xcode)
19:05:32
Bike
these seem like they don't go through the scraper. i've never actually understood how the .def forms work
19:06:03
kpoeck
There seem to be 1390 unicode characters that have a uppercase characted perhaps we are really better of with a table approach
19:12:38
drmeister
Before you tackled it with the scraper I was going to add the dynamic environment to the template functions.
19:22:31
Bike
i did it in the scraper because i knew how to do so, more than out of an idea that it was the right way to do it
19:30:15
yitzi
kpoeck: Just for the heck of it I am trying this: https://plaster.tymoon.eu/view/3126#3126
19:37:27
drmeister
I'm running the static analyzer for clasp+cando now - I'll get to the bottom of what's going on.
19:54:04
yitzi
drmeister: So I think this is a bug. https://github.com/clasp-developers/clasp/blob/0e207e7cfea6548d34635001477e1b7f01885147/src/koga/ninja.lisp#L704
20:01:54
Bike
ooh, there's an ErrorSimpleDepthCounter thing i missed, so there's all kinds of spurious warnings about error depth during the ansi tests
20:02:25
Bike
actually, is this something else we should remove? it again seems ot use a global variable instead of a threadlocal
20:13:36
drmeister
It looks like analyze in koga writes to clasp/src/clasp_gc.sif. It's not getting whatever clue I had waf pass to the static analyzer to put it in extensions/cando/src/clasp_gc_cando.sif
20:15:40
drmeister
yitzi: Yes - we need to tell it to put the clasp_gc_cando.sif file in the extensions/cando/src/ directory.
20:33:34
Bike
i'm inclined to simply remove ErrorSimpleDepthCounter, since we already have another depth counter for errors in general
20:39:43
yitzi
kpoeck: Can you try the unicode-3 branch when you have time? Testing on Linux now, but it passed on Mac here.
20:43:02
drmeister
yitzi: Could you change that so that it uses the name of the single extension if one exists to define the path and the name of the sif file?
20:44:56
yitzi
The `(extensions configuration)` is a keyword list of the extensions. Not sure what you would do with `:seqan-clasp`
20:46:56
yitzi
Plus, there is a lot places where `(member :cando (extensions configuration))` is treated specially. I get what you are saying, but changing the logic there is gonna really fix anything.
23:07:11
pfd
Fortunately, I discovered that Devuan Ceres contains clang/llvm 13.01.x. Therefore I'm converting my Devuan Chimaera system(s) to Devuan Ceres. Hopefully ./koga will be approve.
23:18:12
yitzi
drmeister: Two things occurred to about the static analyzer. First could we make the sif files generated by it composable? Basically have tags recognized by the scrapper that identifies which sif file the stuff from the analyzer goes in? Or if that doesn't work, maybe don't put clasp_gc_cando.sif in the cando folder. Instead put all of them in the clasp folder with a name concatenated from the extensions, i.e. `(format nil "src/clasp_gc~{_~a~}"
23:18:26
pfd
I'm still in the process of updating packages to reach llvm13, through apt in Devuan. Previously I tried building LLVM 13 and installing it afterwards, but the process didn't proceed due to my system's ID not being 'Debian' or 'Ubuntu'
23:21:23
pfd
Yes, but it's not smart enough to know that Devuan is virtually identical to Debian. It doesn't proceed with the sysId saying 'Devuan'
23:22:12
pfd
As I stated above the bleeding edge / rolling version of Devuan has LLVM 13 within APT.
23:24:35
pfd
I just changed my repos list to make this massive shift. I'll just have to ignore/be selective about a lot of updates, if I want to stay relatively stable.
23:26:41
yitzi
Well, I've already gone to the darkside with Arch long ago so the claims of "stability" with Debian are foreign to me. ;)
23:30:28
pfd
I also still use CMUCL with Hemlock. Debian/Devuan is closest and most dedicated to that project and to GNU Hurd, etc.
2:18:15
drmeister
yitzi: I'm not sure about composable. The static analyzer crawls through all the code and finds all classes that have smart_ptr in them and maps out their offset.
2:18:47
drmeister
If we have a class in clasp and we inherit from it in Cando - then we need the layout of the clasp part.
2:21:49
drmeister
We have been able to improve things. It used to be that we had to run the static analyzer for every combination of extensions. That was because back then the stamps were calculated by the static analyzer.
2:22:17
drmeister
Now we leave that for the scraper and it can merge the .sif files from each extension into a correct layout of every class.
2:30:29
Bike
like generate the clasp sif and the cando-clasp sif, and then take everything out of the cando-clasp sif that's in the clasp sif
2:30:47
Bike
or, somewhat more sophisticatedly, change the analyzer so you can give it an input base sif, and it won't bother recursing into anything in the base
2:31:04
Bike
then we could set the base as the clasp sif and run it on cando-clasp and just get the cando stuff