freenode/#clasp - IRC Chatlog
Search
14:24:36
kpoeck
Drmeister why did you change in load.cc the outputs for :verbose and :print to BFORMAT_T (initially eval::funcall(cl::_sym_format ....)
14:27:45
kpoeck
Can I change that back to eval::funcall, so that some ansi-tests that capture output from load work again?
14:31:12
drmeister
Ok, so I changed from my old, weird _lisp->print(...) to BFORMAT_T because BFORMAT_T prints properly to the standard output.
14:32:21
kpoeck
to eval::funcall(cl::_sym_format, _lisp->_true(), SimpleBaseString_O::make("~&;;; Loading ~s~%"), lsource);
14:33:26
drmeister
Right - but that may have trouble or at least look weird in the interpreter or aclasp - because format isn't fully installed at that point.
14:34:18
drmeister
The interpreter provides a seriously crippled version of format and only bclasp contains the proper version implemented in Common Lisp.
14:36:56
kpoeck
In compile-file there is describe-form using only write-string and terpri, is that bootstrap safe?
14:37:17
drmeister
What I mean to say is "yes", we can change it to eval::funcall(cl::_sym_format,...) to make it work properly.
14:37:43
drmeister
I'm just messing more with the interpreter and aclasp and it would be nice to have load print stuff properly in there as well.
14:39:51
kpoeck
Is there something like if (format-already-works) eval::funcalll .... else bformat();?
14:39:54
drmeister
If you can do it with write-string and terpri - then do that. If it has a problem - it's easier to back port a solution to the interpreter
14:41:54
drmeister
It writes out to *standard-output* - isn't that all that matters for load printing messages for :verbose and :print?
14:42:40
kpoeck
well and that (with-output-to-string (*standard-output*)(load ...)) captures the output
14:43:35
drmeister
Ah - so the output from BFORMAT_T doesn't get captured by (with-output-to-string (*standard-output*) ...) - that's not good.
14:47:32
drmeister
Ah - I see - that's the central problem. I use BFORMAT_T and BFORMAT a lot in the C++ code because I wanted something like format that worked right from the beginning - but it's not working right.
14:52:01
kpoeck
ok is #define BFORMAT_T(fmt) core::core__bformat(_lisp->_true(),(fmt).str(),_Nil<core::T_O>());
14:54:34
kpoeck
(with-output-to-string (*standard-output*)(core:bformat *standard-output* "Foo")) -> "Foo"
14:54:56
drmeister
It's not working properly (with-output-to-string (*standard-output*) (bformat t "whatever")) should return a string containing "whatever"
14:55:16
kpoeck
and (with-output-to-string (*standard-output*)(core:bformat *standard-output* "Foo")) works
14:56:57
drmeister
Yeah - if you pass T to it you get cl::*terminal-io* rather than cl::*standard-output*
15:00:15
Bike
it shouldn't be hard. destination can be T, NIL, a string with a fill pointer, or a stream. T means standard output, nil means write to string, a string means put it on the end, and a stream is itself;.
15:02:23
Bike
we handle it fine for actual format https://github.com/clasp-developers/clasp/blob/master/src/lisp/kernel/lsp/format.lsp#L505-L517
15:02:41
kpoeck
Bike, do you say that changing outputStreamDesignator to return *standard-output* instead of *terminal-io* is wrong?
15:03:40
Bike
if bformat is supposed to mimic format, it shouldn't be using an output stream designator.
15:05:02
kpoeck
http://www.lispworks.com/documentation/HyperSpec/Body/26_glo_s.htm#stream_designator ?
15:06:16
kpoeck
nil (denoting the value of *standard-input* for input stream designators or denoting the value of *standard-output* for output stream designators)
15:07:49
drmeister
NIL converts to my_thread->_BFormatStringOutputStream - a thread local string-output-stream ... for speed.
15:08:00
kpoeck
yes, but than I would expect (with-output-to-string (*standard-output*)(core:bformat nil "Foo")) to return "Foo"
15:09:04
Bike
bformat returns the wrong thing, i guess? it has nothing to do with outputStreamDesignator though.
15:10:41
drmeister
Something else must be wrong kpoeck - I'll compile the change above and then see what's going on.
15:12:55
drmeister
I'm seeing at least two bugs - I'll fixed one - it's compiling - and then i'll take a look at the next one.
15:14:37
kpoeck
I have written some regression to test to verify correct behaviour, so I am prepared to test :-)
15:18:45
drmeister
Right - it would be more consistent with the rest of the code to keep using BFORMAT
15:24:13
Bike
as far as those go, i looked at https://github.com/clasp-developers/clasp/issues/522 but i'm a bit stumped
15:25:02
Bike
i don't see anything that actually alters the fill pointer, so i think it's maybe returning the underlying data vector, or something
15:31:54
drmeister
There's no second bug: (with-output-to-string (*standard-output*) (format nil "Hello")) --> "" is correct.
15:35:49
karlosz
is there a way to invoke partial inlining without the liveness analysis? i do my own dead code elimination later - i feel like the liveness computation outweighs any gains in not analyzing dead variables
15:41:10
drmeister
(with-output-to-string (*standard-output*) (format nil "Hello")) --> "" - there is no output being written to *standard-output* so the form returns an empty string.
15:50:13
Bike
oh, and as long as you're looking at aref stuff... is that TEMPLATED_HALF_STRING whatever stuff actually helpful? does more type tests, i guess, and ends upw ith a specialized function...?
15:55:11
drmeister
nsvector.asAbstractSimpleVectorRange(bsv,start,end) fills bsv, start, end with the backing simple vector, the start in the bsv and the end in the bsv.
15:55:34
drmeister
Then it's doing the nreverse on that. I think I can just return the original non-simple vector
16:05:18
drmeister
Oh yeah - its to deal with all the combinations of SimpleString, SimpleBaseString, Str8Ns, StrWNs
16:05:41
Bike
is it required, though? the general equalp method just worries about simple versus mdarray
16:07:51
drmeister
You could do everything with a general equalp - but there will be type checking when comparing every character - bleh.
16:08:21
drmeister
I wanted to check the types outside of the loop and then have specialized loops for every combination.
16:08:30
Bike
i feel like for equal and string= and stuff we might be able to swing memcmp, but i'm not 100% sure how the characters are stored in the arrays
16:11:01
drmeister
There are 8bit characters and 32 bit characters - we don't want tests in the hot loops
16:12:00
drmeister
The functions that do the actual comparisons are template functions - so you could specialize them for the base-char/base-char pair with memcmp
16:13:00
drmeister
But the base-char/character and character/base-char case - you can't - of course.
16:51:00
drmeister
::notify kpoeck I fixed the bformat problem - is there an issue that deals with (with-output-to-string (*standard-output*) (load ...)) or issues that would be effected by this that we can reevaluate if they can be closed?
16:57:18
kpoeck
Drmeister: I defined regression tests regarding the (with-output-to-string (... (load)) and (with-output-to-string (... (compile-file ))) so I can easily retest
16:57:18
Colleen
kpoeck: drmeister said 6 minutes, 18 seconds ago: I fixed the bformat problem - is there an issue that deals with (with-output-to-string (*standard-output*) (load ...)) or issues that would be effected by this that we can reevaluate if they can be closed?
17:16:52
drmeister
kpoeck: I accepted all of your pull requests except one - you had a question in a comment about using strtof rather than strtod - I think you are correct and asked if you could go a bit further and make the change to use strtof.
17:34:08
nivpgir
remember me as that guy that couldn't build clasp 2 days ago? any chance someone here is willing to look at my errors now?
17:37:23
nivpgir
File "/home/nivpgir/gits/clasp/.waf3-2.0.6-bca992597e6f087c1b15baec6344c36f/waflib/Scripting.py", line 118, in waf_entry_point
17:37:23
nivpgir
File "/home/nivpgir/gits/clasp/.waf3-2.0.6-bca992597e6f087c1b15baec6344c36f/waflib/Scripting.py", line 178, in run_commands
17:37:27
nivpgir
File "/home/nivpgir/gits/clasp/.waf3-2.0.6-bca992597e6f087c1b15baec6344c36f/waflib/Scripting.py", line 169, in run_command
17:37:32
nivpgir
File "/home/nivpgir/gits/clasp/.waf3-2.0.6-bca992597e6f087c1b15baec6344c36f/waflib/Scripting.py", line 366, in execute
17:37:36
nivpgir
File "/home/nivpgir/gits/clasp/.waf3-2.0.6-bca992597e6f087c1b15baec6344c36f/waflib/Build.py", line 93, in execute
17:37:40
nivpgir
File "/home/nivpgir/gits/clasp/.waf3-2.0.6-bca992597e6f087c1b15baec6344c36f/waflib/Build.py", line 100, in execute_build
17:37:47
nivpgir
File "/home/nivpgir/gits/clasp/.waf3-2.0.6-bca992597e6f087c1b15baec6344c36f/waflib/Tools/errcheck.py", line 140, in check_compile
17:37:51
nivpgir
File "/home/nivpgir/gits/clasp/.waf3-2.0.6-bca992597e6f087c1b15baec6344c36f/waflib/Build.py", line 176, in compile
17:37:56
nivpgir
File "/home/nivpgir/gits/clasp/.waf3-2.0.6-bca992597e6f087c1b15baec6344c36f/waflib/Errors.py", line 26, in __init__
17:38:07
nivpgir
{task 139637121536632: link_fasl boehm-fastgf-cxx.a,boehm-builtins-cxx.a,boehm-intrinsics-cxx.a,start.o,prologue.o,min-start.o,init.o,after-init.o,runtime-info.o,jit-setup.o,clsymbols.o,packages.o,foundation.o,export.o,defmacro.o,helpfile.o,evalmacros.o,claspmacros.o,source-transformations.o,arraylib.o,setf.o,listlib.o,mislib.o,defstruct.o,predlib.o,cdr-5.o,seq.o,cmuutil.o,seqmacros.o,seqlib.o,iolib.o,sharpmacros.o,backtrace.o,trace.o,
17:38:09
nivpgir
cmpexports.o,cmpsetup.o,cmpglobals.o,cmputil.o,cmpintrinsics.o,primitives.o,cmpir.o,cmpeh.o,debuginfo.o,codegen-vars.o,arguments.o,cmplambda.o,cmprunall.o,cmpliteral.o,typeq.o,codegen-special-form.o,codegen.o,compile.o,codegen-toplevel.o,compile-file.o,disassemble.o,external-clang.o,cmpname.o,cmpbundle.o,cmprepl.o,min-pre-epilogue.o,epilogue-aclasp.o,min-end.o -> aclasp-boehm-image.fasl}
17:38:20
nivpgir
File '/home/nivpgir/gits/clasp/.waf3-2.0.6-bca992597e6f087c1b15baec6344c36f/waflib/Scripting.py', line 130, in waf_entry_point
17:39:51
nivpgir
yeah, I'm sorry... I'm new to IRC, I didn't predict it would come out like this ...
17:41:31
Bike
"requires dynamic R_X86_64_PC32 reloc against '_ZZN5boost13serialization16singleton_module8get_lockEvE4lock' which may overflow at runtime; recompile with -fPIC". well, that's new to me
17:54:14
drmeister
Because I think there is an incompatibility between the boost library and what Clasp expects or what boost expects or something.
17:56:32
drmeister
Although there may be an issue on the Clasp side - clasp is looking for a static version of the boost library - that may not be a good idea.
17:57:26
drmeister
With waf you either look for the static library or the dynamic library. My understanding is that I have to hard code the search in the wscript file.
17:58:41
drmeister
I started with static libraries for everything because I want to run this on supercomputers that require static linking. I should probably make it switchable with an option.
17:59:19
drmeister
I'm going to try that - I'll add an option and make the default dynamic libraries.
17:59:42
drmeister
We haven't had a test case like this for a while - are you working on this for the next couple of hours - could you work with me to resolve this?
18:01:03
drmeister
Then I'll push - then if you could try it and report back - that would be very helpful.
18:05:05
drmeister
No, it's ok - there are two places that need to be changed. I'm also pretty sure we should link boost dynamically in most cases.
18:12:47
drmeister
If there were a problem in array.h then I should have seen it as well - mine is building fine [63/190]
18:13:19
nivpgir
oh, no, I didn't, I was hoping it's not necessary cause if I do it's gonna take hours
18:22:19
drmeister
Clasp is fundamentally a C++ project - with long C++ project build times. Even when Clasp is building things - more than half of the compilation time is spent in llvm - which is a C++ backend.
18:27:53
drmeister
Good news - on my system (macOS) I switched to dynamic linking of the boost libraries and it works fine.
18:41:28
drmeister
Since we are doing full inlining by repeatedly using partial inlining - that we are spending a lot of time doing partial inlining?
18:43:01
drmeister
When we do HIR inlining we are doing full inlining of functions by repeatedly doing partial inlining. It sounds like this is very expensive - is that the case?
18:45:04
karlosz
im not so familiar with the code, but if the inlining is to be done at the hir level and not the ast level then either way youd have to copy all the instructions to be inlined
18:49:00
karlosz
would it be more than just adding the copied instruction to the ownership table with the caller as its owner?
18:59:00
drmeister
Some other way - whole function at a time. The "trivial" way that compiler books say is "trivial".
18:59:37
drmeister
How much time is spent recomputing ownership information? We can profile that with dtrace.
19:05:34
Bike
i couldn't tell you how long it takes. i'm making an estimated guess based on my dumb code
19:26:50
drmeister
Shit - I was doing a (gctools:garbage-collect) for every top-level form read into the REPL.
19:32:20
karlosz
and it should account for at least 75% of the time in partial inlining, if im reading the profile correctly
19:32:29
drmeister
I think I thought in an interactive session it would be a reasonable thing to do. But the same code was being used during LOAD
19:38:37
Shinmera
It could be more excusable if it happens during a REPL clear (C-c M-o) but the implementation doesn't know about that.
19:40:17
drmeister
I saw it immediately now that I can profile MPS. count-calls showed 64% of the time was in gctools__garbage_collecte
19:41:15
drmeister
It never showed up with Boehm - I guess Boehm has a test and returns early if there is no point to a GC.
19:42:46
drmeister
./waf build_bmps (after ./waf build_amps was run) takes 8m54s - that's comparable to Boehm.
19:46:41
drmeister
This is the first time I am running using MPS with the new bclasp optimizations I added a few months ago - the ones that rewrite the llvm-ir to move lexical variables that can be register out of closures.
19:49:30
drmeister
So, I have this big, curved Samsung monitor - I think when I plug it in to my macbook pro that compilation slows down - could that be?
19:52:11
Shinmera
Either way, for instance a 1080p truecolor image only takes up about 6MB. Hardly noteworthy.
19:54:15
drmeister
It was every two years - but this looks like a significant refresh and we need another mac laptop.
19:55:19
Shinmera
I wonder when I'll buy a new workstation. Had this baby for I think 7 years now and it still runs just fine.
19:57:41
drmeister
If I can shave a few minutes off of clasp build time - cumulatively it will improve my quality of life.
20:01:03
drmeister
Bike: Could you write a make-load-form-saving-slots for classes and slot-descriptors?
20:01:50
drmeister
Doesn't have to be now. But I'd like to explore generating code to save the system.
20:02:44
Shinmera
drmeister: No, I mean setting up your new system until everything runs again as you need it to
20:04:14
drmeister
I think I do a clean rebuild this time though - I think I'm accumulating problems.
20:06:24
drmeister
But if I'm compiling on all cores and close it - I might as well have cycled the power.
20:17:51
nivpgir
at first my laptop froze for a while, and when it returned the build crashed, so I thought It was related, so I ran ./waf build_... again, and it compiled a few more files, but then crashed again on the same error
20:27:00
drmeister
Ok, previously ... 'STLIB': ['boost_filesystem', 'boost_date_time', 'boost_program_options', 'boost_system', 'boost_iostreams'],
20:30:08
drmeister
Could you do that? Then I'm sure I'm not wasting time chasing some phantom problem.
20:30:28
nivpgir
but with one job I'm afraid it's gonna have to stay running for the night, it's getting late here
20:30:55
drmeister
Ok. I'll be on tonight and tomorrow - if you come back with the results and they are the same I'll dig deeper.
20:37:45
drmeister
I really thought switching from static to dynamic libraries would make the difference.
20:40:07
nivpgir
my initial though was that I didn't install the correct dependencies, that I didn't convert package names correctly somehow
20:41:10
drmeister
What we are seeing looks exactly like what is described here: https://tecnocode.co.uk/2014/10/01/dynamic-relocs-runtime-overflows-and-fpic/
20:43:26
drmeister
We followed the prescription "Another (highly related) solution is to link against a shared version of the static object."
20:43:50
drmeister
So we had a static object, we switched to the dynamic object and the problem persists?
21:28:09
drmeister
Once compile-file starts MPS slows down because mps_ap_fill takes 67.4% of the time.
21:28:32
drmeister
That's the function that is called when an allocation point runs out of space and has to allocate a new page to start filling.
21:59:43
drmeister
Cleavir uses hash-tables to traverse trees of instructions. It has to rapidly build up a hash table and then it throws it away.
22:30:48
drmeister
Bike: beach suggested that we could convert the map-instructions-xxx instructions into generic functions so that we can redefine them in clasp.
22:31:52
Bike
would take a lot of editing, and i don't know if it would be hugely better if we did the thread local thing
22:31:56
drmeister
That means adding a required parameter to each of them and defining a dynamic variable cleavir-ir:*instruction-mapper* that is by default NIL and calls the current functions.
22:34:22
drmeister
But is this how we would do it? Right - can't we define a special variable (defvar cleavir-ir:*instruction-mapper* nil) and then pass that everywhere in Cleavir? Or do we need to add a system argument to their callers and to their callers and so on?
22:35:06
Bike
by thread local thing i meant the "putting an extra slot in instructions that indicates whether they've been mapped over" dealie
22:36:39
drmeister
It wouldn't be hugely different but for the fact that allocations due to map-instructions-xxx are hammering MPS.
22:37:32
drmeister
There are there options - but it's hard to figure out if they will improve things wrt MPS performance until we try them.
22:38:02
drmeister
(1) Move to open hashed hash-tables - eliminate cons cells from hash tables. There are other good reasons to do this.
22:38:49
drmeister
(2) use stealth mixins and rewrite map-instructions-xxx to add a 'touched' slot to instructions.
22:43:21
drmeister
Well, I would argue that it's a bit perverse to be allocating so much memory just to walk trees over and over again.
22:45:20
drmeister
I probably won't hear from Ravenbrook until next week to get any ideas of how to improve this from the MPS size.
22:46:35
drmeister
Do we need to pass a system parameter into every function that calls map-instructions-xxx and so on?
22:47:01
Bike
either that or have a dynamic variable for the system, or something, but that would be unusual yeah
0:31:20
Bike
why would mps_ap_fill be called more by one kind of allocation in particular? unless that allocation is really common i guess...
0:33:28
drmeister
It is pretty common - every time you enter a function. They are supposed to be pruned if there is no return-from that can target them
0:49:01
Bike
i mean, if i do like (concatenate 'string "foo" "bar") i thought it'd give back a character strin
0:51:34
drmeister
foo allocate bar allocate baz allocate biz allocate tang allocate wang allocate cc_gatherRestArguments allocate
0:52:54
drmeister
How difficult would it be to write an optimization that checks if the rest variable escapes and if it doesn't use a &va-rest?
0:55:34
Bike
i was going to ask if it's called by the caller or by the &rest function itself, but i guess that's obvious
0:57:52
Bike
even without my fancy optimization, i could put in something to eliminate almost all calls to the symbol method. might as well give it a shot i guess.
0:59:16
Bike
though... how are there 129 calls to make-instance/symbol and only 3 to make-instance/class
1:02:19
drmeister
https://github.com/clasp-developers/clasp/blob/dev/src/lisp/kernel/clos/builtin.lsp#L40
1:03:37
Bike
like i said, that one specifically i can deal with by other means. i'm wondering about some other ones, though
1:04:56
Bike
the standard allocate-instance methods ignore the initargs, but i guess the compiler doesn't know that so it gathers them anyway