freenode/#clasp - IRC Chatlog
Search
15:57:32
scymtym
are you planning to set up a new jenkins server each time something has to be built or should it (the jenkins "master", that is) persist?
15:58:31
drmeister
So imagine a script that you run on a fresh ubuntu that does the following. (1) git clone github.com/drmeister/clasp.git $HOME/Dev/clasp (2) git clone github.com/drmeister/clasp.git $HOME/Dev/cando (3) git clone github.com/drmeister/cando.git $HOME/Dev/cando/extensions/cando (4) installs jenkins (5) installs jenkins configuration files that run makefile? targets in $HOME/Dev/clasp/makefile and
16:01:03
drmeister
Is it? I don't really know jenkins - I thought it was a web server that started jobs
16:02:29
scymtym
no, that's all true. the difference is that you normally don't install and configure jenkins for one build and then tear it down again
16:02:56
drmeister
This is a jenkins server that builds cando - but it is fatally flawed because jenkins runs in a Docker container and tries to run Docker containers within it: https://hub.docker.com/r/drmeister/cando-ci/
16:03:46
scymtym
since jenkins is often used for continuous integration, the same instance usually runs many jobs so that it can track and archive things over time
16:03:58
drmeister
Ok - I ask for that because we can use Amazon Web Services spot instances - which are very inexpensive (1.5cents/hour) - but they can be pulled at any time.
16:04:21
drmeister
So rapid set up and the expectation that they could be taken away at any moment is important.
16:05:23
scymtym
i see. you can also do that. in fact, automating (4) and (5) is one of the things we build tools for
16:07:46
drmeister
I'm not wed to jenkins at all - I don't even know the program. If something else would be better suited for this I would be happy to use that as well.
16:09:05
drmeister
I currently have a spot instance that I've been running for weeks to build docker images and run the static analyzer. It could disappear at any moment.
16:10:20
drmeister
Those template functions specify the classes and class-ranges (the actual header values are elsewhere in the file) for TYPEQ
16:10:26
Shinmera
Autobuild as it stands right now is not really suited for generating deliverables.
16:11:22
scymtym
is was going to ask what the motivation for using jenkins in this scenario was? jenkins' strengths, as i see it, are mostly not applicable to the setup-do-one-build-tear-down use-case
16:11:30
drmeister
Maybe I don't even want jenkins. Maybe I just want a bunch of scripts that provision a machine and do one build job?
16:12:35
drmeister
I suggested jenkins because that is what my friend set up for me - but it has that flaw that neither of us knew about until I spent a couple of days trying to get it to work.
16:14:28
drmeister
Dammit - there are 11 (eleven!) copies of that nglview-js-widgets directories in the Docker image that runs the cando jupyter notebook
16:14:43
scymtym
i think jenkins could be really useful for other things (assuming autobuild does not already cover that area), it just doesn't seem optimal for what you want to do in this case
16:16:39
drmeister
Bike: each template function will create a class symbol -> header-value or header-value range entry in a map.
16:26:31
Shinmera
scymtym: Autobuild maintains a git repository clone and creates a clean clone of that for each new commit that arrives from a remote. It then launches an arbitrary script to "build" that commit, whatever that might mean.
16:26:57
Shinmera
In its current form it can't distinguish between different kinds of actions you might perform, and it can't do stuff like providing a build fragment to the user.
16:28:22
drmeister
It would use the type specifier to either generate code to check the tag (fixnum, single-float, character, cons) or load the header-value for the object and compare it to what it gets out of the class symbol-> header-value(-range) map based on the type specifier argument (a symbol) of the typeq-instruction.
16:28:52
Shinmera
I want to fix those problems, as well as others, in the second iteration. I don't know when motivation to actually write it will strike me though
16:33:25
scymtym
Shinmera: what autobuild does sounds complementary to what we do. an option to replace the jenkins part with something else has been on my todo-list for some time. but that's probably off-topic and i have to leave now anyway. maybe we can talk about that later
16:36:05
drmeister
Bike: Hmm we will need an additional map for those. Those are represented by calls like: add_single_typeq_test<core::SimpleMDArray_byte8_t_O>();
16:37:45
drmeister
When you make an array of (simple-array ext:byte8) it gets translated to core::SimpleMDArray_byte8_t_O
16:37:51
Bike
it's not. i was just looking at make-array yesterday, it just passes the element type to a c++ function.
16:38:48
drmeister
Here: https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/lsp/predlib.lsp#L1363
16:40:41
drmeister
Would you be able to work from that? The map needs to go in the opposite direction for typeq's needs
16:41:28
drmeister
Either that or we change the way things are defined in predlib.lsp so that we specify the C++ class name <-> CL type specifier once and it builds both what predlib needs and what we need.
16:43:06
Bike
there should probably be a variable with information about array element types for optimization purposes
16:43:28
Bike
probably the uaet, the default initial value, the C++ simple vector class, and the C++ mdarray class
16:45:23
Bike
i think i can dig out the default initial elements, though, i'm not totally sure how to organize it overall
16:45:38
Bike
i'd like to inline make-array (this is a separate concern from typeq, ofc, but i mean, kinda related)
17:00:24
drmeister
Currently, the only place anything like (simple-vector ext:byte8) -> core::SimpleVector_byte8_t_O is expressed in Clasp is here:
17:01:35
drmeister
And I knew that had a bad code smell to it when I wrote it - but "when needs must the devil drives".
17:01:57
Bike
in the most common case of (make-array ... :element-type some-constant) we could just call StrWNs_O::make or whatever directly
17:02:53
drmeister
Is there a way in Common Lisp to reduce a vector or array type specifier to a canonical type specifier?
17:03:57
drmeister
Meaning we make the map of type-specifier -> header-value(-range) use EQUAL and throw the canonical type specifier for every array type in as a key
17:05:04
drmeister
Then the typeq-instruction takes whatever type specifier it is given, canonicalizes it, looks it up in the map and if it's there generates code and if not it generates a call to typep.
17:06:52
drmeister
There is a call generated like this: add_range_typeq_test<core::SimpleMDArray_O,core::SimpleMDArray_int32_t_O>();
17:11:55
drmeister
The values above are the stamp values, the header value for these are ((stamp<<4)|1)
18:15:34
Bike
something is conceptually weird with typeq and i'm not sure how to fix it. type inference wants typeq instructions that deal with a sublattice of types. but typeq also means actual checks of actual user-provided types, which should not be a sublattice, and which should be efficient code
18:44:56
Bike
i guess (if (typeq foo bar) baz bad) could be compiled as (if ([check] foo bar) (progn (the bar foo) baz) (progn (the (not bar) foo) bad))
19:21:58
drmeister
It maps class symbols that correspond to types to either a fixnum (header-value) or a (CONS header-value-min header-value-max)
19:22:49
drmeister
Bike: Do you have any quick fixes for the looong (40 min) compilation of that one form in ironclad?
19:23:16
drmeister
I need to build a new docker image for the nglview stuff and a quick fix there would save me 40 min.
19:24:37
Bike
uhm, depends no your parameters for quick fix. i could add a policy to turn off dynamic extent analysis specifically that you could use
19:24:53
drmeister
It turns out that the Python version of nglview was old and the Javascript code that the Common Lisp code was using would use the old Javascript if I ran the Python version first and then use the new Javascript code if I ran the Common Lisp version first.
19:25:57
Bike
or you could just (declaim (optimize (clasp-cleavir::analyze-flow nil))) and then it wouldn't do any type inference or dx or anything.
19:27:51
Bike
the less principled thing to do would just be to comment out dx from my-hir-transformations
19:35:58
drmeister
It will take a little while to shake out the bugs of the typeq-instruction code generation.
22:58:51
Bike
one input. if the input is of the instruction's type, go to one branch. otherwise go to the other
0:00:56
Bike
well, the upshot for us is that profiling is nice. though we have a lot more fires to put out before we can care about this, probably
0:33:21
drmeister
We can get rid of fixnump-instruction, consp-instruction, characterp-instruction and replace them with typeq-instruction.
0:34:41
Bike
well, what i want to do is keep the existing typeq-instruction, and have eliminate-typeq turn it into mir instructions, one of which is a primitive-typeq-instruction
0:35:04
Bike
because like, we want (typeq x (or cons null)) to be like (or (primitive-typeq x cons) (eq x nil)), not a typep call
0:38:28
Bike
might be ok to organize it as having fixnump, consp, characterp, single-float-p, and then have a uh, "headerq" that does the bit starting with let* ((header-check-br
0:40:54
Bike
eliminate-typeq has its type, it checks type-header-value-map, if the type is present it generates a headerq-instruction using the header-value-min-max in the table
0:42:50
Bike
really, for a constant type the only reason we need to call typep is if the type is undefined or it's a class(name), and the latter we can eliminate later
0:45:49
Bike
anyway, that snippet and the variables should be enough for me to figure out the rest myself
0:52:41
drmeister
I dunno - I haven't used it yet. It's part of this function: http://llvm.org/doxygen/classllvm_1_1IRBuilder.html#a3393497feaca1880ab3168ee3db1d7a4
0:53:24
drmeister
I tried to make sure the comparisons are always in the sense where branch-weights mean the same thing.
1:00:10
Bike
so, on this topic, in each running thread there's a top-of-the-shadow-stack variable that holds information about the currently executing lisp function, right?
1:03:10
drmeister
Each mp:process points to a thread-local structure that contains a pointer to the top of the invocation history stack.
1:05:56
drmeister
There will always be something. It's so painful to debug without it - I can't tolerate it.
1:06:22
Bike
statistical profiling is done by just checking where running code is every millisecond or whatever. so, we can just start up a profiler thread that checks the current function in the profiled thread
1:15:18
drmeister
Every Clasp function is like a C++ function. The big issue are anonymous lambda's - they are everywhere.
1:16:04
drmeister
But if you (defun foo (...) ...) in package BAR the function name is something like FN_BAR::FOO
1:24:00
drmeister
Maybe it's used to fingerprint the sample? joker-eph in #llvm told me that they get around the limitation by spawning the compiler in another thread.
1:30:28
drmeister
I'm really annoyed. I installed an older version of nglview according to the nglview instructions and then started developing code to mimic it. I got it working - but only when I ran the python version of the nglview widget first - that loaded the old Javascript code into the browser.
1:31:49
drmeister
So things took longer to get working because more than half the time I was running the Common Lisp code without running the Python version first and it loaded a newer version of the Javascript code that it wasn't designed to talk to properly.
1:35:27
drmeister
Annoyance #2 is now I am rebuilding the docker image with (hopefully) the latest version of everything and then I have to change the Common lisp code to bring it up to talk properly to the latest version of the Javascript.
3:24:00
beach
It is still very early in the morning here, so I may not be very coherent, but I think TYPEQ is meant for the type inferencer, so there is no need for it to handle arbitrary types; only the ones that the type inferencer can deal with.
3:25:34
Bike
okay, say you have (declare (type B var)), where B is a strict supertype of A, which is an inferencer type. and it's safe code. how's that compiled?
3:25:38
beach
Nobody says that a call to TYPEP with a constant type argument or a declaration should turn into a TYPEQ with the same type given.
3:27:01
Bike
i mean, the basic issue is that typeq is for the inferencer, but it also has actual semantics
3:33:03
Bike
say A is fixnum and B is integer = (or fixnum bignum). typep will redundantly test fixnum, unless it expands into typeq(s) itself
3:35:32
Bike
because then the inferencer would have one join/meet set of operations, and then there'd have to be another one at generate-ast level subtracting things out of full common lisp types
3:37:49
beach
It seems to me the alternative, if you want to do things once only, is to avoid type inference and have everything done by TYPEP.
3:40:54
Bike
right now i'm moving forward as if typeq can handle anything, and in the near future generate-ast will convert typeq's type specifier into an environment-independent implementation-defined type object. then probably some phase in hir but before type inference breaks typeqs apart into usable pieces, and then the inferencer simplifies further into inference types.
3:42:10
Bike
but i'm only just getting to the point of avoiding full typep calls anyway, so i'm just thinking ahead and worrying'
3:43:15
beach
I can't parse the phrase starting with "then probably...". But it is probably not important.
3:44:36
Bike
like (if (typeq x list) a b) is turned into (if (typeq x cons) a (if (typeq x null) a b)), so that if there's an earlier (typeq x cons) it can eliminate that part of the test
3:46:07
beach
That is why I don't think any THE or TYPEP or DECLARE should turn into the same TYPEQ.
3:47:11
beach
It seems to me very complicated to let TYPEQ handle everything, only to remove it later.
3:49:29
Bike
right now i'm not doing anything major to rock the boat. probably moving loop invariants will do more good than a lot of this stuff, but that doesn't mean it's irrelevant, i don't think
3:51:02
Bike
...moving code might also be easier with typeq, since it has known semantics, unlike a function call where there have to be annotations introduced or something
3:51:57
beach
The way I see it is that TYPEP needs to exist anyway. TYPEQ can handle a subset, and is used for type inference. I don't see the additional separate facility here, other than type inference, which we need for reasons of performance.
3:52:43
Bike
typeq is used for type inference but it also affects control flow based on types of objects, same as calls to typep
3:53:30
beach
So there is nothing wrong with a combination of TYPEQ and calls to TYPEP. No additional separate facility here.
3:55:49
beach
Yes, the compiler macro for TYPEP can be arbitrarily complex, so that it contains knowledge of what TYPEQ can handle and what it can't.