freenode/#clasp - IRC Chatlog
Search
15:19:06
drmeister
Yesterday it seemed like sometimes when I made changes and then hit Command-s (save?) - the changes seemed to take.
15:19:37
drmeister
I could put console.log("something") and the "something" would print to the console when I thought that code should execute.
15:20:17
drmeister
Am I supposed to be able to edit the code there and the browser will recompile it?
15:21:18
drmeister
I can edit the source code on the host beforehand - that always works, but it would be really convenient if I could edit the code in that window for debugging.
15:21:45
Shinmera
JS is, at its basis, just a sequential language that executes top level statements like in a script. When you save an already loaded file, the browser probably does some magic to try and figure out which things are "static definitions" that it can update and which are not.
15:22:03
Shinmera
This means that stuff it deems dynamic, or things that really are dynamically constructed at runtime, won't be affected by your change.
15:25:51
Shinmera
This is pretty ingenious. I don't really know how it knows what it can update or how.
15:35:13
drmeister
Searching on the web suggests it is a widespread issue that the code in the development console doesn't always compile when changes are made to it.
15:37:29
drmeister
The next question - is there a way to figure out where the extension is coming from on the host system?
15:39:56
drmeister
I thought I had two versions of this nglview-js-widgets javascript code in the docker image. I just dug out the version#2, put it in the widget-dev directory and mounted it back into the docker image. Changes to the index.js file in the widget-dev/nglview-js-widgets/index.js are not showing up in the Javascript console - Grrrrr.
15:41:26
Shinmera
If you have the network tab open while the page loads it should list all sources, how long they load, and so forth
15:42:51
Shinmera
all the browser will know about though is the address. to which physical file that might map it cannot possibly know
15:43:09
Shinmera
there might not even be one or the server might do whatever it wants to deliver the content.
15:49:28
scymtym
drmeister: i managed to generate a group of jenkins jobs that builds llvm 4.0 and clasp, but not cando, yet (based on (almost) this model: https://techfak.de/~jmoringe/clasp-using-vanilla-llvm-nightly.png ). this can be used to build inside or outside of docker or generally for different purposes. can you explain the goal in more detail? for example, should the container contain the build environment or should its size
15:50:56
drmeister
We don't need to build llvm4.0 and clang - they can be obtained using the package manager.
15:51:38
scymtym
Bike: thanks, i though erc handled that. the remainder was just "… or should its size be minimal?"
15:52:34
drmeister
What I'm looking for is a jenkins server that can run certain docker-compose commands as if they were run from the clasp/ root directory.
15:53:36
drmeister
And the configuration files to set up that jenkins server on an Amazon Web services machine.
15:54:32
drmeister
The idea is a push button server that can spin up in a few minutes, provide jenkins that we can connect in to with the browser and allow the user to select a job that will build (1) a cando docker image (2) run the static analyzer (3) a clasp docker image.
15:54:45
scymtym
drmeister: there's a funny reason for that. the machine on which i have root (and docker) is not beefy enough. on the beefy machine, i don't have root (and no docker) and it runs ubuntu xenial. so i had to build llvm there. i made multiple variants, though: using-vanilla-llvm, using-externals-clasp, using-system-provided-packages
15:54:52
drmeister
Actually, there are two static analyzer jobs - one for clasp and another for cando.
15:57:32
scymtym
are you planning to set up a new jenkins server each time something has to be built or should it (the jenkins "master", that is) persist?
15:58:31
drmeister
So imagine a script that you run on a fresh ubuntu that does the following. (1) git clone github.com/drmeister/clasp.git $HOME/Dev/clasp (2) git clone github.com/drmeister/clasp.git $HOME/Dev/cando (3) git clone github.com/drmeister/cando.git $HOME/Dev/cando/extensions/cando (4) installs jenkins (5) installs jenkins configuration files that run makefile? targets in $HOME/Dev/clasp/makefile and
16:01:03
drmeister
Is it? I don't really know jenkins - I thought it was a web server that started jobs
16:02:29
scymtym
no, that's all true. the difference is that you normally don't install and configure jenkins for one build and then tear it down again
16:02:56
drmeister
This is a jenkins server that builds cando - but it is fatally flawed because jenkins runs in a Docker container and tries to run Docker containers within it: https://hub.docker.com/r/drmeister/cando-ci/
16:03:46
scymtym
since jenkins is often used for continuous integration, the same instance usually runs many jobs so that it can track and archive things over time
16:03:58
drmeister
Ok - I ask for that because we can use Amazon Web Services spot instances - which are very inexpensive (1.5cents/hour) - but they can be pulled at any time.
16:04:21
drmeister
So rapid set up and the expectation that they could be taken away at any moment is important.
16:05:23
scymtym
i see. you can also do that. in fact, automating (4) and (5) is one of the things we build tools for
16:07:46
drmeister
I'm not wed to jenkins at all - I don't even know the program. If something else would be better suited for this I would be happy to use that as well.
16:09:05
drmeister
I currently have a spot instance that I've been running for weeks to build docker images and run the static analyzer. It could disappear at any moment.
16:10:20
drmeister
Those template functions specify the classes and class-ranges (the actual header values are elsewhere in the file) for TYPEQ
16:10:26
Shinmera
Autobuild as it stands right now is not really suited for generating deliverables.
16:11:22
scymtym
is was going to ask what the motivation for using jenkins in this scenario was? jenkins' strengths, as i see it, are mostly not applicable to the setup-do-one-build-tear-down use-case
16:11:30
drmeister
Maybe I don't even want jenkins. Maybe I just want a bunch of scripts that provision a machine and do one build job?
16:12:35
drmeister
I suggested jenkins because that is what my friend set up for me - but it has that flaw that neither of us knew about until I spent a couple of days trying to get it to work.
16:14:28
drmeister
Dammit - there are 11 (eleven!) copies of that nglview-js-widgets directories in the Docker image that runs the cando jupyter notebook
16:14:43
scymtym
i think jenkins could be really useful for other things (assuming autobuild does not already cover that area), it just doesn't seem optimal for what you want to do in this case
16:16:39
drmeister
Bike: each template function will create a class symbol -> header-value or header-value range entry in a map.
16:26:31
Shinmera
scymtym: Autobuild maintains a git repository clone and creates a clean clone of that for each new commit that arrives from a remote. It then launches an arbitrary script to "build" that commit, whatever that might mean.
16:26:57
Shinmera
In its current form it can't distinguish between different kinds of actions you might perform, and it can't do stuff like providing a build fragment to the user.
16:28:22
drmeister
It would use the type specifier to either generate code to check the tag (fixnum, single-float, character, cons) or load the header-value for the object and compare it to what it gets out of the class symbol-> header-value(-range) map based on the type specifier argument (a symbol) of the typeq-instruction.
16:28:52
Shinmera
I want to fix those problems, as well as others, in the second iteration. I don't know when motivation to actually write it will strike me though
16:33:25
scymtym
Shinmera: what autobuild does sounds complementary to what we do. an option to replace the jenkins part with something else has been on my todo-list for some time. but that's probably off-topic and i have to leave now anyway. maybe we can talk about that later
16:36:05
drmeister
Bike: Hmm we will need an additional map for those. Those are represented by calls like: add_single_typeq_test<core::SimpleMDArray_byte8_t_O>();
16:37:45
drmeister
When you make an array of (simple-array ext:byte8) it gets translated to core::SimpleMDArray_byte8_t_O
16:37:51
Bike
it's not. i was just looking at make-array yesterday, it just passes the element type to a c++ function.
16:38:48
drmeister
Here: https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/lsp/predlib.lsp#L1363
16:40:41
drmeister
Would you be able to work from that? The map needs to go in the opposite direction for typeq's needs
16:41:28
drmeister
Either that or we change the way things are defined in predlib.lsp so that we specify the C++ class name <-> CL type specifier once and it builds both what predlib needs and what we need.
16:43:06
Bike
there should probably be a variable with information about array element types for optimization purposes
16:43:28
Bike
probably the uaet, the default initial value, the C++ simple vector class, and the C++ mdarray class
16:45:23
Bike
i think i can dig out the default initial elements, though, i'm not totally sure how to organize it overall
16:45:38
Bike
i'd like to inline make-array (this is a separate concern from typeq, ofc, but i mean, kinda related)
17:00:24
drmeister
Currently, the only place anything like (simple-vector ext:byte8) -> core::SimpleVector_byte8_t_O is expressed in Clasp is here:
17:01:35
drmeister
And I knew that had a bad code smell to it when I wrote it - but "when needs must the devil drives".
17:01:57
Bike
in the most common case of (make-array ... :element-type some-constant) we could just call StrWNs_O::make or whatever directly
17:02:53
drmeister
Is there a way in Common Lisp to reduce a vector or array type specifier to a canonical type specifier?
17:03:57
drmeister
Meaning we make the map of type-specifier -> header-value(-range) use EQUAL and throw the canonical type specifier for every array type in as a key
17:05:04
drmeister
Then the typeq-instruction takes whatever type specifier it is given, canonicalizes it, looks it up in the map and if it's there generates code and if not it generates a call to typep.
17:06:52
drmeister
There is a call generated like this: add_range_typeq_test<core::SimpleMDArray_O,core::SimpleMDArray_int32_t_O>();
17:11:55
drmeister
The values above are the stamp values, the header value for these are ((stamp<<4)|1)
18:15:34
Bike
something is conceptually weird with typeq and i'm not sure how to fix it. type inference wants typeq instructions that deal with a sublattice of types. but typeq also means actual checks of actual user-provided types, which should not be a sublattice, and which should be efficient code
18:44:56
Bike
i guess (if (typeq foo bar) baz bad) could be compiled as (if ([check] foo bar) (progn (the bar foo) baz) (progn (the (not bar) foo) bad))
19:21:58
drmeister
It maps class symbols that correspond to types to either a fixnum (header-value) or a (CONS header-value-min header-value-max)
19:22:49
drmeister
Bike: Do you have any quick fixes for the looong (40 min) compilation of that one form in ironclad?
19:23:16
drmeister
I need to build a new docker image for the nglview stuff and a quick fix there would save me 40 min.
19:24:37
Bike
uhm, depends no your parameters for quick fix. i could add a policy to turn off dynamic extent analysis specifically that you could use
19:24:53
drmeister
It turns out that the Python version of nglview was old and the Javascript code that the Common Lisp code was using would use the old Javascript if I ran the Python version first and then use the new Javascript code if I ran the Common Lisp version first.
19:25:57
Bike
or you could just (declaim (optimize (clasp-cleavir::analyze-flow nil))) and then it wouldn't do any type inference or dx or anything.
19:27:51
Bike
the less principled thing to do would just be to comment out dx from my-hir-transformations
19:35:58
drmeister
It will take a little while to shake out the bugs of the typeq-instruction code generation.
22:58:51
Bike
one input. if the input is of the instruction's type, go to one branch. otherwise go to the other
0:00:56
Bike
well, the upshot for us is that profiling is nice. though we have a lot more fires to put out before we can care about this, probably
0:33:21
drmeister
We can get rid of fixnump-instruction, consp-instruction, characterp-instruction and replace them with typeq-instruction.
0:34:41
Bike
well, what i want to do is keep the existing typeq-instruction, and have eliminate-typeq turn it into mir instructions, one of which is a primitive-typeq-instruction
0:35:04
Bike
because like, we want (typeq x (or cons null)) to be like (or (primitive-typeq x cons) (eq x nil)), not a typep call
0:38:28
Bike
might be ok to organize it as having fixnump, consp, characterp, single-float-p, and then have a uh, "headerq" that does the bit starting with let* ((header-check-br
0:40:54
Bike
eliminate-typeq has its type, it checks type-header-value-map, if the type is present it generates a headerq-instruction using the header-value-min-max in the table
0:42:50
Bike
really, for a constant type the only reason we need to call typep is if the type is undefined or it's a class(name), and the latter we can eliminate later
0:45:49
Bike
anyway, that snippet and the variables should be enough for me to figure out the rest myself
0:52:41
drmeister
I dunno - I haven't used it yet. It's part of this function: http://llvm.org/doxygen/classllvm_1_1IRBuilder.html#a3393497feaca1880ab3168ee3db1d7a4
0:53:24
drmeister
I tried to make sure the comparisons are always in the sense where branch-weights mean the same thing.
1:00:10
Bike
so, on this topic, in each running thread there's a top-of-the-shadow-stack variable that holds information about the currently executing lisp function, right?
1:03:10
drmeister
Each mp:process points to a thread-local structure that contains a pointer to the top of the invocation history stack.
1:05:56
drmeister
There will always be something. It's so painful to debug without it - I can't tolerate it.
1:06:22
Bike
statistical profiling is done by just checking where running code is every millisecond or whatever. so, we can just start up a profiler thread that checks the current function in the profiled thread
1:15:18
drmeister
Every Clasp function is like a C++ function. The big issue are anonymous lambda's - they are everywhere.
1:16:04
drmeister
But if you (defun foo (...) ...) in package BAR the function name is something like FN_BAR::FOO
1:24:00
drmeister
Maybe it's used to fingerprint the sample? joker-eph in #llvm told me that they get around the limitation by spawning the compiler in another thread.
1:30:28
drmeister
I'm really annoyed. I installed an older version of nglview according to the nglview instructions and then started developing code to mimic it. I got it working - but only when I ran the python version of the nglview widget first - that loaded the old Javascript code into the browser.
1:31:49
drmeister
So things took longer to get working because more than half the time I was running the Common Lisp code without running the Python version first and it loaded a newer version of the Javascript code that it wasn't designed to talk to properly.
1:35:27
drmeister
Annoyance #2 is now I am rebuilding the docker image with (hopefully) the latest version of everything and then I have to change the Common lisp code to bring it up to talk properly to the latest version of the Javascript.