freenode/#clasp - IRC Chatlog
Search
10:59:11
Shinmera
beach: A while back you said you'd see if you came across any suitable topics for my bachelor's thesis. Could you send me a mail outlining the ideas you've had so far? I'd like to notify my professor sometime soon to see what he thinks of the things I've managed to think of so far.
12:49:45
Shinmera
Bachelors thesis usually give you 6 months to work on, with an expected "actual work time" of about 2 months tops.
12:51:08
beach
The project is then: Implement path replication according to that paper. Analyze the result with respect to cost and performance gain.
12:52:40
beach
Shinmera: Other possibility: Design and implement a complete flow-control analysis for Common Lisp, in the presence of nested functions and threads. This work would extend what is discussed in the literature because of the Common Lisp specific features and because of threads.
12:53:33
beach
Shinmera: Third possibility: Use local graph rewriting for (partial) inlining of local functions. Also a HIR project.
12:54:08
beach
Shinmera: I can come up with more, but I might need a few days. When do you plan to notify your professor?
12:54:40
beach
Shinmera: Also, I could come up with non-HIR projects, non-Cleavir projects, or non-SICL projects, if you prefer.
12:55:04
Shinmera
It'd be great if you could send me a mail with a short summary for each idea. Perhaps outlining what to do and how it connects to prior work.
12:55:27
Shinmera
Since I think a large part of what he wants me to do for the bachelor's thesis is also to analyse prior work and compare.
12:56:56
Shinmera
I have two ideas of my own, but they have pretty much nothing to do with cleavir or clasp.
12:58:40
Shinmera
One of them is about a system for the modular combination of opengl shader sources. The prior work would be to analyse how other engines and projects handle that, and the new idea is to use a parser and semantic analyser to merge code together.
13:00:14
Shinmera
The other is about cross-distribution deployment of binaries. Prior work being stuff like AppImage, Linux Standard Base, distribution packaging. The "new" approach being my ld-preload.so shenanigans.
13:00:52
Shinmera
I already have a proof of concept for the OpenGL thing that I know works to an extent.
13:02:26
beach
Both sound fine as far as I can tell. But I don't know much about prior art, so don't trust my judgment.
13:22:25
beach
Shinmera: Sent you the first one. Please check it and see whether this kind of summary is sufficient.
14:04:04
beach
I have a hunch that almost all compiler-optimization techniques that require SSA (static single assignment) in fact do not use the fact that SSA is static single assignment, and that they instead rely on other properties of SSA.
14:04:42
beach
The project would be to dig up a variety of such published techniques, read the papers and determine whether they actually use the SSA property of SSA.
14:06:53
beach
OK, you have three proposals using HIR and Cleavir. I'll write up more as I think about them.
14:07:31
Shinmera
By the way, I was pleasantly surprised that this introductory compiler design class I took did a lot more than just parsing and basic compilation.
14:07:52
Shinmera
It also went into static analysis and the kinds of optimisations you can do with that, though we didn't go too far with it.
14:08:43
beach
When I was in charge here, I split up the typical course into one for parsing and one for compiler optimization.
14:09:09
beach
The first one is useful for things other than compilers, like analyzing data and for DSLs.
14:10:19
Shinmera
Right. The course just introduced the basic theory behind top down and bottom up parsers and how to compute the tables for them.
14:10:51
beach
Parsing used to be all about LALR and perhaps some recursive descent. Now, Earley and GLR are practical and there are new techniques such as combinator parsing.
14:10:53
Shinmera
We didn't have to write parsers by hand or anything, which was actually a bit disappointing to me.
14:19:20
beach
Courses on algorithms and data structures have the same problem: use a real language and the message is drowned in idiosyncratic aspects of it. Use an artificial language and you don't get enough power to do modern stuff.
14:20:58
Shinmera
I think the way our class did it was fine. The lecture was presented with just explanations of the structures and algorithms in abstract, but the book that came along with it (it's huge) has code samples as well.
15:14:39
drmeister
The nglview thing is annoying - it appears there's two copies of nglview in the docker image and that the Common Lisp code uses version #2 if I start it first and version #1 (that Python uses) if I start the Python notebook first. Version #1 works right now, version #2 still crashes - Grrrr.
15:16:08
Shinmera
My JS debugging skills are: -looking at stack traces -cursing a lot -changing things here and there hoping it works -getting annoyed trying to reproduce things for the 100th time
15:18:23
drmeister
I have some basic questions. When I make changes to the source code on the right side, upper half, middle of three...
15:19:06
drmeister
Yesterday it seemed like sometimes when I made changes and then hit Command-s (save?) - the changes seemed to take.
15:19:37
drmeister
I could put console.log("something") and the "something" would print to the console when I thought that code should execute.
15:20:17
drmeister
Am I supposed to be able to edit the code there and the browser will recompile it?
15:21:18
drmeister
I can edit the source code on the host beforehand - that always works, but it would be really convenient if I could edit the code in that window for debugging.
15:21:45
Shinmera
JS is, at its basis, just a sequential language that executes top level statements like in a script. When you save an already loaded file, the browser probably does some magic to try and figure out which things are "static definitions" that it can update and which are not.
15:22:03
Shinmera
This means that stuff it deems dynamic, or things that really are dynamically constructed at runtime, won't be affected by your change.
15:25:51
Shinmera
This is pretty ingenious. I don't really know how it knows what it can update or how.
15:35:13
drmeister
Searching on the web suggests it is a widespread issue that the code in the development console doesn't always compile when changes are made to it.
15:37:29
drmeister
The next question - is there a way to figure out where the extension is coming from on the host system?
15:39:56
drmeister
I thought I had two versions of this nglview-js-widgets javascript code in the docker image. I just dug out the version#2, put it in the widget-dev directory and mounted it back into the docker image. Changes to the index.js file in the widget-dev/nglview-js-widgets/index.js are not showing up in the Javascript console - Grrrrr.
15:41:26
Shinmera
If you have the network tab open while the page loads it should list all sources, how long they load, and so forth
15:42:51
Shinmera
all the browser will know about though is the address. to which physical file that might map it cannot possibly know
15:43:09
Shinmera
there might not even be one or the server might do whatever it wants to deliver the content.
15:49:28
scymtym
drmeister: i managed to generate a group of jenkins jobs that builds llvm 4.0 and clasp, but not cando, yet (based on (almost) this model: https://techfak.de/~jmoringe/clasp-using-vanilla-llvm-nightly.png ). this can be used to build inside or outside of docker or generally for different purposes. can you explain the goal in more detail? for example, should the container contain the build environment or should its size
15:50:56
drmeister
We don't need to build llvm4.0 and clang - they can be obtained using the package manager.
15:51:38
scymtym
Bike: thanks, i though erc handled that. the remainder was just "… or should its size be minimal?"
15:52:34
drmeister
What I'm looking for is a jenkins server that can run certain docker-compose commands as if they were run from the clasp/ root directory.
15:53:36
drmeister
And the configuration files to set up that jenkins server on an Amazon Web services machine.
15:54:32
drmeister
The idea is a push button server that can spin up in a few minutes, provide jenkins that we can connect in to with the browser and allow the user to select a job that will build (1) a cando docker image (2) run the static analyzer (3) a clasp docker image.
15:54:45
scymtym
drmeister: there's a funny reason for that. the machine on which i have root (and docker) is not beefy enough. on the beefy machine, i don't have root (and no docker) and it runs ubuntu xenial. so i had to build llvm there. i made multiple variants, though: using-vanilla-llvm, using-externals-clasp, using-system-provided-packages
15:54:52
drmeister
Actually, there are two static analyzer jobs - one for clasp and another for cando.
15:57:32
scymtym
are you planning to set up a new jenkins server each time something has to be built or should it (the jenkins "master", that is) persist?
15:58:31
drmeister
So imagine a script that you run on a fresh ubuntu that does the following. (1) git clone github.com/drmeister/clasp.git $HOME/Dev/clasp (2) git clone github.com/drmeister/clasp.git $HOME/Dev/cando (3) git clone github.com/drmeister/cando.git $HOME/Dev/cando/extensions/cando (4) installs jenkins (5) installs jenkins configuration files that run makefile? targets in $HOME/Dev/clasp/makefile and
16:01:03
drmeister
Is it? I don't really know jenkins - I thought it was a web server that started jobs
16:02:29
scymtym
no, that's all true. the difference is that you normally don't install and configure jenkins for one build and then tear it down again
16:02:56
drmeister
This is a jenkins server that builds cando - but it is fatally flawed because jenkins runs in a Docker container and tries to run Docker containers within it: https://hub.docker.com/r/drmeister/cando-ci/
16:03:46
scymtym
since jenkins is often used for continuous integration, the same instance usually runs many jobs so that it can track and archive things over time
16:03:58
drmeister
Ok - I ask for that because we can use Amazon Web Services spot instances - which are very inexpensive (1.5cents/hour) - but they can be pulled at any time.
16:04:21
drmeister
So rapid set up and the expectation that they could be taken away at any moment is important.
16:05:23
scymtym
i see. you can also do that. in fact, automating (4) and (5) is one of the things we build tools for
16:07:46
drmeister
I'm not wed to jenkins at all - I don't even know the program. If something else would be better suited for this I would be happy to use that as well.
16:09:05
drmeister
I currently have a spot instance that I've been running for weeks to build docker images and run the static analyzer. It could disappear at any moment.
16:10:20
drmeister
Those template functions specify the classes and class-ranges (the actual header values are elsewhere in the file) for TYPEQ
16:10:26
Shinmera
Autobuild as it stands right now is not really suited for generating deliverables.
16:11:22
scymtym
is was going to ask what the motivation for using jenkins in this scenario was? jenkins' strengths, as i see it, are mostly not applicable to the setup-do-one-build-tear-down use-case
16:11:30
drmeister
Maybe I don't even want jenkins. Maybe I just want a bunch of scripts that provision a machine and do one build job?
16:12:35
drmeister
I suggested jenkins because that is what my friend set up for me - but it has that flaw that neither of us knew about until I spent a couple of days trying to get it to work.
16:14:28
drmeister
Dammit - there are 11 (eleven!) copies of that nglview-js-widgets directories in the Docker image that runs the cando jupyter notebook
16:14:43
scymtym
i think jenkins could be really useful for other things (assuming autobuild does not already cover that area), it just doesn't seem optimal for what you want to do in this case
16:16:39
drmeister
Bike: each template function will create a class symbol -> header-value or header-value range entry in a map.
16:26:31
Shinmera
scymtym: Autobuild maintains a git repository clone and creates a clean clone of that for each new commit that arrives from a remote. It then launches an arbitrary script to "build" that commit, whatever that might mean.
16:26:57
Shinmera
In its current form it can't distinguish between different kinds of actions you might perform, and it can't do stuff like providing a build fragment to the user.
16:28:22
drmeister
It would use the type specifier to either generate code to check the tag (fixnum, single-float, character, cons) or load the header-value for the object and compare it to what it gets out of the class symbol-> header-value(-range) map based on the type specifier argument (a symbol) of the typeq-instruction.
16:28:52
Shinmera
I want to fix those problems, as well as others, in the second iteration. I don't know when motivation to actually write it will strike me though
16:33:25
scymtym
Shinmera: what autobuild does sounds complementary to what we do. an option to replace the jenkins part with something else has been on my todo-list for some time. but that's probably off-topic and i have to leave now anyway. maybe we can talk about that later
16:36:05
drmeister
Bike: Hmm we will need an additional map for those. Those are represented by calls like: add_single_typeq_test<core::SimpleMDArray_byte8_t_O>();
16:37:45
drmeister
When you make an array of (simple-array ext:byte8) it gets translated to core::SimpleMDArray_byte8_t_O
16:37:51
Bike
it's not. i was just looking at make-array yesterday, it just passes the element type to a c++ function.
16:38:48
drmeister
Here: https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/lsp/predlib.lsp#L1363
16:40:41
drmeister
Would you be able to work from that? The map needs to go in the opposite direction for typeq's needs
16:41:28
drmeister
Either that or we change the way things are defined in predlib.lsp so that we specify the C++ class name <-> CL type specifier once and it builds both what predlib needs and what we need.
16:43:06
Bike
there should probably be a variable with information about array element types for optimization purposes
16:43:28
Bike
probably the uaet, the default initial value, the C++ simple vector class, and the C++ mdarray class
16:45:23
Bike
i think i can dig out the default initial elements, though, i'm not totally sure how to organize it overall
16:45:38
Bike
i'd like to inline make-array (this is a separate concern from typeq, ofc, but i mean, kinda related)
17:00:24
drmeister
Currently, the only place anything like (simple-vector ext:byte8) -> core::SimpleVector_byte8_t_O is expressed in Clasp is here:
17:01:35
drmeister
And I knew that had a bad code smell to it when I wrote it - but "when needs must the devil drives".
17:01:57
Bike
in the most common case of (make-array ... :element-type some-constant) we could just call StrWNs_O::make or whatever directly
17:02:53
drmeister
Is there a way in Common Lisp to reduce a vector or array type specifier to a canonical type specifier?
17:03:57
drmeister
Meaning we make the map of type-specifier -> header-value(-range) use EQUAL and throw the canonical type specifier for every array type in as a key
17:05:04
drmeister
Then the typeq-instruction takes whatever type specifier it is given, canonicalizes it, looks it up in the map and if it's there generates code and if not it generates a call to typep.
17:06:52
drmeister
There is a call generated like this: add_range_typeq_test<core::SimpleMDArray_O,core::SimpleMDArray_int32_t_O>();
17:11:55
drmeister
The values above are the stamp values, the header value for these are ((stamp<<4)|1)