freenode/#clasp - IRC Chatlog
Search
15:14:19
Bike
using the compiler while using the compiler to rebuild the compiler turns out to be difficult
15:15:14
beach
That's why I have been thinking about using first-class global environments for that.
15:15:28
Bike
if it hits a class definition, it redefines the class, which undefines all the readers, then calls class-slots, which updates the constructors, which calls the compiler, which tries to use a reader
15:15:40
beach
One environment would contain the working compiler that could be used to compile the compiler in a different environment.
15:16:06
Bike
drmeister: she's been trying, no luck. this conda "environments" thing seems a little weird
15:17:50
drmeister
I'll have to separate out ironclad - it compiles really, really slowly when building the docker environment but builds pretty fast on my laptop.
15:17:56
hana
also in the readme for rdkit ipython, it says that it was written for python3 so im not sure if it would work with the python2 notebook?
15:18:33
drmeister
Hana - Is it possible to install the python3 notebook stuff alongside python2 notebook stuff?
15:19:29
hana
what do you mean by alongside? I think it should be possible to have a python3 notebook, if I recall correctly, I saw some in youtube tutorials
15:19:45
beach
Bike: As I recall, drmeister has made some noise in the past about wanting first-class global environments, but I think it was never worked on any further.
15:20:32
Bike
maybe i'll do it. currently when you build cclasp it just... redefines everything as it goes. all i foresee is more problems
15:22:18
Bike
"if it hits a class definition, it redefines the class, which undefines all the readers, then calls class-slots, which updates the constructors, which calls the compiler, which tries to use a reader"
15:22:25
drmeister
I (1) set up this in the wscript.config --> INSTALL_PATH_PREFIX = '/Users/meister/local/clasp/'
15:25:01
beach
Seriously, I think first-class global environments is a great solution to that problem. If drmeister is serious about wanting them in Clasp, maybe that should be worked on first.
15:26:38
Bike
i think i would run into the same problem, because it would still redefine the class and things would proceed in the same order.
15:30:50
Bike
i'll give it a shot. i'm not sure how it will work with some things, most obviously classes
15:32:01
beach
I can walk you through the issues with first-class global environments so that you can decide what design decisions to make.
15:32:05
Bike
actually. yeah. when you hit an eval-when and you're supposed to evaluate something /in the compiler/, does that affect the compiler's environment or the environment in which you are compiling?
15:33:02
beach
It affects the "compilation environment". The problem is that the Common Lisp HyperSpec gives to many options for an implementation to make that environment different or the same as some others.
15:33:48
beach
Can I assume that you read the page in the Common Lisp HyperSpec about different environments?
15:35:01
beach
Also, with first-class global environments you get an excellent testing platform. You can have multiple versions of the compiler and whatnot, and you can compare them without re-building your system.
15:35:27
Bike
yeah, more than once i've recompiled some particular function in the compiler, and it was wrong, and then i can't fix it because the compiler is broken
15:35:35
drmeister
Hana: Try building one that sets up a web server first - there are lots of tutorials.
15:36:40
drmeister
Hana: You can use my Dockerfiles as reference and as a starting point - hang on...
15:37:24
Bike
that is sort of what's happening here, really. the compiler hits a defclass, so it redefines the class. while it does that, the compiler is in an inconsistent state.
15:40:19
beach
And I had a long day. I fixed lunch for my favorite co-author, and then we worked in the afternoon.
15:41:13
hana
drmeister: so the goal is to set up a docker image that contains jupyter notebooks for python 2, python 3 and rdkit?
15:42:53
beach
One of the uses for first-class global environments that I imagined after phoe's suggestion was to have an environment in which the compiler functions have been converted to ordinary functions, so as to facilitate bootstrapping the real compiler in a different environment.
15:43:50
drmeister
hana: Yes - except we don't need python 2. It's a docker image that just runs rdkit. We want to see what it does and how it works.
15:44:50
drmeister
hana: It's cleaner to set up jupyter notebook running in a Docker container. You can keep rebuilding it and throwing it away if it doesn't work and you don't pollute your system with bad configuration info.
15:45:56
drmeister
You keep manipulating a file called a "Dockerfile" and rebuilding the docker image until you get it right.
15:47:48
drmeister
Bike: Figure out how first class environments work - then I can help you implement them within Clasp.
15:50:24
drmeister
hana: You will need to install 'docker-compose' on your system - it's an application that runs on top of docker.
16:10:12
drmeister
hana: I've created a Dockerfile that builds a docker image that just provides the python 2.7 jupyter notebook with nglview. You can try massaging it to install python 3 and rdkit and see if you can get that working.
16:13:51
drmeister
hana: An important thing to learn about docker is how to clean up broken images with: docker images; docker rmi xxxxx
16:25:19
Bike
anyway, seems like it'll be an undertaking, so i should do something else for now. inlining svref i think would be doable. basically just have to know the offset of the stretchy vector
16:43:13
Bike
seems like it would be like car_offset and cdr_offset. dunno why those aren't just offsetof, but regardless it's probably not as smooth since template_SimpleVector is just a template
16:43:33
beach
Bike: Have you and drmeister set specific goals for what you might try to accomplish during the year, or are you free to work on anything that improves Clasp?
16:45:16
drmeister
Let's talk about goals in light of the new development that you will be staying until next summer.
16:47:50
Bike
i mean, i have specific targets to work on, but if there's a delay in one i have other ones
17:06:08
derrida
Hey all! Looks like you guys have made a lot of progress since I last poked around :) Trying to build from testing on OSX and I'm running into a linker problem near the very end of building clasp. https://gist.github.com/mgs/ca213570f44dc64a588bd6b8d6fb3b4a
17:07:06
derrida
ld: could not parse object file src/gctools/gc_interface.cc.3.o: 'Invalid record (Producer: 'LLVM4.0.0svn' Reader: 'LLVM 3.9.1')', using libLTO version 'LLVM version 3.9.1' file 'src/gctools/gc_interface.cc.3.o' for architecture x86_64
17:07:44
Bike
says it's using /opt/local/bin/ld instead of externals clasp, i wonder if that is problematic
17:10:47
Bike
i mean, it looks like a version mismatch, but i'm not sure how 3.9.1 is getting involved
17:13:29
derrida
I didn't catch that and I thought I had prevented macports/brew conflicts, I guess not. :[ That's fixable anyway.
17:27:08
drmeister
If you pull the master branch of widget-dev there are some new files related to building docker images
17:27:47
drmeister
It's a series of instructions to docker to build a thing called a docker 'image'.
17:28:12
drmeister
At the top it starts with a fresh ubuntu image and each successive command layers changes on top of it.
17:29:05
drmeister
You build everything using a separate docker related program called 'docker-compose' - which you need to install on your system.
17:30:12
drmeister
I'm not exactly sure what the purpose of docker-compose is - I think it sort of drives docker to do things using another configuration file called 'widget-dev/docker-compose.yml'
17:32:04
drmeister
docker-compose will read the docker-compose.yml file, and use the pythonnb entry to load the widget-dev/dockerfiles/pythonnb/Dockerfile and use that to build the image.
17:33:00
drmeister
Then bring up your browser and you will have the familiar jupyter notebook interface.
17:39:18
drmeister
Error response from daemon: conflict: unable to remove repository reference "pythonnb" (must force)
17:43:13
drmeister
hana: I'm going to duplicate everything so that you have a copy you can hack to get rbkit install
17:51:36
drmeister
Ok, there is a widget-dev/dockerfiles/pythonrdkit/Dockerfile and an entry in the docker-compose.yml file
17:52:07
drmeister
You can edit widget-dev/dockerfiles/pythonrdkit/Dockerfile and run docker-compose build pythonrdkit
17:54:15
drmeister
Sometimes it gets screwed up. That's when you: docker rmi <xxxx> and start from scratch.
18:03:10
drmeister
Here I unpack the contents of the alist and put things into Common Lisp special variables - but that isn't necessary - the a-list returned by that function is the ultimate authority.
18:03:13
drmeister
https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/cmp/cmpintrinsics.lsp#L123
18:04:17
drmeister
Right - there are two kinds of arrays in Clasp. The stuff in array.h is pretty new. Vec0<xxxx> is a C++ template class for a vector that doesn't need to be initialized at runtime.
18:04:42
drmeister
You can't expose Vec0<xxxx> to CL directly - they are contained within other objects.
18:07:51
Bike
well, all i'm immediately concerned with is the _Data offset. if those aren't relevant, so it goes
18:16:55
hana
so I just add on install commands for rdkit in the dockerfile under pythonrdkit, then docker-compose build pythonrdkit, docker run -it -p 8888:8888 pythonnb and test to see if it installed correctly? if not, remove the image and start again?
18:17:46
drmeister
The docker-compose command will generate errors until the pythonrdkit/Dockerfile is perfect.
18:18:13
drmeister
Sometimes it will stop on an error and other times it will just proceed. You need to look through the output for errors.
18:23:13
drmeister
Bike: I added this: ENTRY(list, "SIMPLE-VECTOR[0]-OFFSET",make_fixnum(offsetof(SimpleVector_O,_Data)));
18:24:02
drmeister
This isn't specific to SimpleVector_O - it will work with any of the simple vectors.
18:31:41
Bike
"ah, i should see what the error message is for out of bounds access" (aref #() 1) => "cc_shutdown_gcroots_in_module"
18:34:10
drmeister
The simple-vector offsets are a bit more complicated - because of tricks I had to play to get the efficient layout - working...
18:35:27
drmeister
Here's the result: (:SIMPLE-VECTOR._LENGTH-OFFSET . 8) (:SIMPLE-VECTOR._DATA-OFFSET . 8)
18:36:23
drmeister
It's not a big deal - I used zero length arrays to get the right layout while still being able to use inheritance
18:39:12
drmeister
I don't want to hard-code it - because if I change it in the future I want it to generate a compiler error
18:45:53
drmeister
General_O derives from T_O General_O is the first class that guarantees a vtable.
18:54:26
drmeister
Conses need to be very efficient - there seemed no point in burdening them with a vtable.
18:58:39
drmeister
If you take a tagged pointer for a simple-vector and add that offset minus the tag (in :GENERAL-TAG in that same alist) the resulting address is the address of the first element.
19:00:39
drmeister
Hmm, that's not a keyword symbol - I defined that before I switched to keyword symbols. It's probably 'core:tsp
19:10:54
Bike
i don't know. aref should probably be lowered to mir, but there's no complex indexing in mir. guess i could add it...
19:13:34
drmeister
This is what I do currently with the memref2 instruction: https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/cleavir/translate.lisp#L633
19:14:45
Bike
i mean, here we don't have a constant offset, we want to take the index and add the _data offset to it (and that's before element types)
19:15:37
drmeister
Can you remind me what arguments a memref2 instruction takes as inputs? There is no untagging going on in that code.
19:17:43
drmeister
I see it here: https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/cleavir/hir-to-mir.lisp#L16
19:19:35
drmeister
So the car-instruction generates an memref2 offset that is (- cmp:+cons-car-offset+ cmp:+cons-tag+)
19:20:53
drmeister
There is only the aref-instruction for reading into arrays - correct? What inputs does that take now?
19:23:14
drmeister
So what if convert the aref-instruction into a memref2 instruction where the offset is the offset of the index minus the tag?
19:30:39
Bike
i think it makes sense to have memref instructions for different kinds of machine memory loads. they're basically all indirect though. i think.
19:36:46
Bike
an array read would i guess be... mov (%rarray, %rindex, 8), %rdest? assembler syntax...
19:37:32
drmeister
Yeah - let's sort it out now - checking that the generated assembly code is sane.
19:37:48
drmeister
I used ptrtoint and integer arithmetic because I just wanted to get something to work.
19:38:12
drmeister
Once we sort out the GEP syntax I will search and replace ptrtoint with the GEP syntax.
19:38:36
drmeister
stassats: Had a suggestion a couple of weeks ago. I think it needs to be reduced to an i8* GEP and then cast to whatever type we need.