freenode/#clasp - IRC Chatlog
Search
16:43:13
Bike
seems like it would be like car_offset and cdr_offset. dunno why those aren't just offsetof, but regardless it's probably not as smooth since template_SimpleVector is just a template
16:43:33
beach
Bike: Have you and drmeister set specific goals for what you might try to accomplish during the year, or are you free to work on anything that improves Clasp?
16:45:16
drmeister
Let's talk about goals in light of the new development that you will be staying until next summer.
16:47:50
Bike
i mean, i have specific targets to work on, but if there's a delay in one i have other ones
17:06:08
derrida
Hey all! Looks like you guys have made a lot of progress since I last poked around :) Trying to build from testing on OSX and I'm running into a linker problem near the very end of building clasp. https://gist.github.com/mgs/ca213570f44dc64a588bd6b8d6fb3b4a
17:07:06
derrida
ld: could not parse object file src/gctools/gc_interface.cc.3.o: 'Invalid record (Producer: 'LLVM4.0.0svn' Reader: 'LLVM 3.9.1')', using libLTO version 'LLVM version 3.9.1' file 'src/gctools/gc_interface.cc.3.o' for architecture x86_64
17:07:44
Bike
says it's using /opt/local/bin/ld instead of externals clasp, i wonder if that is problematic
17:10:47
Bike
i mean, it looks like a version mismatch, but i'm not sure how 3.9.1 is getting involved
17:13:29
derrida
I didn't catch that and I thought I had prevented macports/brew conflicts, I guess not. :[ That's fixable anyway.
17:27:08
drmeister
If you pull the master branch of widget-dev there are some new files related to building docker images
17:27:47
drmeister
It's a series of instructions to docker to build a thing called a docker 'image'.
17:28:12
drmeister
At the top it starts with a fresh ubuntu image and each successive command layers changes on top of it.
17:29:05
drmeister
You build everything using a separate docker related program called 'docker-compose' - which you need to install on your system.
17:30:12
drmeister
I'm not exactly sure what the purpose of docker-compose is - I think it sort of drives docker to do things using another configuration file called 'widget-dev/docker-compose.yml'
17:32:04
drmeister
docker-compose will read the docker-compose.yml file, and use the pythonnb entry to load the widget-dev/dockerfiles/pythonnb/Dockerfile and use that to build the image.
17:33:00
drmeister
Then bring up your browser and you will have the familiar jupyter notebook interface.
17:39:18
drmeister
Error response from daemon: conflict: unable to remove repository reference "pythonnb" (must force)
17:43:13
drmeister
hana: I'm going to duplicate everything so that you have a copy you can hack to get rbkit install
17:51:36
drmeister
Ok, there is a widget-dev/dockerfiles/pythonrdkit/Dockerfile and an entry in the docker-compose.yml file
17:52:07
drmeister
You can edit widget-dev/dockerfiles/pythonrdkit/Dockerfile and run docker-compose build pythonrdkit
17:54:15
drmeister
Sometimes it gets screwed up. That's when you: docker rmi <xxxx> and start from scratch.
18:03:10
drmeister
Here I unpack the contents of the alist and put things into Common Lisp special variables - but that isn't necessary - the a-list returned by that function is the ultimate authority.
18:03:13
drmeister
https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/cmp/cmpintrinsics.lsp#L123
18:04:17
drmeister
Right - there are two kinds of arrays in Clasp. The stuff in array.h is pretty new. Vec0<xxxx> is a C++ template class for a vector that doesn't need to be initialized at runtime.
18:04:42
drmeister
You can't expose Vec0<xxxx> to CL directly - they are contained within other objects.
18:07:51
Bike
well, all i'm immediately concerned with is the _Data offset. if those aren't relevant, so it goes
18:16:55
hana
so I just add on install commands for rdkit in the dockerfile under pythonrdkit, then docker-compose build pythonrdkit, docker run -it -p 8888:8888 pythonnb and test to see if it installed correctly? if not, remove the image and start again?
18:17:46
drmeister
The docker-compose command will generate errors until the pythonrdkit/Dockerfile is perfect.
18:18:13
drmeister
Sometimes it will stop on an error and other times it will just proceed. You need to look through the output for errors.
18:23:13
drmeister
Bike: I added this: ENTRY(list, "SIMPLE-VECTOR[0]-OFFSET",make_fixnum(offsetof(SimpleVector_O,_Data)));
18:24:02
drmeister
This isn't specific to SimpleVector_O - it will work with any of the simple vectors.
18:31:41
Bike
"ah, i should see what the error message is for out of bounds access" (aref #() 1) => "cc_shutdown_gcroots_in_module"
18:34:10
drmeister
The simple-vector offsets are a bit more complicated - because of tricks I had to play to get the efficient layout - working...
18:35:27
drmeister
Here's the result: (:SIMPLE-VECTOR._LENGTH-OFFSET . 8) (:SIMPLE-VECTOR._DATA-OFFSET . 8)
18:36:23
drmeister
It's not a big deal - I used zero length arrays to get the right layout while still being able to use inheritance
18:39:12
drmeister
I don't want to hard-code it - because if I change it in the future I want it to generate a compiler error
18:45:53
drmeister
General_O derives from T_O General_O is the first class that guarantees a vtable.
18:54:26
drmeister
Conses need to be very efficient - there seemed no point in burdening them with a vtable.
18:58:39
drmeister
If you take a tagged pointer for a simple-vector and add that offset minus the tag (in :GENERAL-TAG in that same alist) the resulting address is the address of the first element.
19:00:39
drmeister
Hmm, that's not a keyword symbol - I defined that before I switched to keyword symbols. It's probably 'core:tsp
19:10:54
Bike
i don't know. aref should probably be lowered to mir, but there's no complex indexing in mir. guess i could add it...
19:13:34
drmeister
This is what I do currently with the memref2 instruction: https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/cleavir/translate.lisp#L633
19:14:45
Bike
i mean, here we don't have a constant offset, we want to take the index and add the _data offset to it (and that's before element types)
19:15:37
drmeister
Can you remind me what arguments a memref2 instruction takes as inputs? There is no untagging going on in that code.
19:17:43
drmeister
I see it here: https://github.com/drmeister/clasp/blob/dev/src/lisp/kernel/cleavir/hir-to-mir.lisp#L16
19:19:35
drmeister
So the car-instruction generates an memref2 offset that is (- cmp:+cons-car-offset+ cmp:+cons-tag+)
19:20:53
drmeister
There is only the aref-instruction for reading into arrays - correct? What inputs does that take now?
19:23:14
drmeister
So what if convert the aref-instruction into a memref2 instruction where the offset is the offset of the index minus the tag?
19:30:39
Bike
i think it makes sense to have memref instructions for different kinds of machine memory loads. they're basically all indirect though. i think.
19:36:46
Bike
an array read would i guess be... mov (%rarray, %rindex, 8), %rdest? assembler syntax...
19:37:32
drmeister
Yeah - let's sort it out now - checking that the generated assembly code is sane.
19:37:48
drmeister
I used ptrtoint and integer arithmetic because I just wanted to get something to work.
19:38:12
drmeister
Once we sort out the GEP syntax I will search and replace ptrtoint with the GEP syntax.
19:38:36
drmeister
stassats: Had a suggestion a couple of weeks ago. I think it needs to be reduced to an i8* GEP and then cast to whatever type we need.
20:32:31
drmeister
This looks like the problem where I can't load any quicklisp system after a certain point.
20:33:39
drmeister
I'm trying to get a reproducable case on my laptop rather than the EC2 instance - because ironclad loads before that and it takes a long time on EC2
20:42:01
drmeister
This happens when I (ql:quickload "cl-jupyter") and then (ql:quickload "cl-json")
20:42:24
drmeister
I'm not sure where it's going off the rails - I'll mess around with it for a bit.
20:45:12
drmeister
I don't think that's it. Remember about a week ago I said I was seeing something where quicklisp systems weren't loading with some kind of FUNCTION 0 issue? I think this is it. All that's changed is you improved the error message.
20:48:59
Bike
there's this function determine-system-directly that calls ensure-absolute-pathname with three arguments
20:49:18
Bike
the third is nil. but the third argument to ensure-absolute-pathname is a function designator
20:50:29
drmeister
It's something that only happens after (ql:quickload "cl-jupyter") I can (ql:quickload "read-cvf") or (ql:quickload "cl-json") from a freshly started cando image. But not after (ql:quickload "cl-jupyter")
21:12:20
drmeister
Just add a %shl. The %xxx functions are to add a thin convenience wrapper around the llvm-sys calls.
21:13:24
Bike
when i say "llvm sys has stuff for shl" i mean there are three different functions and i barely understand how llvm works
21:16:58
Bike
https://github.com/drmeister/clasp/commit/2dbfb892f9902b911592f0652975a44d52902fb0 memref and memset are new mir instructions to support more general addressing modes. both take any number of inputs, memset's first is the value being set. they have a constant offset, and a list of power of two scalings
21:19:19
drmeister
We need to switch from using the type 'boolean' to using 'bool'. I'll change all the names in the master branch.
21:21:07
drmeister
A mature Common Lisp implementation would have warned us about this - but Clasp is immature in many ways.
21:21:53
hana
okay. have you looked at image.lisp yet? Im not sure how to populate the b64value slot, all the info is in the value slot
21:22:35
drmeister
hana: Not yet - I've been trying to get the docker image running with all the new changes - I should have it together by tomorrow.
21:24:00
hana
does the docker image have anaconda? ive been trying to install python3 and the ipython docs says to use conda to create a python 3 env
21:25:56
drmeister
Maybe "pip" is used more for installing python 2.7 stuff and "anaconda" is considered the more modern package manager?
21:27:04
hana
maybe. I installed anaconda3 to get python3 but its 3.3gb so adding that to the dockerfile will double the size
21:29:07
drmeister
Yeah - all available hard disk space. They way it tells you it is full is via a segfault
21:30:38
drmeister
hana won't push this pythonrdkit image to dockerhub. We will build and use it as needed.
21:30:58
drmeister
It's just there to figure out what rdkit is doing with the JSME molecular editor and jupyter notebooks.
21:31:43
Shinmera
It seems a bit silly to me to futz about with docker when what you're doing seems more suited to a full VM with a properly integrated desktop.
21:32:19
drmeister
I don't think so - docker is very convenient because it is so lightweight and images disposable.
21:32:56
Shinmera
Yeah but when you need to keep on making images rather than just having a normal system you can develop with...
21:33:03
drmeister
The drmeister/cando docker image has been wonderful on this end. People are up and running with it in minutes.
21:33:41
Shinmera
Well yeah. Deployment is what docker is for. I just don't think it's meant to provide an active development platform is what I'm saying.
21:34:00
drmeister
It's the distribution part that makes it so convenient. (1) we can distribute Dockerfile's to build the image very quickly (2) we can distribute the docker images.
21:34:47
drmeister
By mounting directories into the docker container we can get code in and out of it very easily. That's how our development environment is working.
21:34:59
Shinmera
Right, but you still need to rebuild an image rather than just "apt install whatever & go"
21:36:06
Shinmera
Not to mention that mounting shared folders with VM software is usually trivial, and you can distribute VM images just as well as docker images. So you could ship out a "development base" just as easy.
21:37:10
drmeister
I may ask you about it later. Getting people to install Docker and running the docker image is not trivial either - maybe there is a better way.
21:37:42
Shinmera
I think Docker is a good way to go for shipping clasp to users. Not so much for devs that need to actively change the system base.
21:37:57
drmeister
Clasp can be deployed a lot of ways. But docker has been a life saver because of the ease of getting it up and running on other peoples windows/OSX and Linux machines.
21:38:23
Shinmera
If I find myself bored I might set up a OVM image of a clasp-ready Ubuntu or something.
21:39:41
drmeister
Well, if you want to do something that will take considerable effort - chat with me first and if we can agree that it's a good idea I can pay for your time.
21:43:14
stassats
['/home/stas/c/externals-clasp/build/release/bin/clang++', 'test.cpp.1.o', '-o/home/stas/c/clasp/build/.conf_check_f75d64da4a04879e4c73bbb265528dad/testbuild/testprog', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-lgmpxx', '-lgmp']
0:01:02
drmeister
I had already made a big push towards this by eliminating the definition of several C++ classes that derived from Class_O
0:03:37
drmeister
What about funcallable instances? Currently all Instance_O classes are funcallable.
0:04:15
drmeister
The only difference between an instance and a funcallable-instance is that funcallable-instance has a function entry point associated with it.
0:04:47
drmeister
"The only difference between an instance and a funcallable-instance should be that funcallable-instance has a function entry point associated with it."
0:09:38
drmeister
It's layout is going to be different than the layout of Function_O and everything that derives from it. The function entry_point pointer is the first thing in Function_O and everything that derives from it.
0:11:50
Bike
i mean, right now you have instance as a subclass of function_o so that they have the same layout, right?
0:17:05
drmeister
So it's more like Instance_O -> FuncallableInstance_O and I have to create an Instance_O class
0:22:03
drmeister
I'm thinking about aligning slots for faster/easier access and asking myself what I can stick into the slot that holds the function entry-point in FuncallableInstance_O. I can stick the instance_stamp (the stamp that instances of a class get) into that slot if FuncallableInstance_O can never be a class.
3:02:45
drmeister
It was just a shock to see you say good morning when it's still light out - no matter - don't mind me.
3:23:58
Bike
beach: so yeah, i think i understand how environments would work here, and why you were surprised at how i did the sbcl cleavir thing
3:27:13
beach
How does Clasp currently access a function in compiled code, as in when it does (my-function ....) or #'(setf my-function)?
3:32:00
Bike
what i figured it would be is you have a hash table in the environment, and then you bind fdefinition to read from that hashtable, and setf fdefinition to write to it.
3:33:01
beach
So When compiled code is loaded, instead of finding symbols for those functions, it must find the cells in the environment.
3:34:09
beach
The only time a hash-table lookup is strictly necessary is when FDEFINITION and (SETF FDEFINITION) are used with non-constant symbol names.
3:35:22
beach
So (my-fun ...) at runtime, instead of turning into something like (funcall (symbol-function 'my-fun) ...) it turns into (funcall (car cell-i) ...).
3:36:36
drmeister
Yes, each symbol has a value slot - but it got a bit more complicated for threading.
3:36:39
beach
I knew I couldn't do hash-table lookup for a function call, and I knew the indirection was required.
3:38:50
beach
The value slot, if you have such a thing, must be similarly disembodied and become part of the environment as a cell.
3:40:03
beach
So now, when compiled code is loaded, the "linking" must be against a particular first-class global environment.
3:40:35
beach
But you probably want to make that a second layer, so that below, you can choose what environment to link to.
3:45:16
Bike
ok, so how i figured this would work is that when clasp is building itself, it makes an environment where there are no packages or bindings or anything for cleavir, and compile cleavir in that environment.
3:46:28
Bike
well, that's the concrete version, but you know what i mean. it's the same as how sbcl does a bunch of package junk (and you want to avoid package junk)
3:47:39
beach
But if that is what is done at the moment, then there could be significant changes required.
3:48:17
Bike
in clasp nothing is done at the moment. you load the compiler, and then have it build itself, meaning it recompiles and changes its own definitions as it goes
3:52:17
Bike
so i guess the compiler has to know to output a different kind of fdefinition based on the environment
3:53:15
beach
In the general case, when fdefinition takes a non-constant symbol, it needs to know the current environment, so that has to be one of the things available to compiled code.
3:56:30
beach
CL:FDEFINITION <mumble>) calls (env:fdefinition <mumle> (car current-environment-cell))
4:01:01
beach
You may want to look in Data-and-control-flow/fdefinition.lisp to see how I do it in SICL.
4:03:22
beach
Otherwise, presumably you have a compiler macro that turns cl:fdefinition into a direct access to the cell at run-time.
4:25:21
beach
In SICL, I define all of the functions that use an implicit global environment to trampoline to a generic function that takes an environment as an additional argument.