freenode/#clasp - IRC Chatlog
Search
14:10:41
drmeister
Bike, karlosz: Firefox is much better at rendering those complex svg flame graphs
14:14:29
drmeister
yitzi is rapidly bringing jupyter notebooks/widgets/lab online and we have a really good idea of how we are going to use it to make user interfaces.
14:14:54
drmeister
Bike and karlosz are making great progress in improving the underlying compiler that everything builds on.
14:15:36
drmeister
Bike: What do you think about the setjmp/longjmp idea now that we've had a few days working on it?
14:18:05
Bike
the intrinsics are documented incorrectly and are listed as only being generated by internal llvm passes
14:19:29
drmeister
Build a small test case and then compile it with just the c-preprocessor. Do you know how to do that?
14:24:01
drmeister
Sure - but nobody is going to fundamentally change it or there will be heck to pay.
14:25:28
Bike
the jump buffer type is also defined by the C implementation, which is another wrinkle
14:26:32
Bike
yes, i'm nervous about some random change in glibc causing bizarre hard to reproduce unwinding crashes
14:27:03
drmeister
We can put in a sanity check to make sure that the invariants remain so. Also, keep the code in one place. Then we leave it for the future to deal with.
14:27:46
drmeister
The only other way out of this is to figure out what is going on in the kernels where we see crazy slowdowns and try and get control there.
14:30:32
drmeister
Also - can you set this up so that it's optional? So we can turn it off if we do run into problems?
14:31:35
Bike
for a start i was thinking i'd do a really stupid implementation that calls an intrinsic that uses setjmp. it will be way more inefficient than it should be, but going off my initial test, exceptions are so very terrible that it'll still be a win
14:32:01
Bike
plus it shouldn't implicate thread anything, but who knows, maybe libc will find a way
14:55:22
yitzi
drmeister: no difference between lab and notebook from your widget code perspective. To answer your question from yesterday
14:56:24
yitzi
That image also has lab in it btw. I just set notebook to be the default since the nglview kernel stuff doesn't work with lab yet.
14:56:25
drmeister
So if we develop widgets in jnotebook then we can port them to jlab once nglview works in jlab.
14:57:32
drmeister
Second question: There is a long pause when the accordion widget starts up. Do you see that as well? Have you any experience debugging issues like that?
14:59:02
drmeister
This is why it's good that we have an sbcl kernel as well. We can compare the two for performance.
15:00:55
drmeister
I run docker with this now: docker run -it -p 8888:8888 -p 4005:4005 -v $HOME:/home/app/home yitzchak/cando-clj;nglview
15:04:40
yitzi
Yep. The molecule spins fine, but the jupyter message indicator turns black b.c. the clasp is getting overwhelmed?
15:08:02
yitzi
I haven't done any attempt to look at the speed of the kernel code, mostly because SBCL seemed fine. So it could need some analysis/optimization?
16:20:39
yitzi
drmeister: I think I found the blocking issues in nglview that prevents loading in Lab. If I'm right then we could use lab with nglview now.
16:31:09
yitzi
It means that our code won't be at 2.7.5 so some stuff might break, but the widget appears to load. They added some new traits that Javascript expects to be there.
16:31:57
drmeister
The guys who are developing nglview have occasionally changed the names of functions in the Javascript and that requires changes to the kernel code.
16:33:23
drmeister
That's why it's good to keep track of the python code that we match so we can diff it against the python code that they develop and then we make corresponding changes to our Common Lisp code.
16:34:09
yitzi
I am testing it in docker right now, if it works you will have jupyterlab, nglview, and .................
16:36:12
drmeister
Good job figuring that out by the way. With everything going on I'd forgotten about that.
16:36:51
drmeister
yitzi: We are Amber developers and we have access to the Amber development code. But the Conda one should be ok.
16:37:11
drmeister
There needs to be an environment variable set so that cando knows where Ambertools is.
16:39:21
drmeister
Once yitzi gets this docker image together I think we should integrate his Dockerfile changes into our "deploy" script.
16:39:47
drmeister
yitzi: We have a thing we call the "deploy" script. It builds everything on our fledgeling CI system:
16:41:24
yitzi
drmeister: Keep in mind that I don't know if all of the common-lisp-jupyter, nglview, cando-jupyter works yet. I don't really think it is ready for mass consumption yet. It's getting close, though.
16:51:53
yonkunas
yitzi: please use ambertools20. We use /opt/amber as the install directory for the distributor so $amberhome top directory should be amber not amber20. This is all easily changeable I assume
16:52:41
drmeister
yitzi: No worries - this isn't going out for mass consumption. I'm trying to get this to a prototyping tool.
16:53:29
yitzi
drmeister: about to open a WIP PR for cando-jupyter so you can see the current changes. Not ready for merge.
16:59:10
yitzi
drmeister: the WIP cando-jupyter PR is here https://github.com/cando-developers/cando/pull/9
17:00:21
drmeister
What is "AmberTools interop" and "code inspection/completion"? Code completion I know.
17:01:03
drmeister
Regarding AmberTools interop - if it's what I'm guessing it is. In the "deploy" script that we are developing we are going to copy the Ambertools way of deploying a separate Python environment.
17:01:46
yitzi
AmberTools interop = make sure that all the ambertools stuff works from the updated cando-jupyter kernel
17:02:57
drmeister
The way that cando works with Ambertools is cando looks at the "AMBERHOME" environment variable and creates an "amber" hostname within cando. Then it refers to #P"amber:xxx" pathnames to find things.
17:04:17
drmeister
Ambertools provides 'sqm' - it's a semiempirical quantum chemistry package that carries out QM calculations that we need.
17:05:17
drmeister
We also call out to 'pmemd.cuda' - but that is not provided by Ambertools. That is part of the larger "Amber" package that includes Ambertools and provides the "pmemd.cuda" application for simulating molecular dynamics.
17:06:57
drmeister
pmemd.cuda is used in our "distributor" application. The "distributor" is a distributed "make"/scheduling facility that distributes calculations across heterogeneous computer systems.
17:07:44
drmeister
The jupyterlab environment and the widgets that we will assemble will provide an easy to use interface to all of this.
17:08:19
drmeister
That's why I want to start prototyping this. I want to get it out of my head and into peoples browsers.
17:10:29
drmeister
If you set up a notebook that has hidden input cells but visible output cells and the output cells are multicomponent jupyter widgets including things like the accordion widget...
17:10:50
drmeister
Then you will see this verticle display of widgets that you can scroll over and interact with.
17:11:18
drmeister
We could lay out the setup of a complex calculation by stringing together a collection of different widget panels that do different things.
17:12:12
drmeister
For a very naive user the input cells can be confusing. So if we fill them in and hide them - then there is less confusion and less chance that they will type something into them and break the notebook.
17:12:49
drmeister
The GenePattern notebooks appear to have lots of markdown to explain what is going on.
17:13:42
drmeister
So markdown to explain what is going on and widgets with hidden input cells are a 'kiosk mode' notebook that sets up the calculation.
17:14:12
drmeister
At any point a more sophisticated user could unhide the input cells - or create a new input/output cell to augment the basic one we give them.
17:15:20
drmeister
From what I've read (and I haven't dug into this too far) you can edit the .ipynb file and change the JSON code in there to hide input cells.
17:16:26
yitzi
https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/hide_input/readme.html
17:16:26
drmeister
-> The codecell hiding state is stored in the metadata cell.metadata.hide_input. If it is set to true, the codecell will be hidden on reload.
17:17:33
drmeister
I want a jupyter notebook that starts in kiosk mode but that anyone a little more sophisticated can customize.
17:17:46
yitzi
I think with that extension you can create and then export it with nbconvert. Seems fairly straightforward
17:19:38
drmeister
Ok. But how does that help here? It seems like we'd have to open the foo.ipynb file and edit the JSON by hand to hide the cells we want to hide.
17:19:58
yitzi
See https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/hide_input/readme.html
17:21:07
drmeister
Does it mean we have to add some kind of extension to the jupyter notebook/lab environment to support this?
17:24:10
drmeister
Do these extensions include kernel Python code? If so - then we need to write Common Lisp code to mimic it.
17:28:28
yitzi
drmeister: Its rebuilding... you want to explain the "code completion" statement in the PR while we wait?
17:32:34
drmeister
But I'd like to be able to edit code in the input cells and hit tab and see function arguments and docstrings.
17:34:10
drmeister
Ok - what am I doing here? I've purposefully left out the function name. I just put a space there.
17:34:50
drmeister
Then I'd like it to come up with a list of suggestions to put in place right after the open parenthesis.
17:35:31
drmeister
It's like code completion in Python. foo.<point> <tab> -> you get all the methods that apply to 'foo'.
17:37:21
yitzi
And it's already implemented in common-lisp-jupyter. Maybe a little naive, but its there.
17:37:52
drmeister
Yes - the "code_inspect" message is how we would implement this. scymtym is one of our collaborators here who is developing a Common Lisp reader and something like a "language server protocol" that could provide support for this.
17:38:08
yitzi
https://github.com/yitzchak/common-lisp-jupyter/blob/262dab1e79bad0bd010f5d3d0429f8ae36ef0f7d/src/kernel.lisp#L141-L168
17:38:15
drmeister
It requires some fancy programming to predict what is reasonable. scymtym has thought about this a lot more deeply than I have.
17:38:59
drmeister
But I want Common Lisp to be as discoverable and have better programmer assistance tooling than we currently have. Both in slime and in jupyter notebooks.
17:39:39
yitzi
See link above. You just implement "inspect-code" method in your kernel subclass. I have a naive implement currently in common-lisp-jupyter.
17:39:45
drmeister
Yes. I'm familiar with the jupyter facilities that could support this and I'm excited by your generic function implementation of it.
17:41:11
drmeister
I don't implement what I described above - my idea above is still half-baked. I don't even know if it's reasonable.
17:42:18
drmeister
With single dispatch methods as you have in Python - it's easier to come up with a list of methods that apply to a variable. You still need to get an idea of the type of the variable to come up with good suggestions. It's probably really tricky. I think it's a worthwhile thing to try and get right though.
17:42:51
drmeister
That being said. The jupyter notebook input cells are a tedious programming environment for lots of code.
17:43:48
drmeister
Let's see what this docker looks like. nglview in jupyterlab is high priority and then hiding input cells.
17:46:36
yitzi
Look in the upper right hand corner next to the kernel name. Do you see a button with a crossed eye icon?
17:49:06
yitzi
Ok. must have been that extension. Thanks, I'll get back to on the docker image when it works.
18:14:01
drmeister
yitzi: The julia widget example that you have. How does the output work? What kind of widget are you using?
18:17:31
drmeister
yitzi: Within Cando we have code to generate svg files that display two-dimensional images of molecules.
18:19:33
yitzi
To display svg data use `jupyter:svg` defined here https://github.com/yitzchak/common-lisp-jupyter/blob/262dab1e79bad0bd010f5d3d0429f8ae36ef0f7d/src/results.lisp#L241
18:29:15
yitzi
::notify drmeister the julia example is probably in ~/quicklisp/local-projects/common-lisp-jupyter/examples
18:30:20
yitzi
::notify drmeister the julia example uses an "image" widget and sends a png to it. Truly hackish and slooooowwwwww!
19:37:41
drmeister
::notify yitzi - I use a service called irccloud.com - it keeps me logged in to IRC so I don't miss messages to me. I saw your comments about the image/svg widgets.
20:18:02
Colleen
drmeister: yitzi said 1 hour, 49 minutes ago: svg output is done with jupyter:svg and jupyter:svg-file
20:18:02
Colleen
drmeister: yitzi said 1 hour, 48 minutes ago: the julia example is probably in ~/quicklisp/local-projects/common-lisp-jupyter/examples
20:18:02
Colleen
drmeister: yitzi said 1 hour, 47 minutes ago: the julia example uses an "image" widget and sends a png to it. Truly hackish and slooooowwwwww!
20:18:22
drmeister
Removing the where tags or whatever was done with Derivable classes has broken Derivable classes.
22:42:38
Bike
is slot-value-using-class still isn't working for derivables... how far back did you rebuild? that was a change in hierarchy so pretty early