freenode/#clasp - IRC Chatlog
Search
14:50:36
SAL9000
Do you have a slimmer Docker image (w/o the CANDO & Jupyter stuff)? I want to try using your ASTMatcher bindings to generate CM-C++ glue code.
15:40:16
drmeister
The drmeister/cando is the slimmest docker image I've generated. It's about 2.8GB - that's down from about 9GB when I build everything.
15:41:20
drmeister
It builds a full Cando build environment and then builds a slim docker image with minimal dependencies and copies the clasp/cando executable and support files into it.
15:42:21
drmeister
If you want something more tailored to ASTMatcher work - tell me - we can tune a docker image to that.
15:42:33
SAL9000
Right. I didn't mean that it was too big in terms of disk space -- I meant that you autostart Jupyter et.al.
15:44:03
drmeister
That docker image starts a jupyter notebook server from within which you can start a swank server and communicate with the same Common Lisp instance in the jupyter notebook.
15:44:26
drmeister
You might want to think about that. You could develop code with slime/swank and then users could use the jupyter notebook interface.
15:45:47
SAL9000
ideally, the docker image would be set up in such a way that I can load up SLIME in the CM-specific emacs, but that's probably a bit much
15:46:46
SAL9000
NB: I have to use Docker Machine, since I'm using VirtualBox in parallel with Docker
15:46:50
drmeister
Previous versions of the docker image started a swank server rather than a jupyter notebook server.
15:48:35
drmeister
I have two undergraduates who have Windows machines. They are running the Docker toolbox thing.
15:50:09
drmeister
With a few hours work I could create a docker-compose.yml entry that would build a standalone clasp that runs a swank server and you mount a Windows directory within it.
15:51:18
drmeister
We are mounting three host directories into the drmeister/cando container - that turns it into a kick-ass Cando development environment where we edit code in the local file system and evaluate it into swank server running in the docker container.
15:51:41
drmeister
The Common Lisp running in the docker container has access to the host file system through mount points.
15:52:11
SAL9000
Long story short -- I have to use Windows for Configura's stuff, but I want to keep easy access to Linux, so I set up VB with disk passthrough
15:54:16
drmeister
For this idea of using a docker containerized clasp to access the ASTMatcher facilities to scrape C++ code - the VB layer looks like it complicates things rather than simplifying them.
15:55:19
SAL9000
since I want to keep using VB, I can't use HyperV, so I can't use Docker for Windows which only works with HyperV
15:55:40
SAL9000
I happen to be using Docker Machine with the VB backend, so there is also a "VB layer" but that's not a hard requirement
15:56:57
SAL9000
if the pro version can play nice with other virt solutions, I'll talk to Goran about forking over for it
15:58:01
drmeister
I don't know much about the different versions other than that they exist. I _think_ the professional version requires the professional version of Windows.
15:58:45
drmeister
And I _think_ the professional version of Windows has more Posix support and so it requires less work on docker's end to support docker.
15:59:14
SAL9000
closest thing to "more POSIX support" I've heard of is WSL, which isn't enough to run docker
16:01:13
drmeister
All I really know is that I spent several hours banging my head against a wall getting Docker CE to work with drmeister/cando and we got it working. In the next couple of days we will be doing it over again with a new drmeister/cando and this idea of mounting multiple directories into the docker container.
16:03:10
SAL9000
instead of taking a week to migrate everything to windows I just keep running my existing setup in VB with disk passthrough
16:04:46
SAL9000
ugh, even if I fork over for VMWare Workstation it'd still be incompatible with HyperV and thus "native" Docker for Windows
16:06:33
SAL9000
Now that I'm back in Adelaide after my 2 month visit to Configura, I've been thinking about switching to a two-laptop solution for other reasons, which /would/ let me run native docker too in the bargain
16:07:22
drmeister
I'm suggesting for this one thing, for developing code to scrape C++ code - do this... (1) install emacs on Windows (2) Run a clasp docker image that starts a swank server from windows (3) from emacs in windows 'slime-connect' to the clasp instance running in docker. No Virtual Box in there anywhere - but you won't even know you are in Windows.
16:08:38
drmeister
No, we don't use TRAMP. We mount the Windows directories into the docker container.
16:09:10
drmeister
So the source files that you edit are also in Windows directories, but you see them from the docker container and you edit them from the emacs running in Windows.
16:09:27
SAL9000
if I understand correctly I can't take that option because I'm actually running Docker in a Docker-created VM thus bypassing the VB incompatibility
16:10:10
SAL9000
port-forwarding works -- I could use the Jupyter notebooks just fine -- but if I try to mount directories I'll probably end up mounting them in that Docker VM... sigh.
16:10:35
SAL9000
Docker website explains that if you want to use Docker and VB at the same time, I should install "Docker Toolbox"
16:11:22
SAL9000
the upshot is that I have a windows cmd shell where docker commands "just work", but it's *actually* created a VB VM where real Docker is running
16:12:00
SAL9000
I'll have to do some googling to confirm in case docker machine has some TRAMP-like magic, but I doubt it
16:12:56
drmeister
But docker run has the -v option - you use it to mount host directories in the docker container.
16:12:58
SAL9000
again, there's an extra layer of indirection forced on me because of HyperV not playing nice with anything else
16:14:24
SAL9000
"native" Docker CE is the former, the setup recommended by Docker (referred to as "Docker Toolbox" for some reason) is the latter
16:16:26
SAL9000
drmeister: correction, it will not run on Win10 Home, because it doesn't support HyperV.
16:17:42
drmeister
Right - ok. For Windows Home you have to use the "Docker Toolbox" which uses Virtual Box - right?
16:18:50
SAL9000
it's just running normal Linux Docker in an auto-created VirtualBox VM and providing a management tool -- normally intended for, say, cloud VMs running docker -- so you can manage your dockers from Windows cmd line
16:19:18
drmeister
When you say "it's just running normal Linux Docker in an auto-created VirtualBox VM and providing a management tool -- normally intended for, say, cloud VMs running docker -- so you can manage your dockers from Windows cmd line" ...
16:19:37
drmeister
You mean "Docker Toolbox [is] just running normal Linux Docker in an auto-created VirtualBox VM and providing a management tool ..." ?
16:20:10
SAL9000
Docker Toolbox is, as the name suggests, a toolbox. The approach USING Docker Toolbox to emulate "native" Docker is that.
16:20:33
drmeister
This is helpful to me because in the next week, two undergraduates are showing up with Windows Home running on their laptops, and I need to get drmeister/cando running on their machines.
16:22:39
SAL9000
the latter is using special features of your CPU to make it multitask at a deeper level than otherwise
16:25:43
SAL9000
Now we get to "hypervisors". This is something of an overloaded term; in the case of software virtualization, the interpreter and it's environment is the hypervisor.
16:26:03
SAL9000
In the case of hardware virtualisation, the code running in Ring -1 is the hypervisor
16:27:17
SAL9000
there are two "types" of hypervisors, although not all of them fit into one of the categories cleanly -- native or bare-metal, and hosted
16:27:43
SAL9000
the former is itself the OS, or built into it; the latter is a program running on a conventional OS.
16:30:48
SAL9000
both of the latter are kinda like device drivers, with userspace management interfaces
16:31:53
SAL9000
on a Windows edition which supports HyperV, if it is enabled, it "reserves" the hardware extensions (VT-x) for itself at boot, and only releases them at shutdown
16:33:08
SAL9000
this means that nothing else can use VT-x while HyperV is enabled, which prevents VirtualBox and friends from using hardware virtualisation and thus achieving decent performance
16:33:24
SAL9000
or it may in fact prevent them from running at all, if they can't fall back to bog-slow software virtualisation
16:34:17
SAL9000
according to the Docker forums, their team had some problems integrating VirtualBox into their Docker for Windows package, so they decided to sacrifice compatibility in favor of "native feel" and ease of installation
16:34:46
SAL9000
end result: native Docker for Windows only runs on editions supporting HyperV (not Home), and does not play nice with ANY other hardware virtualisation
16:35:38
SAL9000
now, before Docker for Windows came out, the only way was to get access to SOME kind of Linux box and run docker there
16:36:02
SAL9000
this could be a physical box, a VM on the same computer or some cloud box, doesn't matter
16:36:26
SAL9000
Docker Toolbox is designed to ease remote management of machines running Docker, such as the above
16:37:21
SAL9000
thus beleagured users who are unable to run Docker for Windows -- whether due to Home edition or conficting virtualisation -- are recommended to install Toolbox and use its automatic VM builder to get a "remote" Docker which is actually a VM in VirtualBox.
16:38:42
Shinmera
SAL9000: isn't software virtualisation decently fast with things like binary rewriting?
16:38:44
SAL9000
NB: I have not in fact tried it myself -- I just googled for docker machine remote mounting, there MAY be a special case.
16:39:14
drmeister
Ah - I'm not sure the transparent local folder mounting will work - we will find out tomorrow.
16:40:43
SAL9000
drmeister: there is a workaround -- virtualbox also supports folder mounting of sorts.
16:40:52
Shinmera
If I remember correctly VMWare uses binary rewriting and I've never had issues with its performance.
16:41:40
SAL9000
I haven't used that in years, but afaik QEMU is using a similar approach, and I have used that for running ARM on x64
16:42:58
Shinmera
I learned a whole bunch about virtualisation stuff in my OS lecture but it's been a year and a half now and I only remember faint detaisl
16:45:50
drmeister
SAL9000: We have a git repository that we are using to modify the drmeister/cando docker image.
16:47:08
drmeister
We have the docker image drmeister/cando - it runs Cando and starts a jupyter notebook server.
16:47:49
drmeister
On OS X we use the "run-docker" in the top directory and it mounts three directories within widgets-dev into three different locations within the docker container.
16:48:17
drmeister
That overrides the directories within the docker container with directories on the host that contain source code that we are developing.
16:49:49
drmeister
Also, leave out the one that mounts into cl-jupyter - the cl-jupyter directory in widgets-dev is broken - I'm working on it this afternoon.
16:50:20
drmeister
The idea is we have this docker container that runs Cando and we patch in host directories with development code within them and turn it into a portable development environment.
16:50:54
drmeister
If we can't mount those directories - then we have a problem. I assumed that the docker toolbox for Windows Home could do that.
16:52:21
drmeister
Ok, so we will know more here tomorrow. If there is some other way to mount host directories we can probably come up with a workaround.
16:53:15
SAL9000
It seems that there is a way to mount host directories -- although it might break if your Users dir is not on C:/ -- but I'm expecting all kinds of fun wrinkles due to the two layers of magic-folder indirection
16:54:24
SAL9000
drmeister: it's important to note that VirtualBox's shared folders are not like Docker -- they work "in the other direction", if that makes sense.
17:11:59
SAL9000
Docker's -v folders are mounting directories in the guest onto/over directories on the host
17:12:25
SAL9000
VirtualBox shared folders are providing directories in the host as "network shares" exposed to the guest
17:12:43
SAL9000
the guest must then use its VirtualBox-specific drivers (Guest Additions) to mount those shares wherever
17:13:45
SAL9000
Docker's -v folders are mounting direcotires in the host onto/over directories in the guest
17:14:39
SAL9000
also, it seems that the default boot2docker image doesn't actually mount /c/Users by default =/
17:48:49
drmeister
I've rejiggered my CMP:WITH-TRY macro - it gives me something that's a hybrid of UNWIND-PROTECT and TRY/CATCH (TRY/CATCH/FINALLY really) within functions using LLVM.
17:53:25
SAL9000
drmeister: Looks like I've found the bug. The default Docker VM is being created with a shared folder using Windows' extended path syntax (\\?\f:\oo\bar), which (at least my somewhat old version of) VirtualBox doesn't grok in this context, so the automount fails, so /c/Users is meaningless, etc. etc.