freenode/#lisp - IRC Chatlog
Search
3:51:12
beach
I have been kind of stymied with respect to changes in SICL and Cleavir because I now have a very important client (Clasp). So I finally decided to bite the bullet and "branch" Cleavir so that I can take care of some problems without disturbing the client(s). This decision turned out to be the right one.
3:55:32
equwal
Yeah, I recognise the difficulty of it. I remember you said you though nobody had done anything like SICL before. How is it different from other bootstrapping papers like the ones from this page of Lisp In Small Pieces? spensertruex.com/static/LiSP-bootstrappers.pdf
3:55:32
equwal
I know you compiled with the object first, and Clasp (a very influential project) now uses your compiler!
3:57:06
equwal
Is primarily that you used more modern stuff (since those are all pretty old papers)? Original or not, Clasp is a really big deal.
3:58:07
beach
For my bootstrapping talk, I figured out the main difficulty of building a Common Lisp system, compared to building ordinary "file-processing compilers", like a C++ compiler, for instance.
3:58:26
beach
The main difficulty is that a Common Lisp system does not contain just code to execute the way a C++ compiler does.
3:59:22
beach
A symbol contains a name and a package. A package contains a list of exported symbols.
4:00:54
beach
For Clasp, this is not so good because they have to create the discriminating function of more than 1000 generic functions at startup time and that takes many seconds.
4:01:46
beach
It is also possible to do something like Emacs does (or used to do, I haven't looked for some time), i.e. load up stuff and dump the current image as an executable.
4:02:24
beach
Image-based Common Lisp systems do that exclusively. They never start from source code. They modify an existing image and dump it.
4:03:14
beach
equwal: Yes, I think so. Unless it turns out that LLVM is a major performance bottleneck as has been hinted in the past.
4:03:21
equwal
My understanding of Clasp is that it is pretty nonperformant (I have never compiled it, I can't on my machine) despite it's purpose of scientific computing.
4:04:22
beach
But we have a long way to go with optimizing Cleavir, so that part will eventually no longer be a problem.
4:05:43
beach
Though, as long as Clasp is written in C++ and they can't dump the image, the startup time is not going to be great.
4:06:18
beach
As I recall, Clasp now has several compilers and several interpreters, for various aspects of the system, in order to get it fast enough.
4:27:45
p_l
beach: a significant issue that some people face with Clasp is that its compiler appears to over-eagerly inline functions
4:30:52
beach
p_l: And we currently have no decision procedure for determining WHETHER something should be inlined.
4:31:14
p_l
beach: also, I think it should be possible to generate discriminating functions and include them in "C++ image"
4:31:18
beach
It is entirely possible that the LLVM slowness is due to the fact that we submit HUGE functions to it.
4:31:48
beach
p_l: Bike has done some work on that as I recall. The limitations of C++ are very puzzling to me.
4:32:01
p_l
beach: it was definitely mentioned to me as an issue by one of the attendees I talked with, except for him it was making it too big to compile usable systems
4:33:13
p_l
beach: in practice, for normal OS, you have an external "image builder" which makes the image from different files
4:33:50
p_l
so I see no reason why it couldn't reuse ECL approach, or even create object files that would get relinked with main executable at some point
4:35:14
beach
I know it is possible in C to create a data structure in the executable. You just make top-level struct definitions and creations, linking them as appropriate.
4:37:08
p_l
I wonder if there might simply be an unholy mess at the crossroads of LLVM codegen, C++ runtime, GC, and Clasp's compiler which makes otherwise simple task into an attempt at unmaking spaghetti
4:37:51
beach
I wouldn't attempt anything like that myself, because I would never be able to sort it out.
4:38:39
beach
Luckily, drmeister has much more time and energy than I do to deal with issues like that.
4:39:29
beach
Bike has been working on inlining, but it has taken a long time just to make the mechanism work correctly, so he hasn't had time to think about the WHEN aspect of it.
4:40:59
beach
But, yeah, for SICL, I will definitely not depend on external software that I have to track. Especially not software written in C++.
4:41:48
beach
p_l: It has been proposed that I use LLVM (by several different people) and MPS (same), but I am not going to attempt that.
4:42:19
p_l
beach: an old idea of mine was to perhaps write a parser/writer for ELF objects and try to make SBCL use them as FASL format that is inherently memory-mappable
4:44:46
beach
I think it's the right decision, especially with respect to CLOSOS, because it would be nearly impossible to verify the safety of a FASL consisting of only executable code.
4:54:54
p_l
kinda want to be proven wrong on certain ideas (because it might be more interesting to be proven wrong than right)
5:07:06
p_l
I'd be probably going for something closer to IBM AS/400 design, where safety is guaranteed partially by hw
5:12:19
beach
p_l: The only way to find out what won't work is to attempt it. At least for me. I am unable to think through a design completely without implementing it.
5:20:58
beach
p_l: With respect to Clasp, if it were up to me, I would have rewritten Cando in Common Lisp (including the libraries it uses) and just used an existing high-performance Common Lisp implementation like SBCL. It is several orders of magnitude simpler to implement an application than to create something as complex as a high-performance Common Lisp system.
5:20:59
beach
However, as I often remark, I am not good with strategic decisions like that, and I would have made the wrong one. The decision drmeister made has generated a huge amount of interest in Clasp by people unrelated to chemistry, simply because it can make Common Lisp and C++ work together. This decision has also attracted millions of dollars in grants, supported several graduate students and developers, etc. With me in charge, none of
5:23:30
loke`
beach: If the development of Clasp had been a strategic decision, I'm not sure it would have been taken.
5:23:40
beach
I know I am bad with this stuff, because I often think about what I would have done in Richard Stallman's position. I would have chosen Lisp as the basis for the free system. And that would have been the wrong strategic decision, because then Unix programmers would not have been attracted, and we would not have the body of free software that we do today.
5:23:51
loke`
Many huge projects started because the original developer didn't know that it was impossible. :-)
5:25:53
loke`
beach: True. But when the initial decisions were taken, I'm pretty sure drmeister didn't know the full scope of the work.
5:27:47
loke`
on a smaller scale... Would I have started making Climaxima if I had known that I would have to learn harfbuzz/freetype and implement a new font renderer, along with the copy&paste stuff, etc, etc? Unlikely. I would probably have targetted GTK+ or something instead.
5:28:58
p_l
as for Linus... he publicly stated that it was only his frustration with Minix and lack of availability of other useful OSes at acceptable prices (as a poor student with a 386)
5:29:28
loke`
p_l: right. and he also stated that he'd never expect it to be big and successful (“like Hurd” if I recall correctly)
5:30:36
loke`
So many developments happening due to random things happening by individual people in the past.
5:31:17
loke`
p_l: a lot of what he set out to do has come true. Of course, from his point of view it's not perfect (not enough GNU and too much BSD license)
5:32:57
p_l
loke`: probably didn't help that some of the projects where RMS was more visible had to take a long road to devolve themselves of RMS (and got better for it), and with how people view RMS and FSF as pretty much the same
5:34:38
p_l
beach: ehh, I remember a rather... militant? side around GPL in late 1990s (might be localized effect though), and I do feel disillusioned especially when more than once I caught FSF on FUD campaign :(
5:36:47
beach
I see, so you are one of the disillusioned people. I think we had better not discuss that here. It is off topic, and I am a member of the FSF, even though I don't follow everything they do in detail.
5:38:18
p_l
I dunno if it got disseminated further, but I've seen a fancy CLIM-based visualizer for Cleavir?
5:42:37
beach
To get back to Clasp, one of the main difficulties is that drmeister wanted Cleavir too early. My idea was to work pretty much alone on making the compiler generate good code. But Clasp has build-time and startup-time performance problems, which suggests the need to make the Cleavir compiler faster, as opposed to making it generate fast code.
5:43:25
beach
So there is currently this "conflict" between working on the code generated by the compiler, and working on the way the compiler generates its code.
5:44:11
beach
As it turns out, the latter often suggests special-purpose code, which I am allergic to, because it means a bigger maintenance burden.
5:45:30
beach
This is partly the reason I "branched" Cleavir. It is better to let Bike and karlosz work on the existing version and for me to work on a branch where I am not concerned with the execution time of the compiler itself, at least not for now.
5:52:44
beach
The good part, of course, is that drmeister has made it possible for Bike to make a lot of improvements to Cleavir.
7:45:06
phoe
vector-pop will work well, but it will not reallocate the vector if the fill pointer is small compared to the vector length
7:47:36
asarch
"4While frequently used together, the :fill-pointer and :adjustable arguments are independent--you can make an adjustable array without a fill pointer. However, you can use VECTOR-PUSH and VECTOR-POP only with vectors that have a fill pointer and VECTOR-PUSH-EXTEND only with vectors that have a fill pointer and are adjustable. You can also use the function ADJUST-ARRAY to modify adjustable arrays in a variety of ways beyond just extending the length of a
7:47:56
asarch
This link?: http://www.lispworks.com/documentation/HyperSpec/Body/f_vec_po.htm#vector-pop
7:49:25
beach
asarch: It is essential that you start trying to interpret Common Lisp HyperSpec dictionary entries, so that you can look things up yourself.
7:49:26
asarch
I know, I know. I just I thought that I would need some other "special" function to pop a vector that has been vector-push-extended
7:50:22
beach
asarch: And, don't you think such a function would be present in the dictionary entry on VECTOR-PUSH-EXTEND?
7:57:27
flip214
beach: sorry, I thought that was the way to get a URL to a CLHS page. Seems I remembered wrong.
8:01:01
specbot
set-funcallable-instance-function: http://metamodular.com/CLOS-MOP/set-funcallable-instance-function.html
11:57:52
selwyn
heisig: do you use maxima as part of your research? either together with petalisp or separately
12:02:53
heisig
selwyn: I have used maxima in two previous projects, but I don't use it for Petalisp.
12:06:02
heisig
Some of my colleagues use SymPy, but it is so slow and full of bugs that I'm tempted to write an alternative based on cl4py and maxima.
12:09:58
heisig
beach: Climaxima looks very promising. But my colleagues want a computer algebra system with Python bindings.
12:12:02
flip214
heisig: perhaps putting effort into https://github.com/pinterface/burgled-batteries would be more interesting?
12:14:32
heisig
flip214: burgled-batteries is about using Python code from Lisp (a horrible thought!). What I want to achieve with cl4py is that one can use Lisp code from Python.
12:15:24
flip214
well, why not provide them an interface "load python file" in climaxima instead? Or have a text REPL for them?
12:20:17
selwyn
given the intention that python programmers should be able to play around with code written in lisp, i suspect that the best approach is to have lisp available from the (c)python REPL, if only to make it more attractive at first sight
12:20:42
selwyn
(as opposed to a FFI approach, or having a python implementation written in Common Lisp)
12:22:15
selwyn
this question does not have a cut and dried answer, since it depends on reasoning about other people's prejudices concerning choosing different software libraries
12:23:29
selwyn
one thing i do notice: most enthusiastic python programmers would not like to program in something other than CPython as that implementation contains the bulk of the libraries that make scientific computation feasible (iiuc)
12:26:21
flip214
still, I would see it as a kind of responsibility to gently nudge them to saner programming languages ... ;)
12:26:27
selwyn
i have not used burgled batteries, but i again suspect that in order to use it practically, one would need to make it aware of which of the various python binaries and environments it should use on a given system.. this seems like a lot of work
12:27:45
selwyn
flip214: well, this is one of my hopes from cl4py. unfortunately, it did not arouse immediate interest from some pythonistas i know
12:30:49
selwyn
they said that there's no point given the range of things they can do with python. it's understandable - most people will not experiment with new things if they are happy with what they have. it's a different matter if some new library, such as a 'python-maxima' were to be presented
12:36:13
heisig
selwyn: That is good to know. One thing that might interest pythonistas already is having access to cl:compile (and cl:disassemble).
12:42:24
selwyn
i have noticed some frustration at the reality of having to switch over to C to write performant code for python
12:43:30
selwyn
heisig: anyway, i ask since i am thinking of using maxima for some symbolic manipulation of n-dimensional tensors and matrices
12:45:16
heisig
selwyn: I haven't used maxima for tensor manipulation yet. But there seem to be some modules for it.
12:49:47
guicho
Hi all, last week I presented my numpy-clone in common lisp in Shibuya.lisp meetup. The talk was in Japanese but here is the youtube link to the recording which starts right at the beginning of the demo on the REPL, which you could understand. https://youtu.be/dWfR25EZNjs?t=6259
12:56:18
flip214
why not enter mostly-python-syntax in a text field in clmaxima, convert that to CL (shown in a second field) and then run that?
13:00:25
guicho
heisig: I basically reimplemented a specialized version of type inference mechanism. Currently every inference happens in the runtime, but they will be shifted to the compile time in the future.
13:02:39
guicho
heisig: specialized because it needs to account for the type upgrading in the common lisp array. Ratios are all converted to single-floats. Bignums are truncated to fixnums, and if overflow happens it signals errors. (COMPLEX FIXNUM) does not exist because common lisp does not allow a complex with imagpart 0.
13:05:26
guicho
flip214: but you cannot make a specialized array for (COMPLEX BIT), SBCL does not have it
13:06:36
guicho
heisig: Still struggling to publish it. I have some backer in my company, and hoping it helps. arademaker and fsmunos
13:14:32
guicho
heisig: since I implemented the inference part, you can focus on the distributed part in PetaLisp with the limited type subset you currently have
13:16:34
guicho
heisig: the distributed computation part and the lazy compilation part are your skill set and are more important for your your research question.
13:22:38
heisig
guicho: One more remark - I had yet another idea for type inference over the last few days.
13:23:30
heisig
guicho: Instead of working with type specifiers, one could work with integers that store a bitvector representation of each type as in Baker's subtypep paper.
15:37:49
jsatk
Thanks for the kind replies and investigation sjl & p_l. (I'm actually using sjl's "A Road to Common Lisp" article to learn)
15:37:51
jsatk
My computer went to sleep and lost connection so I may have lost some chat history. Was their ever a resolution?
15:39:26
p_l
jsatk: yeah, just run it as normal - some warnings are expected, because some implementations are bit more strict about what they see as "code smells" and sometimes you get a warning on "hey, I couldn't figure how to optimize that part, sorry"
15:39:53
p_l
jsatk: I have personally used compiler warnings about unsuccessful optimization as guide when optimizing code :)
15:41:55
jsatk
$ sbcl --load ~/.vim/pack/minpac/start/vlime/lisp/start-vlime.lisp 👈 This is all I'm running and it produces the aforementioned errors.
15:42:46
selwyn
p_l: sometimes i get remarks from SBCL about the 'cost' of various operations. does this usually indicate that the 'code smells?' or is it not possible to conclude anything without investigating more