freenode/#lisp - IRC Chatlog
Search
7:54:38
Shinmera
Statistically, compared to the things I haven't written yet, I have not written anything at all :)
7:56:52
aeth
Shinmera: did you write an infix library? it would be interesting to compare the various solutions people have for implementing one
7:58:24
aeth
I have no use for infix, but I got distracted by it because I think there can be some elegant solutions there.
8:00:38
aeth
I made a solution that at the moment is only designed to work with binary operators with no precedence rules (it's an error to write 1 + 2 - 3 because there's no precedence). https://gitlab.com/snippets/1747132
8:04:20
Shinmera
I mean if all you're interested in is the parser logic, here you go I guess https://github.com/Shirakumo/glsl-toolkit/blob/master/grammar.lisp#L176-L238
8:10:21
Shinmera
Hopefully at some point I can convince Baggers to make Varjo output to glsl-toolkit's AST so that the two can be combined easily.
8:13:31
aeth
The other GLSL generators' goals are actually to make it interchangeable with CL for easy refactoring afaik.
8:16:58
aeth
I can definitely see why Varjo's integration with a math library has it's users, though.
8:38:47
Ober
ACTION images ELS to be the same crowd of folks who would rush to greet an Alien saucer landing.
8:40:35
aeth
Ober: It's unlikely that an alien visitor to Earth would still be biological, so of course everyone would want to know what their source code looks like.
11:23:28
makomo
what is the proper way to ensure a function is available at compile-time for a macro, *if* it's from another package you don't control?
11:24:16
makomo
i.e. you can't just add an EVAL-WHEN around the DEFUN because you don't control the source
11:25:02
Shinmera
? If you have a system as a dependency it's loaded before your system even starts compiling.
11:26:47
makomo
right, that's what i thought but are those DEFUNs from the loaded system required to be in the compilation environment?
11:27:23
makomo
i mean, what's the difference between those and anything i control/compile? i need an EVAL-WHEN, why don't they?
11:27:50
_death
because you put your function in the same file and that file contains uses of the macro
11:29:43
_death
no, the point is the macro-using forms being in the same file as the macro and its dependencies..
11:36:17
Shinmera
It does not. Consider (compile-file "a.lisp") (compile-file "b.lisp") (load "a.lisp") (load "b.lisp")
11:37:21
Shinmera
You're just "lucky" that ASDF compiles and then immediately loads everything sequentially.
11:39:55
_death
if b.lisp is dependent on definitions in a.lisp, then it means a.fasl must be loaded before compiling b.lisp
11:42:59
pjb
If b.lisp needs at macroexpansion time functions defined in a.lisp, then use: (load (compile-file "a.lisp")) (compile-file "b.lisp")
11:43:30
pjb
makomo: the trick here, is that the compilation environment used by compile-file is the run-time environment where compile-file is called.
11:48:39
makomo
but isn't that a separate problem? i guess i wasn't too clear. what i was getting at with "it must be in the same file" was for the case when all of the macro's dependencies are within the same file as that macro
11:49:35
Shinmera
If the dependencies are in separate files, but the files are all compiled without being loaded, you still need to use eval-when.
11:50:10
_death
again, no.. if you move the macro-using forms to another file that gets loaded afterwards, then you don't need eval-when
11:50:33
Shinmera
In the general case of an ASDF sequential-plan, this case won't occur, but it really has nothing to do with files, but simply with the order in which the phases happen to which parts of code.
11:51:35
_death
makomo: me too. if you have a macro and a function that it calls when expanded, there's no need for eval-when if there's no expansion of it happening in the same file
11:51:59
Shinmera
defun is specified to only have load-time side-effects, so you need to eval-when it if you need it to have compile-time side-effects, that's all.
11:52:40
pjb
It's worse than that! If you compile the files without loading them, then it's possible that no compiled definition is known after compile-file returns!
11:52:42
makomo
Shinmera: yeah, true, it isn't intrinsically related to files. it's just that i think of it that way since the lisp compilation process specifically mentions files
11:53:16
pjb
The compilation environment is not necessarily the same as the startup environment! It can be a new environment created by compile-file, and thrown away once compilation of the compilation-unit is finished.
11:54:56
pjb
(with-compilation-unit (compile-file "a.lisp")) (compile-file "b.lisp") would let the compiler know what has been compiled in a.lisp while compiling b.lisp, but it wouldn't be sufficient, since the result of that former compilation may be not available. You would still have to wrap the functions definition in a needed by b in an eval-when.
12:07:15
pjb
makomo: the thing is that even inside the compilation environment (and this is why it's a distinguished concept), compiling eg. a defun doesn't insert into the compilation environment the body of the function, but only the declaration that the name will be fbound at run-time (and possibly other notations such as types of arguments and results, etc).
12:07:37
pjb
This is why loading into the environment before calling compile-file, or using eval-when is required.
12:08:54
pjb
It may seem strange when you think about usual CL implementation, but consider ecl, where the compiler calls gcc to generate an object file. Without an explicit dynamic linking of those object files at run-time, there's no way the compiled definitions would be available in the lisp process.
12:09:49
pjb
This allows also the compiler to be non-atomic and to take its time to compile and optimize. Then loading a fasl can be made atomic, and replace in a consistent way a set of functions.
12:10:20
makomo
pjb: i'm half familiar with that. i read about the compilation environment being distinct and the various uses for EVAL-WHEN but have forgotten some of it by now.
12:11:05
pjb
makomo: that's where asdf (or other system definition systems) come about. It manages the dependencies between the files, and ensure they're loaded before they're needed.
12:16:17
makomo
when the spec says "The evaluation environment is a run-time environment ..." and "The run-time environment is the environment ...", are the uses of the word "run-time environment" here distinct?
12:17:17
makomo
the former is a characterization of the "evalution environment", while the latter is an actual name for a distinct environment?
12:20:25
pjb
There's a category of environments called "run-time environment", and there is a current environment that is a "run-time environment" and that is called the run-time environment.
12:21:06
pjb
(There may be several run-time environments, but in usual implementations, they're all the same; but it's not necessary).
12:21:11
makomo
right. a bit convoluted but oh well. i have nothing against such terminology really because it's natural to overload words, but i don't like when it's not mentioned/clarified by a document
12:21:53
pjb
The important point here, is that the environment where macroexpansions are evaluated (which is a run-time environment), MAY BE DIFFERENT from the startup environment (which is a run-time environment).
12:22:32
pjb
This means that: (defvar *m* 0) (defmacro m () (incf *m*) nil) (m) *m* -> 0 is possible.
12:23:22
pjb
It will surely occur if you put (defvar *m* 0) (defmacro m () (incf *m*) nil) in a file and use compile-file, then load the fasl in a new image.
12:24:38
pjb
I would advise in general, to replicate the side effects in the expansion: (defmacro m () (incf *m*) `(progn (incf *m*) nil))
12:25:12
pjb
(like, when you want to store metadata in tables, that you want to be available both at compilation time and at run time).
12:29:22
pjb
Yes, in a way. But often the side effects are used by the macro itself to determine the expansion, so eval-when wouldn't do.
12:30:36
pjb
in practice you can also have the case: (defmacro m () (incf *m*) `(progn (incf *m*) ',*m*)) where (m) returns the value of *m* from macroexpansion time.
12:31:07
pjb
Of course, *m* is just an example, it could be some type information, some grammar rule compiled at macroexpansion time, real stuff.
12:32:22
pjb
Also, it is better if you can just define simplier macros and functions in separate files and load them and compile in the right order, rather than using eval-when and sophisticated plays between the environments like that. But it's not always possible.
12:33:40
pjb
From a software engineering point of view, I'm not saying that having a single environment and mutating at will at any time is worse. It can simplify things. But it is much less reproduceable, since dependencies can easily become circular. Nowadays, we prefer to have reproducible builds, where dependencies are clear and explicit.
12:36:36
makomo
pjb: i see. and DEFVAR doesn't need any EVAL-WHEN treatment for this incrementing at compile-time?
12:40:58
makomo
so the relation between the environments is compilation >= evaluation >= startup. the fact that it's an inequality is where the portability issues appear
12:44:02
makomo
and how is the "startup environment" different from the "run-time environment"? the "startup environment" is a "snapshot" of the "run-time environment" before compilation took place, but can anything change the "run-time environment" during compilation?
12:45:23
pjb
makomo: defvar and defparameter initialize the variable at run-time. At compilation time, only the fact that those symbols are special (dynamic binding) is noted.
12:46:10
pjb
If a defvar or defparameter form appears as a top level form, the compiler must recognize that the name has been proclaimed special. However, it must neither evaluate the initial-value form nor assign the dynamic variable named name at compile time.
12:48:58
pjb
For example, since any form can be evaluated at compilation time, it would be a perfectly good design, to fork a new unix process into a chroot jail at the beginning of compile-file. Then you could compile code obtained from the wild. In that case, the startup environment would technically be the runtime environment in the new unix process, but it would also be a clone of the original runtime environment in the calling image.
12:49:35
makomo
pjb: right, so with the first *m* example above, the name *m* is unbound at compile-time (assuming a clean startup environment, even if startup == compilation, *m* would still be unbound). how is the macro able to incf it then?
12:49:40
pjb
makomo: A lot of those definitions are actually legalese written to allow different kinds of implementations.
12:50:15
pjb
makomo: good point. It's bug, I would have to use eval-when :compile-toplevel around the defvar.
12:50:58
pjb
This is important. The problem is that it is not a problem until you use compile-file and a clean environment.
12:51:37
makomo
right, because the compilation environment inherits stuff from the startup environment?
12:52:13
makomo
i was a bit thrown off by the "even at the REPL", because i thought that somehow the REPL case was the curious one
12:52:45
pjb
Interactively, it's COMPILE that's used, not COMPILE-FILE, and with slime if we use compile-file, it's called from the interactive runtime environment where such toplevel forms have already been evaluated. (C-x C-e or loading the file).
12:52:52
makomo
well both are, because you were demonstrating the relationship between the environments
12:55:04
makomo
i still don't see the exact difference between "startup environment" and "runtime environment", or rather, i don't see how "startup environment" is a new "type" of environment when it's just a particular instance of the run-time environment
12:55:09
pjb
But when it's time to generate the application, I use a generate.lisp script that is loaded without the rc files, and that will load ~/quicklisp/setup.lisp and quickload the system to compile and load it (often after having cleaned up ~/.cache/common-lisp/ for a clean build), and to save the lisp image. This clean environment is quite different from the development repl/slime environment where we accumulate cruft. So often such
12:56:29
pjb
It's just to distinguish it from eg. the "runtime environment" where macroexpansion occurs.
12:56:45
pjb
If they are different, then side effects from macroexpansion won't be seen after compilation.
12:56:59
pjb
If they're the same, then side effects from macroexpansion will be seen after compilation.
12:59:48
pjb
(defun compile-file (…) (let* ((startup-environment (current-environment)) (evaluation-environment (clone-environment startup-environment)) (compilation-environment (clone-environment evaluation-environment))) (compile-with-environments evaluation-enviroment #| = for macroexpansion |# compilation-environment #| for eval-when :compile-toplevel and compilation-time effects |# …)))
13:00:24
pjb
clone-environment can effectively duplicate the environment or can be identity, returning the same, depending on the implementation.
13:08:36
makomo
pjb: just to dig a little deeper: the fact that this potentially new environment inherits from the startup environment means that it shares the variables (the bindings). however, the name *m* exists in different namespaces (environments) so when (incf *m*) is done, it changes the binding only in this new environment?
13:10:22
makomo
a thing that ties into this and that's been bugging me for a while is how exactly "value cells" behave, namely how the assignment vs. binding thing fits into that picture
13:10:54
makomo
i know that "value cells" are traditional terminology and that the "cell" doesn't really belong to the symbol but to the environment, right?
13:12:55
makomo
but when you do (setf *m* 10), what exactly happens? is it even assignment if it changes the symbol-value of *m* which is supposedly part of the environment, not the symbol?
13:18:25
makomo
why this confuses me is the fact that, if the environments share the bindings (assuming this is even correct) and point to the same integer 0, and if SETF does in fact assign (which it surely does, but i'm confused again) wouldn't SETF-ing either of the *m*s then change the underlying integer?
13:19:42
makomo
(cont.) ... change the underlying integer? the side-effect would then surely be visible in both environments whether or not the environments were independent or not?
13:26:39
beach
makomo: The standard is written so that it is possible to have a slot in the symbol for the global value as a special variable.
13:27:00
beach
makomo: If an implementation does that, then all global environments are basically the same.
13:27:46
beach
Put differently, there is only one global environment in the system. And it is "spread out" in that part of it is symbol slots, part of it in hash tables that are the values of global variables, etc.
13:28:25
beach
makomo: But, a better way of thinking about it is that there is some kind of first-class global environment object that contains the bindings.
13:28:45
beach
makomo: Then you could separate the compilation environment, the startup environment etc.
13:29:22
beach
makomo: And in fact that is precisely the point of this paper: http://metamodular.com/environments.pdf
13:29:55
makomo
beach: what would be the difference between "symbol slots" and "the values of global variables"? aren't these values contained within these slots?
13:31:13
beach
makomo: In a system where symbols have slots for special variable values, the symbol slot would contain the global value of the variable.
13:31:18
Bike
if you write a simple metacircular lisp interpreter, you'll usually have a separate environment object with the values in it, rather than storing them in the symbol itself (since there's no way to do that yourself exactly)
13:31:50
beach
makomo: In a system like SICL, there is no slot in symbols that have anything to do with the global value of that symbol as a variable.
13:32:22
makomo
beach: right, so in a system like that you wouldn't have the hash tables? in other words, the slots and the hash tables can't coexist, right?
13:35:11
makomo
ohhh, now i see. it never occured to me that you could have stuff other than variables/functions in the environment :-D
13:35:57
makomo
that's why it's spread out if you decide to use slots -- value/function cells as part of the symbols, but then the type definitions, etc. within the hash tables?
13:36:34
makomo
and i guess you could go full on with the "slot implementation" and for example store type definitions as part of the symbol too?
13:36:49
makomo
i understand it's purely an implementation thing, but you could do it theoretically i guess
13:36:50
beach
In some systems it is even the case that the function names corresponding to symbols have slots in the symbols, but the ones that have names like (setf <symbol>) are contained in a separate hash table.
13:38:03
makomo
the confusing thing now is that thinking in terms of these "cells"/"slots" gives you a "location"/"place" to think about when talking about assignment
13:38:23
makomo
but when you get rid of these slots and go the "hash table way", what happens when you SETF a symbol?
13:38:37
beach
Yes, it is not very good on the part of the Common Lisp HyperSpec to use terminology that suggests this kind of implementation.
13:39:21
beach
So, in SICL, the bindings associated with variables etc are "cells", mainly just an indirection. I use CONS cells for that.
13:39:48
beach
So when you SETF a variable or (setf (fdefinition ...)) of a function, then you change the contents of that cell.
13:41:46
beach
makomo: But in code, if you just do *variable-name* then the first part is resolved at load time.
13:46:43
beach
makomo: There are probably two reasons that most existing implementations use the "spread out" environment. One is that they were implemented before the standard was finalized, so there was not the concept of compilation environment, etc. The other is that people probably couldn't figure out how to do it without doing a hash-table lookup for each variable reference.
13:47:42
beach
As I recall some implementations of environments in Scheme (where apparently performance is not essential) use a hash-table lookup on every reference.
13:50:37
makomo
Bike: if going the "hash table way" (and before i knew about beach's "trick" with using cons cells as "cells"/"slots") i thought of "value cells" as just bindings and therefore that SETF-ing a symbol just replaced that binding
13:55:07
beach
A global function call turns into (funcall (car (load-time-value (find-function-cell '<function-name>))) arg1 ...)
13:57:02
makomo
beach: cool, i'll have to read about load-time-value again :-). do you use the cdr for anything?
13:58:15
Bike
it's only a cons cos there's no one-element structure built into lisp. could save some memory by making one, but that's about it
13:59:49
beach
makomo: LOAD-TIME-VALUE just evaluates the form (or a pre-compiled version of it) at load-time and treats the result as a literal object.
13:59:54
makomo
the assignment/binding confusion is finally clear to me now. i think i also mixed up the specification and a hypothetical implementation which kept throwing me off -- i was thinking concretely instead of looking at what exactly the spec guaranteed about "cells"
14:00:21
Bike
it's not like the implementation has any reaosn to mandate a memory layout or whatever
14:00:37
makomo
but for that i would have to do a much more detailed reading of the spec so i never bothered, i guess?
14:03:05
beach
makomo: I invented them for bootstrapping purposes. What SBCL does (for instance) is to use different package names at bootstrapping time so that there is no confusion between the host and the target. The Cleavir compiler does not access the host environment at all. It works with a first-class global environment to look up all the information it needs.
14:04:43
beach
makomo: In fact, I can even create a REPL that works in a particular environment, so I can execute SICL code inside SBCL.
14:05:25
beach
I can have part of the environment inherited from the host (like CONS, CAR, CDR) and the other defined by SICL-specific code.
14:09:12
beach
It allows me to debug SICL code without having a finished system yet. That makes the entire thing faster, because I can use SLIME, the inspector, etc.
14:10:12
makomo
pjb: what i wrote above is clear to me now but the run-time environment in which macroexpansion occurs is the evaluation environment, right? i was thinking of *the* run-time environment, but now it's clear to me also that compiled code doesn't have to be loaded within the same system that compiled it
14:11:09
makomo
so the startup environment is the run-time environment which initiated the compilation, as you said, and the run-time environment is whichever environment the code is executed in (perhaps a completely different lisp image)
14:12:17
makomo
beach: hadn't thought of that. truly nice when you can steal parts from other implementations while you work on your own :-)
14:12:17
beach
As I recall there is some restriction that the environment into which a FASL file is loaded must be very similar to the one it was compiled in.
14:13:24
makomo
i remember reading this when we had that discussion about clisp not compiling a function :-)
14:13:34
Bike
there's also a few implied by the externalization rules, like that packages have to still exist