freenode/#clasp - IRC Chatlog
Search
10:31:12
Shinmera
drmeister: https://github.com/trivial-garbage/trivial-garbage/pull/12#issuecomment-421313074
10:32:48
makomo
beach: did you see my question regarding the :identity initarg for the various *-INFO classes within Cleavir?
10:33:45
makomo
beach: i was wondering what the internal purpose of that initarg was. Bike told me that lexical variables for example use the lexical-ast as the identity
10:34:16
makomo
but i was just curious what internal purpose that serves. i was reading metamodular.com/cleavir.pdf and the only guarantee that the user has to make is that always the same object is passed as the :identity argument
10:34:41
makomo
Bike also said that Clasp never passes the :identity initarg, but cleavir.pdf says that it's a mandatory initarg
10:35:09
makomo
so the docs might be stale (or maybe the last version isn't the one that's on your site)
10:54:57
beach
makomo: It seems that if you have a local named function (with FLET or LABELS), then if you do #'<name-of-that-function> then the result of converting that form is the contents of the :IDENTITY slot.
10:56:47
beach
So that must contain the AST for the function because that is what is returned by CONVERT-SPECIAL.
11:03:31
beach
makomo: It contains the AST corresponding to the lexical variable naming the function. So when there is a #'<name> in the code, then it turns into an AST that accesses the corresponding variable.
11:08:47
makomo
i haven't seen that terminology used before though, i.e. calling NAME in that context a lexical variable
11:09:13
makomo
because, (flet ((hello () (print "hello"))) hello) will fail with the error that "the variable HELLO is unbound"
11:10:08
makomo
i've only seen "local function" until now (i guess you could also say "lexical function")
11:10:21
beach
makomo: Once the source code is translated to AST, there is no distinction between the variable namespace and the function namespace.
11:12:45
beach
But when the code is translated, the first form has F as a variable-environment and the second one has F as a function-environment. When #f is seen, Cleavir asks for a function-environment with f.
11:14:31
makomo
so, if my <- was a macro (which it will be), i couldn't code walk and look for occurences of <-, but whatever internal function that performs the signal assignment instead
11:15:28
makomo
instead i have to look for occurences to whatever internal function performs the signal assignment*
11:15:52
makomo
i phrased that badly. i can code walk, but i can't look for occurences of <-, since <- is a macro and it'll be gone
11:17:21
makomo
i guess that's ok. the DSL will just have "primitives" which are normal functions. the user of the DSL uses the macros, but my AST traverser looks for the primitive functions
11:19:03
makomo
i tried the macroexpansion solution, but i don't like it too much. once i store away the information during macroexpansion-time, i have to retrieve that information when the form is actually evaluated
11:19:29
makomo
i can't retrieve it during that same macroexpansion/compiler invocation, because i can't rely on the order of macroexpansions
11:20:09
makomo
so if i compile my DEFCOMPONENTS in one image, and try to load them in another, it'll fail. the hash tables that kept the state that was computed during macroexpansion are gone
11:22:26
makomo
e.g. DEFCOMPONENT --(expands into)--> (ensure-component-spec ... :body (lambda () <body with state-computing-macros>) ... :stuff <state-computed-by-the-macros>)
11:23:15
makomo
<state-computed-by-the-macros> will truly be available only once <body-with-state-computing-macros> is macroexpanded/compiled
11:25:03
makomo
i was reading about how Cleavir needs method definitions in order to operate on the environment
11:25:12
beach
You will need an object to represent the global environment. You can check what the Clasp people are doing for that.
11:25:21
makomo
i suppose Clasp uses Cleavir from the bottom-up and has the "leisure" of defining their own env object
11:25:55
beach
I think they must define functions on the environment that trampoline to ordinary Common Lisp functions.
11:26:14
makomo
all of the global function/macro definitions will be stored within SBCL's environment
11:26:45
drmeister
Shinmera: I added you to clasp-developers - if there are any systems that you want to access and push upstream but don't have access - just ask.
11:27:36
makomo
beach: oh hmm, that trampoline technique makes sense. what about lexical variables though?
11:28:13
drmeister
Shinmera: I pushed trivial-gray-streams upstream about three weeks ago - there is no reason to keep a fork of it now - right. It's actually counterproductive to keep a fork.
11:28:44
makomo
beach: the macroexpand-with-a-different-environment-object thing? i heard you discussing that with mraskin but i'm not sure whether i completely understood
11:28:58
beach
makomo: If you macroexpand a macro in a lexical environment and the expansion function calls MACROEXPAND, then the host MACROEXPAND will receive a Cleavir lexical environment.
11:29:27
Shinmera
drmeister: and yes, the fork will only grow outdated if the patches are already upstream
11:31:05
makomo
beach: does that concern only MACROEXPAND, or various other environment-taking functions as well? what about GET-SETF-EXPANSION for example?
11:32:29
makomo
so if GET-SETF-EXPANSION is problematic as well, this would wreck all of the place macros then?
11:33:08
beach
It is not very likely that those will be called in a lexical environment, but it can happen, yes.
11:33:34
beach
The absolute best solution would be to create a SICL first-class global environment that imports most things from the host, but that replaces those sensitive functions. I have wanted to do that for SBCL for some time.
11:45:59
beach
Bike: What if the result of expanding MAKE-METHOD were wrapped in LOAD-TIME-VALUE. Then it would be evaluated in the null lexical environment.
11:48:00
beach
That book "Common Lisp for language implementers" really has to be written. It would explain solutions like this. Not just assume that everyone knows what to do, as the Common Lisp HyperSpec does.
11:48:37
beach
Since it very likely won't be a bestseller, perhaps turning it into a collaborative project would be the way to go.
12:00:08
makomo
beach: i'm still thinking about what you said regarding that problem. so the core of the issue is that a local macro will receive a Cleavir environment as its &environment parameter?
12:03:42
makomo
SETF will use GET-SETF-EXPANSION and pass the environment and that would fail somewhere within the internals of GET-SETF-EXPANSION i suppose?
12:05:13
beach
MichaelRaskin hinted that his code walker can deal with this situation, but he utterly failed to explain to me how.
12:05:49
makomo
yeah, i heard that as well, but i also didn't quite understand what he was trying to say. some trickery and heuristics is what i remember
12:06:46
beach
You could replace the host functions get-setf-expansion and macroexpand-1 by a function that first checks if it is a cleavir environment and if so, calls a cleavir-specific version, and if not, calls the old function.
12:10:16
makomo
hm yeah, and that should be portable across implementations right (aside from the fact that it's already UB since you're modifiying the COMMON-LISP package)?
12:19:39
beach
Wow, I am *so* pleased I made progress on MAKE-METHOD and CALL-METHOD. Now I can clean up the rest of the code and continue with my bootstrapping effort.
12:31:43
drmeister
The problem with trivial-garbage is this test: (deftest pointers.1 (weak-pointer-p (make-weak-pointer 42)) t)
12:40:03
Shinmera
Surely you can put fixnums on the heap, so why can't you make a weak pointer to one?
12:43:10
beach
The point of weak pointers is that when all other references to an object go away, then the weak reference will no long refer to the object, so it can be garbage collected.
12:45:11
pfdietz
makomo: to answer your question of yesterday: the information in the macro environment is used when expanding other macros. Think of this as happening at compile time.
12:52:08
Shinmera
The implementation can just allocate a word on the heap, put the immediate there and give you the pointer to it.
12:52:55
Shinmera
I agree that I don't see why one would want a weak pointer to an immediate, but I also fail to see the problem with implementing it
13:08:06
pfdietz
I was reading Cracauer's blog post about how sbcl does it (CONS is just incrementing a register).
13:14:21
pfdietz
Yes, and it's done for performance reasons. (defun foo (n) (loop repeat n collect 1)) will run much faster with that low overhead allocation.
13:31:15
drmeister
How about this? (make-weak-pointer 42) returns the immediate fixnum 42 and (weak-pointer-p <immediate>) -> T
13:34:13
pfdietz
Well, except if you have an immediate as a key in a weak key hash table, it will never be collected.
13:41:28
stassats
while checking that it doesn't overflow, and ensuring that it's not interleaved with other allocations due to asynchronous interrupts
13:48:40
Bike
beach: as far as i know, the identity of variable and function infos is only used for lexical infos that cleavir makes itself. identities for global infos, which are created by the client, are unused
13:50:11
scymtym
pfdietz: there is a thread-local allocation region data structure consisting of a fill-pointer-like value that is incremented and an end-pointer against which the fill-pointer is checked for overflow. overflow of the allocation region triggers an out-of-line handler
13:52:48
pfdietz
Anyway: if clasp is looking for optimization opportunities, speed of CONS is one of them. (loop repeat n collect 1) is about 6x faster in SBCL.
14:14:12
Colleen
kpoeck: drmeister said 16 hours, 8 minutes ago: I'd like to incorporate the tests you have developed into the CI framework that I'm developing. I have clasp and cando building using 'buildbot'. What do you think?
14:17:03
kpoeck
I think all my tests that would crash current clasp are with the respective Pr to solve the problems
14:19:39
kpoeck
There are "expected-errors" in the test. In the case of regression there should be unexpected errors. Perhaps this can be used in the CI
14:28:01
Shinmera
Bike: can you give me access to the usocket fork or do I have to wait an drmeister for that?
14:55:37
makomo
pfdietz: thanks for the answer. right, but can you use the collected information at macroexpansion-time within the compilation invocation of that same top-level form?
14:56:16
makomo
so i have a top-level form F that is being compiled (and hence macroexpanded). i want to both store and use certain information within that same invocation?
15:04:38
makomo
pfdietz: here's the relevant code http://plaster.tymoon.eu/view/921#921. perhaps another layer of macro indirection can fix it? wouldn't that be relying on the order of macroexpansion though?
15:06:01
pfdietz
That was either used as a constructor, or as a pattern matcher, depending on where it occurred.
15:07:12
pfdietz
When parsing the string, the parser invoked macroexpand on a fake macro, and the env arg, to pull out a symbol table. The flow of information was always downward.
15:07:30
makomo
pfdietz: do the quotation marks indicate that it was used within strings or was it just to make it explicit?
15:08:25
pfdietz
The string occurred inside a thing that identified it as something to be parsed, and that thing was created by a reader macro for syntactic sugar.
15:09:45
pfdietz
Passing information upwards would involve a different mechanism: the surround macro would have to fully expand <body>, then walk over the expansion to pull out things the lower macros had stashed there.
15:16:29
makomo
hm right, i think my case is the latter. i need to pull stuff from the body of the macro
15:18:33
makomo
pfdietz: why does the outer macro have to walk the body's expansion? can't the local macros within body communicate information via some global state (as i'm doing with hash tables)?
15:23:44
pfdietz
Because in normal macroexpansion, the outer macro is expanded before the inner ones. The outer macro's function is going to have to "manually" cause its parts to be expanded "early" if it wants to get any results of those expansions before it is done expanding itself.
15:29:29
drmeister
I got the buildbot to build independently for the first time using AWS spot instances.
15:29:39
makomo
pfdietz: the walking i was thinking of was the one you said happens *after* the manual macroexpansion. once i expand the body, why do i have to walk the body? is that just one of the alternatives (and the other is using global state)?
15:30:20
makomo
you said "(...) the surround macro would have to fully expand <body>, then walk over the expansion (...)"
15:30:40
drmeister
It's easy to add non-AWS workers, save build products, chain builds and add tests. I'm interested in discussing what we should do with these new CI capabilities.
15:32:21
pfdietz
The macro expansion function for the outer macro needs to get some results computed by macros inside its body. Those results are not yet computed. The outer macro's function needs to explicitly cause those inner macros to be expanded if it wants those results. This will be done by walking DURING expansion of the other macro: that outer macro function will explicitly call a code walker to expand the macros in its body. There
15:32:22
pfdietz
will be another pass over that code afterwards, by the normal expansion process, but by that time all the macros will already have been expanded so that will be a no-op.
16:14:36
Bike
changing a lisp-exposed C++ function's signature to have Fixnum_sp instead of size_t is causing a build failure for reasons i don't understand
16:40:00
kpoeck_
Regarding the weak-pointer for an immediate, can‘t we put the immediate value in a vector and return a weak-pointer to the vector?
16:42:16
Bike
i don't understand why trivial-garbage even wants pointers to immediates. uniformity, i guess?
16:43:30
Bike
if we just do an actual weak pointer to a vector, the weak-pointer-value or whatever would be wrong, though
16:58:41
drmeister
A weak pointer contains a single slot 'value' and this is a pointer to an object on the heap.
16:59:27
drmeister
https://github.com/clasp-developers/clasp/blob/dev/include/clasp/gctools/gcweak.h#L535
16:59:43
Bike
there's an assert saying "value can never contain anything but a pointer - if it does then when it gets set to NULL by the GC it will be interpreted as a Fixnum 0"
17:00:09
Bike
make-weak-pointer seems to return a WeakPointer_O, which has a WeakPointerManager in it
17:00:53
drmeister
Right - disabling the ability to allocate weak pointers to all immediates might have been an overreaction.
17:01:37
pfdietz
Since common lisps are allowed to copy numeric (and character) values at any time, the whole notion of weakness for them is dubious.
17:01:56
drmeister
I think I could allocate weak pointers to any immediate value that isn't 0x0 - I'll just NOT tell the GC to watch that pointer and splat it because the value won't be a pointer.
17:03:41
Bike
it's dubious, but on the other hand it would be kind of dicey to make the user aware of what's immediate and what's not
17:04:33
Bike
also i thought making a parameter size_t would imply a sign check when converting from fixnum, but no. too bad
17:05:00
drmeister
Yeah - it doesn't make sense to allocate weak pointers on immediate values. But different lisps have different types as immediate values - so I guess you need a consistent API.
17:06:27
drmeister
I didn't sign check when converting fixnum -> size_t - you can find the translator and add that.
17:06:57
drmeister
Actually, we shouldn't put those kind of tests in translators - they will slow things down. Type inference is a better way to handle it.
17:07:50
drmeister
I think unsafe translators and type inference that generates appropriate bounds checking is the best way to deal with this.
17:11:35
pfdietz
Users already can tell by using EQ and EQL. I've encountered cases in SBCL where use of EQL type declarations alters the EQ-behavior of a function (allowed by the standard, but weird.)
17:13:45
pfdietz
Yes. And the constant in the EQL declaration is not EQ to the constant being passed to the function.
17:13:45
stassats
the only way for the type check to succeed is to have the same constant, so it substitutes all the cases where it's used
17:18:49
stassats
in that case, nothing's actually optimized, loading 1d0 is more expensive than returning the argument
17:21:36
drmeister
stassats: The only problem in my mind when I wrote this is that the value cannot contain 0x0 because the Boehm GC uses 0x0 to splat the pointer. Now I see a solution to this. I could add a flag that tells me if value contains a pointer or an immediate. If it contains a pointer - then 0x0 means it was splatted. If it contains an immediate - then I don't tell the GC that the value contains a splatable pointer.
19:13:20
drmeister
I was going to ask about updates - I'll do it in slack - I don't know if Steven sees posts here.
20:48:07
drmeister
::notify Bike The cst compiler behaves differently than the ast compiler. This is what I get in the Activity Monitor. https://usercontent.irccloud-cdn.com/file/1R6CRBUG/image.png
20:52:09
drmeister
::notify Bike - It seems to be spending a lot of time in sigtramp https://usercontent.irccloud-cdn.com/file/wQjK39GH/image.png