freenode/#lisp - IRC Chatlog
Search
11:46:09
beach
So I think I am incapable of writing and debugging these MAKE-METHOD and CALL-METHOD wrappers without also having a version of macroexpand-all available.
12:01:48
MichaelRaskin
That's true, and if you start from the top-level/null lexical environments I only use my own implementation of lexical environments
12:02:16
MichaelRaskin
Which is, to be honest, quite limited, because I only need rough idea of what-means-what.
12:02:34
beach
So how do you deal with the possibility of the client version of MACROEXPAND being called with one of your environments?
12:03:14
MichaelRaskin
The client version of macroexpand cannot be called with one of my environments.
12:04:27
MichaelRaskin
Because my implementation's type is a class I define. And macroexpand wants implementation-specific environment object.
12:04:30
beach
It suffices to have a macro expanded in a lexical environment that calls macroexpand.
12:05:46
MichaelRaskin
If you want to macroexpand-all in a complicated lexical environment, you need one of the few tricks.
12:07:30
MichaelRaskin
Option 2: I can take the lexical environment object and a list of names I should check for macro definitions in that object.
12:08:03
MichaelRaskin
Option 3: if the local macros are used in a simple enough way, my heuristics might be enough.
12:08:46
beach
OK, I'm lost again. I don't know what a "macro-based macroexpand-all" is. And I don't know whether by "I can ..." you mean that this is something you could implement in the future, something I need to do in order to use your tool, or something else.
12:09:15
beach
I am sorry, I seem to be having this problem of understanding what is said in IRC. You need to be more explicit for me to understand.
12:10:51
MichaelRaskin
macro-macroexpand-all on a form will replace a form with some weird code that returns hthe full macroexpansion and should always be enough to take into account the local lexical environment
12:12:25
MichaelRaskin
(macroexpand-all form environment :names names) should also always work as long as all local names are listed in «names» parameter (listing extra is not a problem)
12:13:00
MichaelRaskin
Of course, it is possible you find a bug, in this case I will thank you and upload an updated version.
12:14:16
beach
I think I'll go with Cleavir anyway, because I need to expand macros that I defined in one of my first-class global environments. And I can't do it in the host, because they are macros that clash with those of the host, like DEFMETHOD.
12:15:24
MichaelRaskin
Well, yes, if you control the environment implementation it makes a lot more sense to use it.
12:15:51
beach
Yes, and in this case, like I said, I pretty much have to. Unless I rename all my macros just for the purpose of testing them.
12:36:56
Bike
was usually harder for me to understand than ir, though, because it's based on an expression mostly-but-not-actually-a-tree instead of a cfg
12:44:47
heisig
The example is nice though, it has blocks, bindings, tagbody, conditionals, type annotations and mutation of local variables. But it is definitely not a small expression :)
12:45:51
beach
The question I am asking myself now is whether an AST viewer would make it easier to understand.
12:47:30
beach
There are plenty of LOAD-TIME-VALUEs (at least in SICL code) for fetching global function cells.
12:52:41
heisig
Another possibility would be to introduce a pattern-based simplifier into AST-to-source, e.g., to eliminate superfluous PROGNs, convert direct lambda calls to LETs and so on.
12:56:29
beach
All these tools (existing and suggested) could be useful, so they should all be written. :)
13:01:46
beach
Let me start with macroexpand-all. I think it may be trivial, given that we have GENERATE-AST.
13:30:51
dim
any SBCL maintainer around? https://github.com/dimitri/pgloader/issues/832 looks like an SBCL issue
13:33:22
scymtym
dim: i think the person is trying to build with a version of nibbles that is too old for SBCL 1.4.11
13:34:18
pfdietz
These packages take terrible liberties with the naughty internal unexported bits of sbcl, and break when those change.
13:46:31
flip214
has anyone used CL-SAT already? With some quite complex input I get "the value NIL is not of type NUMBER" in CL-SAT:PRINT-CNF.
13:51:09
dim
pfdietz: pgloader build process should check out directly from QL, so I'm surprised; what might have happened is that the OP had a cache of quicklisp from a previous pgloader build
13:51:52
dim
I know nothing of the env where they try to use it, they know nothing of CL and QL and other bits, and it's all fine, usually
14:15:52
pfdietz
The problem here is that a widely used library is depending on unexported internal details of sbcl. Perhaps this means sbcl should have some sort of exported interface that the library could use instead, but sbcl developers are always going to feel free to change things others don't have license to depend on.
14:17:37
pfdietz
Any time one builds something that depends on unexported things, one is living in sin, at least to some extent.
14:28:26
beach
Bike: Oh, that's not what I meant. I meant that I could copy generate-ast and modify it to expand instead.
14:39:25
pfdietz
If you want to selectively inline it later, you declaim notinline right after the defun, then at local uses you can locally declare it inline.
14:40:04
pfdietz
The initial inline declaim causes the compiler to store the source for the later selective inlining.
14:58:23
shrdlu68
First macro I've written in a while: (defmacro post-incf (var) `(prog1 ,var (incf ,var)))
15:06:15
pfdietz
There's infrastructure in the CL standard for solving this problem, since it has to be solved for various builtin macros on places.
15:10:23
pfdietz
See the example I gave: the place expression can have side effecting subexpressions.
15:13:26
pfdietz
One also has to make sure the subexpressions of the place continue to be executed in left-to-right order. ansi-tests checked for all this.
15:15:37
shrdlu68
pfdietz: Hmm, shiftf suffers from the same shortcoming. Is there a way of avoiding that?
15:18:11
pfdietz
You can use GET-SETF-EXPANSION to explode places into pieces that can be assembled in the macro expansion. http://www.lispworks.com/documentation/lw70/CLHS/Body/f_get_se.htm
15:22:08
pfdietz
(LET* ((#:Y1080 Y) (#:G1081 (INCF I)) (#:OUT1082 X) (#:NEW1 (AREF #:Y1080 #:G1081)) (#:NEW1 Z))
15:48:15
beach
I believe I have a first version of MACROEXPAND-ALL working. It takes a cleavir environment that it operates in.
15:52:11
makomo
pfdietz: do the subforms of a place *have* to be evaluated left-to-right? i thought that was just a nice property/convention, but that it wasn't required for user-defined places
15:54:37
makomo
pfdietz: for example, i wrote an IFF place, which conditionally writes/reads one of the two places you provide to it as arguments, along with a condition form
15:56:22
Bike
' The evaluation ordering of subforms within a place is determined by the order specified by the second value returned by get-setf-expansion. For all places defined by this specification (e.g., getf, ldb, ...), this order of evaluation is left-to-right. '
15:57:43
makomo
i was going to write condf and other conditional places by basing them on iff, but i have to make iff take an optional second place first
16:26:50
slyrus1
right. I was going to say "presumably the AST stuff makes it possible to write such a thing nicely"?
16:27:28
beach
Sure. I did it the easy way. I basically translated the code from generate-ast (by hand) to cover all the cases.
17:00:40
Demosthenex
i'm just trying to confirm some edge cases in what i'm working on, and honestly i could just repeat things in the repl, but thought i should document it a bit better
17:04:37
beach
makomo: When possible, I write "random tests", i.e. I generate huge numbers of test cases to my code. when applicable, I write two version of my code, one "production" version, and one "trivial" version.
17:04:58
beach
makomo: The are supposed to behave the same way, but the "trivial" one is too slow for the final version.
17:06:00
makomo
beach: oh i see. how do you generate the random tests without basically solving the problem though? how do you bootstrap, i guess? :-)
17:06:34
Colleen
Clhs: function funcall http://www.lispworks.com/documentation/HyperSpec/Body/f_funcal.htm
17:07:59
beach
makomo: Then I check the tests against the code cover. If there are places that have not been executed, I try to modify my random-test generator so that those are included.
17:12:57
makomo
beach: hmm, i see how one could easily generate sequences of those operations, but how do you generate the results to test against, without already implementing the API? maybe i'm thinking about it the wrong way or something
17:13:34
makomo
or perhaps you rely on an already existing implementation of the same API (or a similar one)
17:15:12
beach
There could be bugs in the trivial one as well of course, but the probability that it would be "the same" bug in both versions is infinitesimal.
17:15:15
makomo
so you're relying on the fact that the trivial version is as trivial as can be, and that you will probably easily catch the errors since the code is simple
17:16:44
beach
But yeah, the "trivial" version is usually so simple that you can just look at it and be convinced that it is correct.
17:18:29
beach
The "trivial" version stays the same unless the API changes, which should not be the case (at least not very often).
17:33:44
pfdietz
The first is generative: have a grammar (explicit or implicit) and attach probabilities to different production rules.
17:34:22
pfdietz
The second is mutational: given a corpus of interesting inputs, generate new inputs by changing or combining them in various ways.
17:34:52
pfdietz
One can also add constraints to the inputs to try to bias them toward more interesting executions.
17:35:33
pfdietz
And if one can instrument the code under test then the inputs can be tweaked based on whether in the code the executions go ("gray box fuzzing").
17:35:39
beach
pfdietz: I often need my generated operations to follow a Markov process. Unless I have a certain probability of generating long sequences of "the same" operation, my coverage won't be adequate.
17:37:17
pfdietz
Yes. There are games you can play with that, like "swarm testing", where you randomly prune down the set of production rules before each run of the test generator. Empirically this tends to find more bugs. I used this technique in the CL random test generator.
17:39:08
pfdietz
The experience with random testing is, I think, that different test generators cover different parts of bug space, so for a really large system you want diversity.
17:42:53
makomo
pfdietz: interesting, but you do arrive at the same "problem" of having to already have a working version of what you want to test, right? i.e. once you generate the inputs (sequence of operations, etc.) you need to somehow generate the outputs (the actual solution to those inputs)
17:43:30
pfdietz
You identify properties the code should have, even if they may not fully specify what the code should do.
17:43:59
beach
makomo: I can assure you that the "working version" is trivial to write in many cases.
17:44:24
pfdietz
For the Lisp testing I was doing, the property was that (for conforming CL) the code should do the same thing if compiled w. various different declarations, or if eval-ed.
17:46:38
makomo
beach: mhm, but it's interesting how there's a fundamental circularity/bootstrapping problem in there. either you have an existing implementation or you use your brain (which is just another implementation) and write out a finite number of tests yourself
17:47:30
pfdietz
If your code is loaded with assertions, each of them is an opportunity for testing. No assertions should fail.
17:49:01
makomo
pfdietz: true. how did you test that the two versions of code did the same thing? i suppose the stuff that was randomly generated were the declarations, not the code itself? the code was predetermined and you knew what to test for?
17:49:56
makomo
beach: yeah, i understand that. i get that it's a non-problem in practice, but fundamentally the ""problem"" is there
17:50:06
pfdietz
I randomly generated code and randomly generated inputs, and then checked that no errors were thrown (this was conforming code that would not throw errors) and that it had the same result on the same inputs.
17:50:07
beach
It takes less time to write the implementation of the trivial version than it takes to write the tests.
17:51:36
beach
makomo: I.e., that I think it is extremely hard to find abstractions for a testing framework.
17:51:49
makomo
beach: not quite. i just wanted to comment on the need of having to have an existing implementation in order to begin testing, no matter how trivial it is
17:59:11
beach
makomo: My main point here is that I don't believe in "testing frameworks" because I don't see how such a thing could capture even the most useful testing techniques that I know of.
18:18:58
shka_
but i like how i can use asdf extension to specify :test-file and i can simply use is and ok assertion
18:19:10
j`ey
this is my testing framework: (defun test (val) (cond ((eq val nil) (princ "E")) (t (princ "."))))
18:19:52
pfdietz
The question you want to ask is not "which testing framework?" but rather "what specifically do you want from a testing framework?"
18:27:33
beach
I am totally convince that this is one abstraction that everyone is looking for but that isn't possible in general.
18:28:49
beach
But the fact that there are so many and that people are not satisfied with what exists, tells me that there is a problem.
18:31:30
pfdietz
There are ten copies of RT in Quicklisp, last I checked. Not only are they easy to write, they're easy to copy. :)
18:39:42
jackdaniel
testing is not an easy task so many people have different ideas how the tool simplifying that should look like
18:43:29
jasom
beach: there are so many testing frameworks for the same reason I accidentally wrote a routing library for clack
18:44:06
jasom
beach: at first I didn't really need a routing library, so I didn't use one. Then I added a couple of features as I needed them. Then I realized that if I just moved it to a separate package, I had a complete routing library...
18:45:29
jasom
just like a testing framework it's easy to grow organically as you need more features *and* not much code when it's done: https://github.com/jasom/cl-fccs/blob/master/src/route.lisp
18:47:37
pjb
jackdaniel: right. IME, testing depends on the kind of software you are testing. Basically, most testing libraries are just unsuitable for a lot of software I had to test.
18:50:35
pjb
perhaps it would be more worthwhile to write down specifications, eg. of testing framework interfaces to testing report systems. eg. https://testanything.org
18:52:07
pjb
If asdf:test-op was specified to produce on *standard-output* a TAP stream, quicklisp could parse the results and produce nice reports…
18:53:14
pjb
You could use any testing framework, even multiple frameworks in a big system, as long as they would all follow a unique protocol, collected and interpreted by the surrounding tools and UI, it would be nice, integrated, and useful.
19:13:49
vsync
before I wanted to use it just because it was out of fashion and I'm a weirdo even among the weirdos
19:14:18
vsync
never had an actual use though... but lately I have found one where you want to have an inner named closure
19:16:39
vsync
though mine is for healthcare provider coordination data exchange between health systems and EMRs
19:17:07
vsync
for that purpose but generalizable! just like embedded in many other systems I'm sure, heh
19:18:33
pjb
vsync: the real use case for &aux, is when you write a macro and you need to add variables around a &body docstring-declarations-and-body <- NOT merely BODY!!!
19:19:14
pjb
I don't mind indentation. It's done automatically, and I've got a 5000+ pixel wide screen :-)
19:22:53
vsync
yeah I stumbled across it the other day when I didn't have CLOC installed and until my clean kernel compile happens the system crashes if I access a reiserfs and then a jfs
19:25:09
vsync
need to leave 3 extra characters for the minimum log graph view with diff, plus 1 for some terms wrap on the last column uselessly somewhy
19:25:37
stylewarning
vsync: sometimes a character in the last column is reserved for an overflow character
19:26:03
vsync
stylewarning: makes sense for consistency... like do you indent context lines in diff 1 char? I say yes
19:30:50
pjb
Well the max line length is not good. You'd have to look at the typical line length. My files contain outliers in comments or in data. Sometimes in code, but rarely.
20:12:48
makomo
ugh, i don't like it when i have to use composite keys for hash tables, it feels kinda clunky
21:15:32
aeth
Does anyone run CL on a processor with many cores? AMD has some 16 core / 32 thread and even (for servers) 32 core / 64 thread.
21:18:58
no-defun-allowed
I usually use lparallel and bordeaux-threads for stuff so I'm not paying much attention, sorry.
21:19:58
jasom
aeth: threads predate parallelism by a lot on PCs. In the 90s a lot of code was written multithreaded to avoid io blocking.
21:20:58
no-defun-allowed
cl-decentralise spawns a thread per connection since they actually do things in SBCL instead of splitting up one interpreter's time (cough CPython cough).
21:23:12
no-defun-allowed
cl-vep uses one generator and a lock for the generator. When a worker needs a new frame it holds the lock and funcalls the generator. This method (work "source" and lock) hasn't given me issues in the past.
21:23:52
no-defun-allowed
I've done high thread counts to avoid http blocking too. That method scaled pretty well up to 64 threads.
21:24:29
no-defun-allowed
However, the ffmpeg interface for cl-vep is shit slow and only puts out 4fps and only makes a load of 1.5.
21:24:41
jasom
aeth: I just spawn more worker threads. I haven't had any issue at 16 worker threads on my 8/16 ryzen
21:25:07
aeth
jasom: anyway, what I mean is things that are architectured so that they'd benefit from e.g. if AMD released a 64 core CPU tomorrow.
21:28:32
makomo
why is the syntax grammar for DEFSTRUCT so weird? i don't get what the keywords/singleton lists are supposed to do http://www.lispworks.com/documentation/lw70/CLHS/Body/m_defstr.htm
21:29:40
makomo
why would you want just :conc-name? is that supposed to construct (within the context of the grammar) a non-list version of (:conc-name conc-name)?
21:30:27
Bike
"A defstruct option can be either a keyword or a list of a keyword and arguments for that keyword; specifying the keyword by itself is equivalent to specifying a list consisting of the keyword and no arguments."
21:31:23
makomo
Bike: guess i should have read further, but it confused me right away so i didn't bother :c. thanks
21:33:42
jasom
aeth: lparallel has bounded queues implemented with vectors; I haven't checked the code, but it ought-not be consing if implemented right.
21:34:28
aeth
jasom: I'm guessing you'd queue or pass messages with things that are unboxed (like fixnums) or with keywords/symbols
21:35:27
jasom
aeth: setf on an array is going to be non-consing. Creating the value to setf in that locatin *might* be consing
21:36:28
jasom
aeth: I'm pretty sure an unspecialized array will not cons if you do something like (setf (aref foo) 3) since 3 is a fixnum
21:36:28
aeth
Would you sync with timers or would you have a thread sleep when done until it receives a start-working-again call?
21:44:49
aeth
Oh, there *is* a Threadripper 32 core (64 thread) CPU, the 2990WX. Wikipedia just put it in a separate chart much further down because it's Zen+.
21:46:55
aeth
So there is a chance that a (rich) Lisper is already running an application with performance on architecture close to manycore in mind (if you count the 64 threads via SMT)
21:47:37
jasom
aeth: Talos is available with 128T (or maybe even more) configurations since POWER8 is 4T per core
21:49:52
jasom
2x 22-core POWER9 is the highest configuration which translates to 196 threads I think?
21:49:54
aeth
Is PPC compatible with POWER8? I'm not sure if any CL runs on POWER8 if not. SBCL seems to have PPC and PPC64.
21:50:52
jasom
POWER9 is PPC64, but I think the talos is designed to run in little-endian mode, and I don't know if SBCL does that.
21:51:31
aeth
PPC64 is marked as in progress on the SBCL download page. I wouldn't be surprised if threading doesn't work
21:52:57
aeth
I wonder if there's a 2 socket AMD Threadripper mobo. That would be 2x 2x 32 at a max (right now... expect it to double next gen), so 128.
21:52:59
jasom
and the IBM E980 can support up to 16 12 core SMT8 processors for more cores than I care to do the math on
21:55:00
aeth
jasom: But yes, IBM's POWER is definitely worth mentioning in this kind of discussion. Last I checked, IBM seems to be keeping up with Intel, which is 10ish cores behind AMD.
21:56:39
aeth
A quick search has a reddit thread where the answer is "Threadripper is single socket only, for dual you'll need to go with Epyc." https://www.reddit.com/r/Amd/comments/6tqg88/dual_socket_threadripper_max_ram/dlmt8fn/
21:56:50
aeth
Of course, random reddit comments are even less reliable than random StackOverflow answers
21:57:48
aeth
(That's probably one of the ways they get people to pay 2x to 4x as much as the consumer line.)
22:17:08
Demosthenex
is there a way to start sbcl with swank using a port number from the command line? i'm trying to work on a few different things on a remote host, and i can connect with ssh/port forwarding. just starting sbcl with swank on separate ports is awkward
22:48:29
aeth
Wikipedia says POWER9 goes up to 24-core (96 threads via SMT4?) https://en.wikipedia.org/wiki/POWER9
22:53:12
aeth
It looks like both AMD and IBM are going to release 48-core CPUs very soon, though. It looks like Qualcomm already has a 48-core ARM server.
23:00:00
Bike
programs relying on particular details of the machine they're running on is something to be avoided. surely this is a basic principle
23:03:27
aeth
Bike: The architecture can also run on an 4 core machine, yes, but there's less of a motivation for it there.
23:05:29
aeth
Bike: But consider e.g. a game or similar application that can be divided into several systems. In today's world, if it's parallel at all, each major system would probably get its own thread. That'll only scale up to the number of major systems, which is probably far less than 90.
23:07:22
Bike
in that case whether you have five or ninety processes isn't important for basic architecture.
23:08:18
aeth
Except for the most part, afaik what you'd typically see when the end user has quad core is maybe putting one task on each a thread to be "parallel", not also subdividing those tasks.