freenode/#lisp - IRC Chatlog
Search
22:36:00
elderK
Man, I'm having trouble wrapping my head around what I need to be available at compile time, load time and execute time :|
22:36:27
|3b|
ACTION thinks it usually isn't worth bothering, since those times tend to overlap too much
22:37:04
elderK
Like, I've written a bunch of stuff. And, most of my macros expand into stuff that doesn't have an eval-when. Because, well, it's not necessary.
22:37:43
elderK
But for another macro, my define-binary-structure, I need a bunch of that information available at compile-time.
22:40:33
jcowan
elderK: If a function is called by a macro, wrap it in triple eval-when; otherwiser not.
22:41:22
elderK
The thing is, like, so far at least, all these "type metathings" are being added to a hash table at runtime, I guess.
22:42:03
elderK
It would be smarter to persist that information, if you say, save your stuff to a FASL.
22:46:07
|3b|
it isn't quite like a .o file in C, since loading it still has all the side effects you would get from LOADing the original .lisp file
22:46:24
elderK
And, like, let's say you ARE doing a bunch of stuff at runtime that's costing a lot of time. Now for a binary-types type library, a lot of the types a user defines will be known when they compile their program. That information could be persisted, and somehow... thawed on load, without having to go through re-executing all the stuff to recreate hte metadata. If you /did/ wanna persist it, so that loading was
22:46:41
|3b|
and it can store some fairly complex data structures, which requires more effort than just mmapping it into memory
22:49:54
|3b|
probably depends on specifics, maybe just a variable in .fasl, maybe in a separate file
22:51:33
|3b|
for example in the case of binary parsers, it would be complicated since you'd probably have data from a bunch of different files
22:52:37
|3b|
but maybe you could do something like caching them per thing that uses it in a way that was quick to merge into the main data set
22:58:51
elderK
:| Man. What started off as something so simple (define-binary-structure), I've wond up spending like, four hours just thinking...
23:04:17
jcowan
Is there any straightforward way to load the whole of Quicklisp so it can be searched?
23:05:44
pfdietz
The problem is that various systems in quicklisp are incompatible, due to package name collisions.
23:06:16
elderK
|3b|: This is my first "real thing" other than, like, the school assignments I was redoing.
23:06:35
elderK
|3b|: I don't have any ambitious things in my little binary library, just the same stuff that's in binary-types really.
23:06:55
pfdietz
I have 'kind of' loaded all of quicklisp, but only in a quick and dirty way to get stuff that looks like lisp for mutation testing purposes. This doesn't have to really read the lisp, and bails out if it encounters any reader errors.
23:07:00
elderK
Just, it's my one. And it has a few things different that are more suited for what I want to do, and how I Want to use it
23:08:27
pfdietz
I think it should be possible to do better, using Eclector and being smarter about things. But in general reading a lisp file may depend on setting up arbitrary lisp capabilities for the reader macros.
23:09:01
Xach
I think the way to go is to load each in turn and use sbcl's introspection to dump a searchable offline database.
23:09:27
|3b|
elderK: yeah, just a potentially complicated task if you want to handle all the odd things various binary formats do
23:09:47
pfdietz
It was pretty hacky. I had to be able to handle undefined packages, so I hacked the SBCL reader to just use *package* when it found a package it couldn't recognize.
23:11:13
pfdietz
I put in a ticket to enhance the SBCL reader so there were some useful restarts for the package not found error, but that hasn't happened yet.
23:12:23
Xach
I've mentioned it before, I'd really like a query system for the image so you could ask things like "show me all functions with two arguments" or "show me all functions with an argument named DOG" or other things (not necessarily only about function argument lists!)
23:13:14
Xach
i think the advantage of load + dump is you don't have to try to fake loading and work around reader macros and package problems.
23:15:39
Xach
the stuff needed to support M-. and who-calls and who-references could be fun to query arbitrarily
23:22:52
fiddlerwoaroof
elderK: I wrote a little thing that's sort of like what you're doing, if I understand correctly
23:25:22
fiddlerwoaroof
The usage looks like this: https://gist.github.com/fiddlerwoaroof/dc1cfe4738a9d82d48bce6046af53fbe#file-zipfile-lisp-L113
23:25:39
fiddlerwoaroof
Here's how I define the format: https://gist.github.com/fiddlerwoaroof/dc1cfe4738a9d82d48bce6046af53fbe#file-zipfile-lisp-L81
23:25:59
fiddlerwoaroof
And here's the entrypoint of the parser: https://github.com/fiddlerwoaroof/fwoar.lisputils/blob/master/bin-parser.lisp#L78
23:48:06
jcowan
Of course the code has long since rotted and was never Maclisp compatible in the first place, but it shows what could be done by introspection
23:50:32
jcowan
http://www.softwarepreservation.org/projects/LISP/interlisp-d/3100186-Interlisp_Oct83.pdf pp. 351-372 (of the PDF)
0:35:33
fiddlerwoaroof
elderK: if I had to guess, you're not properly escaping # when you call xdg-open
0:41:08
jcowan
I'm still trying to get a list of all quicklisp projects. (ql:system-apropos "") returns only 73; using quickdocs with an empty search field returns only 0. What's the righteous way to do this?
0:41:50
fiddlerwoaroof
git clone https://github.com/quicklisp/quicklisp-projects && ls quicklisp-projects/projects | wc -l
0:45:57
fiddlerwoaroof
I was wrong anyways, it looks like source-disabled.txt is used in cases where there is an old repository
0:52:55
fiddlerwoaroof
Hmm, that's going to be a bit tricky to do precisely, without reading in the source
0:57:43
pfdietz
(length (remove-duplicates (mapcar #'ql-dist:release (ql-dist::provided-systems (ql::dist "quicklisp"))))) ==> 1682
1:09:50
jcowan
okay, created a shell script that repeatedly calls sbcl --disable-debugger --eval '(unwind-protect (progn <import ql> <load project>) (quit)'
1:18:22
pfdietz
Goodness. ql-dist::dependency-tree can produce a very large tree for some systems. Need a dependency-dag.
1:31:20
pfdietz
There are just 344 systems in the current ql dist that are not in the required-systems list of another system.
1:48:42
aeth
pfdietz: It might be better to use projects as the basic unit and see if there are any projects where every contained system in that project is not depended on by any external project's systems.
1:58:27
fiddlerwoaroof
I believe ql-dist::provided-releases gives you all of the quickloadable projects
2:29:48
elderK
Guys, what is a good way to like, make sure that ... well, all stuff evals when it is meant to?
2:42:22
elderK
I see that you can for cons, but defining a list of a certain type that way seems kind of painful
2:53:46
elderK
I'm not even sure if I could like, write a nice type for a list of certain stuff - it would be recursive :|
2:56:44
fiddlerwoaroof
(defun all-strings (ss) (every 'stringp ss)) (the my-list (satisfies all-strings)
3:44:12
elderK
fiddlerwoaroof: :) So, what's a good way to check that like, all my stuff... uh, works when it should? :D
3:51:43
fiddlerwoaroof
the rule is fairly simple, though: every function a macro calls had to either be in a different file or in an eval-when
3:54:18
elderK
fiddlerwoaroof: It's just, well, I need to test that all the metadata I've gathered is availalbe at the appropriate time.
4:06:06
elderK
fiddlerwoaroof: :P Everything loads okay even if the things the macro calls, are not (:compile-toplevel)
4:06:22
elderK
:P I even forced the macro expansion in a :compile-toplevel eval-when, it still woriks.
4:08:38
elderK
fiddlerwoaroof: I'm not sure how to test this. I mean, asdf:load-system from REPL works just fine, even if I kill the REPL and start fresh
4:09:23
fiddlerwoaroof
if you compile each function as you write it, they will all be available to be called: to really test this, you either need to use a fresh lisp process or delete-package on the package containing your definitions before loading the sustem
4:10:18
fiddlerwoaroof
forming a mental model of file loading was the trickiest part of learning CL for me
4:10:21
elderK
fiddlerwoaroof: If I start a fresh SBCL, I can successfully load my system using asdf:load-system, :force t
4:10:50
elderK
fiddlerwoaroof: The thing is, it doens't make sense that this succeeds. I've purposely made some stuff not available at compile-time. The macro should fail, but it succeeds all the same.
4:17:10
elderK
Okay. So, if I am populating a hash table in one compilation unit, does that mean all of that data is visible to the next, at compile time?
4:17:23
elderK
I haven't like, got any eval-when to say "Hey, I want this stuff available at compile-time."
4:18:20
elderK
Well, yes. But I need to verify that the types people are using for slots in a "binary structure" are actually "binary types."
4:19:01
elderK
Since populating that hash table is a runtime thing, at least, atm. It'll generate code to do that, rather than do it at compile-time.
4:19:23
fiddlerwoaroof
but, im pretty sure that once a file has been compiled and loaded, you can rely on the side effects have being run
4:44:50
elderK
(Starting fresh CCL process, requiring ASDF, then load-systeming my system, ensuring it is built)
5:02:54
elderK
:P I removed all the eval-whens from my code, until I know I need them. I'm sure something will go wrong eventually and I'll need them :P
5:53:00
aeth
elderK: Easiest way to remove eval-whens is to put them in a separate file that's loaded first
5:53:32
aeth
elderK: Afaik you probably don't need an eval-when unless it's called unquoted in a defmacro (directly or indirectly, I guess) while being in the same file as that defmacro
5:54:12
aeth
Sometimes CCL needs eval-whens when no other implementation needs them. I think that's for constants used in macros
6:46:31
Necktwi
finally got rid of those ghost windows they are due to # to ## redirects of some of my autojoin channels
6:58:50
elderK
aeth: That's the thing: The macro is using functions defined in the same file, and is not encountering any issues on either SBCL or CCL.
7:00:24
elderK
Still seems weird. The macro makes several function calls, these functions generate the expansion. These helper functions call other functions, to get information that has been accumulated in a hash table. It seems bizarre that I am not encountering problems.
7:57:10
beach
elderK: One thing that can bite you is that you have already loaded your system before, so the functions are defined. Then, when you recompile, no warnings will be issued. To check that you have no such problem, it is good to compile from a fresh instance of your Common Lisp implementation.
7:58:15
aeth
That's basically the only time my build can fail, if I was referring to stale functions (i.e. an old version pre-rename), or if I have a circular dependency, etc., that the image won't expose unless it's fresh
7:58:35
beach
elderK: It is unfortunate that one has to restart the system to do that, of course. In SICL, I plan to use a fresh first-class global environment for this purpose without having to restart the system. Current Common Lisp implementation have a single global environment, so they can't do that.
8:07:59
beach
Sure, if someone makes a list of functions and put the list in a special variable, it will be around.
8:09:12
ggole
Perhaps I'm misunderstanding and the new environment won't include anything which can reference old bindings in that way.
8:09:23
beach
All I am saying here is that I can start the compilation from a fresh first-class global environment. This new environment will not contain any old definitions, so it will be obvious if a macro is referring to a function that does not exist at compile time. The compilation will generate a FASL as usual and that FASL can be loaded into the working environment (as opposed to a fresh one).
8:10:16
beach
Exactly. The new environment can be a pristine one, like when the Common Lisp system starts up.
8:13:42
aeth
at the moment you have to M-x slime-restart-inferior-lisp before pushing to git if you want to be polite and catch various stale environment bugs
8:14:44
beach
ggole: Sure. I keep thinking in terms of a Lisp operating system. Starting a fresh Lisp is to me like booting up a second version of your operating system. If you have things like editor buffers and such defined, they will not be present in the fresh system.
8:14:53
aeth
ggole: If you had first class environments, then *building* in a fresh environment will catch anything that you'd catch with a fresh Lisp image starting up, without having to kill the old running things
8:15:40
aeth
ggole: So you could run your IRC client and develop in the same Lisp image, and not have to bring your IRC client down when you want to test a fresh build/load to see if you had stale function bugs
8:17:03
aeth
ggole: of course the way to do it right now is to just run one Lisp per application and you know use like 100-1GB more RAM than you need to
8:17:31
beach
MichaelRaskin: Again, I am sorry for being dense. I am notorious for not understanding what other people refer to. I guess I am unable to fill in the blanks.
8:17:35
MichaelRaskin
I thought that creating a new global environment would probably put them into this environment in a more efficient way than starting a new image
8:18:15
MichaelRaskin
Rest assured that I _know_ you ask in good faith, and I understand the value of making sure about the intentions.
8:48:44
no-defun-allowed
has anyone ever used cl's runtime compilation to write a JIT compiling emulator?
8:53:11
no-defun-allowed
gonna write a chip8 one for the hell of it, but i think at the very least i can (ab)use (compile nil _) to generate thunks for each instruction
8:55:29
no-defun-allowed
also, how can i tell when to jit? should i do it when the machine jumps into new code or wait a bit?
8:58:13
ggole
You could do either - usually a fair amount of code is cold, so it makes sense to avoid compiling it
9:07:10
jackdaniel
no-defun-allowed: sjl wrote chip8 emulator with a fantastic blog post series explaning how it is made