freenode/#lisp - IRC Chatlog
Search
12:29:05
adlai
beach: STYLE-WARNING: definition 4.1 ("simple instance") is not referred to by the rest of the document.
13:23:22
|3b|
looks like either something wrong with your quicklisp, or something wrong with ql infrastructure fetching github releases, since you seem to have iolib 0.8.1, and 0.8.2 claims to fix problems with asdf 3.3 (and newest is 0.8.3, over a year old)
13:24:33
|3b|
or something odd with your asdf confifguration, does ~/quicklisp/dists/quicklisp/software/iolib-v0.8.3/ (or .2) exist?
13:27:48
|3b|
looks like fetching things by github release seems to be working, so probably not ql server-side, i'd guess overly broad asdf search path
13:29:55
pjb
Unfortunately, for ecl, it looks like I have to do that, since quicklisp prevents it to save executables…
13:30:52
|3b|
doesn't ql have some way of dumping a list of systems for building executables without ql?
13:32:32
jackdaniel
(my life would be certainly happier, if ASDF had similar funcionality: dump compilation instructions to file so it may be loaded without ASDF in the image to build a binary)
13:55:23
beach
Wow, that's vicious. Together with ASLR, pretty soon they are going to prevent us from writing decent Common Lisp implementations.
13:57:31
pfdietz
OpenBSD support in SBCL is already decayed; this will just mean the platform gets written off entirely.
13:58:57
shka_
what about julia? or any other dynamic language built on top of llvm for that matter?
14:01:55
beach
Note to self: In the CLOSOS specification, add W^X as an example (together with ASLR) of a kludge on top of a hack in order to make a broken model (i.e. UNIX) a bit safer, but also much harder to use.
14:03:44
pfdietz
What's the overhead for changing a page? Also, if you are compiling in small chunks, will you waste a lot of space in pages that need to be executable before they are filled?
14:03:57
|3b|
beach: yeah, possibly the status quo will change if openbsd users care enough to patch implementations when it is
14:04:04
ggole
It's iOS that is problematic here, as it doesn't allow user programs to change map permissions.
14:05:19
ggole
If you compile in small chunks you could either use a new page or do make writable -> write the code -> make executable
14:07:31
jackdaniel
ECL won't be affected by this either, because it either works with bytecodes or with modules load from disk (with dlopen), so there is no code segment write whatsoever
14:08:58
pfdietz
For people writing CL, I suggest they do not depend on things defined in foo.lisp in order to compile things later in foo.lisp. Stick your macros and class definitions in another file.
14:10:21
ggole
There's another cute trick that might be useful: double map the pages, with one mapping writable and the other executable
14:11:44
beach
It sounds to me like W^X is so easy to circumvent that it won't fix the problem it is intended to solve, but it will make life harder for compiler writers.
14:14:59
ggole
The reasoning, I suppose, is that the solution - ditching C - would not be acceptable. Thus they do what they can.
14:41:07
jcowan
beach: gcc generates trampolines in certain conditions, so I don't think you have to worry about being completely locked down. Furthermore, a strict interpretation of W^X would make JIT impossible, and Java depends on it.
15:04:48
jackdaniel
so here's how they implemented JIT in Firefox: https://jandemooij.nl/blog/2015/12/29/wx-jit-code-enabled-in-firefox/ . It is not that you can't write to the same pages which you execute, but rather permissions are either write or execute
15:04:52
MichaelRaskin
Double-mapping will probably not work; re-protecting memory as as RW -> RO -> RX seems to be the official recommendation and very unlikely not to be supportred. The problem they mitigate is overwriting already-executable page by buffer overflow.
15:05:39
jackdaniel
but in this scenario it is an explicit action taken by the environment (not accidental thing)
15:05:51
MichaelRaskin
Note that strict W^X without run-time protection level chage will break userspace dynamic loader
15:17:31
White_Flame
jackdaniel: "temporarily mark page RW" might be impossible in the future. allocate->RW->RO->RX->deallocate might be a one-way path
15:26:20
White_Flame
well, any time you want to update a JIT compilation, you'd have to allocate a new page with its own permissions
15:27:29
White_Flame
oh, interfere patching, correct. It'd have to effectively be manual copy-on-write
17:29:22
sjl
no-defun-allowed: jackdaniel: cl-6502's 25-line JIT is also worth reading as inspiration
18:48:59
shka_
it is modular, neatly designed, i never figured out how to write my own modules (but i didn't have to)
18:50:00
p_l
I recently did stuff in rails 5, and some of the stuff is still some of the better tooling I have ever used :O
18:50:50
p_l
But I do prefer lisp, so I'm thinking of looking what's the current landscape and either using something existing or using some lower layer stuff
18:52:47
p_l
Which is very minimal, good-for-microservice kind of thing (Sinatra and thus Caveman, that is)
20:25:41
makomo
hm, say there's a function F within file A and a macro M which uses F within file B. further, usages of the macro M appear within file C. assuming no EVAL-WHENs, in order to compile file C it has to hold that (1) we compiled file B and (2) we compiled and loaded file A. is this correct?
20:28:15
makomo
if the above is correct, does that mean we lose the possibility of compiling A, B and C in isolation (since now we need to actually *load* A to be able to compile C)?
20:30:52
makomo
is that dependency a bad thing? should one then wrap every DEFUN that's used at compile-time into an EVAL-WHEN? what about external system though? i guess we depend on them being loaded anyway so it's fine?
20:31:49
makomo
... external systems (which we don't control and can't change and wrap their DEFUNs)* ...
20:32:51
makomo
i'm mostly wondering whether the fact that we have such "load-before-compilation" dependencies is bad or not
20:33:15
makomo
jackdaniel: do we have to load B though? won't the macro be defined when B is compiled?
20:34:10
shka_
makomo: what jackdaniel's is saying i think is that you will need your function definition ONLY when you need it (during macroexpand)
20:34:51
makomo
jackdaniel: well i peeked at the expansion of SBCL's DEFMACRO and it is defined at compile-time
20:34:57
makomo
also, "If a defmacro form appears as a top level form, the compiler must store the macro definition at compile time, so that occurrences of the macro later on in the file can be expanded correctly."
20:37:22
elderK
Hey guys, quick question: When does it make sense to use a symbol's plist vs. an external hash table?
20:37:56
elderK
In my situation, I have a hash table that stores information about types and things. The symbol denoting hte type's name is the key of the hash table.
20:38:17
elderK
I was wondering if attaching type information, a structure instance, as a property on the types I define instead, would be better.
20:38:39
makomo
jackdaniel: alright, so only A's left. so is this "loaded-before-compilation" dependency a bad thing?
20:39:38
shka_
secondly, it they would be just symbols, defgeneric and defmethod with eql specializer would be probably a better option
20:41:19
makomo
jackdaniel: so assuming :serial t, not only does ASDF load them in that order, but it also compiles them in such a way that before compiling the file N, files 1, ..., N-1 have been loaded?
20:42:21
elderK
shka_: I'll have to come back later and describe it in more detail. As it is, I have a hash table mapping symbols to type metadata.
20:42:35
jackdaniel
makomo: that depends on what you tell asdf to do, it may only load them, or only compile them. my impression is that asdf sometimes does a random thing, but I didn't find any passage with (funcall (nth *funs*) (random 8)), so maybe I do not understand whole thing ,)
20:42:48
elderK
shka_: It's not the symbols I care about, it's the associated type information I store relating to them.
20:43:04
jackdaniel
but more seriously: asdf constructs a plan which is more complicated than simple load/compile
20:44:08
elderK
shka_: That isn't sufficient here :P The type metadata contains information I need :P
20:44:11
jackdaniel
but in principle it works like: if A depends on B, load B (or its compilation result) before A
20:45:19
makomo
jackdaniel: lol :D. right, it depends on what i want to do. whether ASDF will compile and then load the fasls if i tell it to LOAD-SYSTEM is a separate thing, so let's assume i'm doing COMPILE-SYSTEM
20:45:23
jackdaniel
makomo: given you are groking style issues, did you make some time to read about splicing?
20:46:37
elderK
shka_: I'm afraid I have to run. Please take a look at the linked binary-types thing to get a feel for well, what I'm talking about :D
20:47:02
jackdaniel
I don't know ASDF enough to describe without looking at source code what it does exactly when you compile-system
20:47:22
makomo
jackdaniel: not yet. sorry if it seems like i'm intentionally avoiding it since i'm asking all these questions, but i'm procrastinating and should be studying. however, asking these questions is different as i know that if i start analyzing that document, i'll waste too much time. :-)
20:47:28
jackdaniel
but I would suspect, that if there is dependency between two files, dependee is loaded before depending file is compiled
20:47:57
makomo
jackdaniel: my first exam is tomorrow and the whole week is packed with them. i'll get to it on the next monday the latest!
20:49:24
jackdaniel
I think you overestimate time needed to read it, but I understand that exams are imminent
20:50:19
makomo
jackdaniel: oh, i did read it (quickly though), but actually writing an anylsis will take more time (i.e. (re)searching the standard and the like)
20:51:47
jackdaniel
if you create binary on ecl on the other hand some dependencies which were available during compilation are not part of the resulting binary, and sometimes such binaries fail (that happens for instance if dependency on uiop is not expressed in asd file)
20:55:51
makomo
jackdaniel: so basically there's a separate compilation and run-time environment. wouldn't the same happen if you restarted SBCL and tried to load a FASL which uses uiop (before actually loading uiop)?
21:08:51
jackdaniel
drawing benchmarking application sketch: http://hellsgate.pl/files/c7111345-bench.gif
21:10:15
pjb
minion: memo for elderK: notice that p-lists are faster than hash-table when they have fewer than 5-35 elements, depending on the implementation. Also, hash-tables have a size overhead that is way bigger than p-list. So if you need to put just one or two entries on thousands or millions of symbols, better use the symbol-plist. On the other ahnd, if you need to put thousands of entries on a few symbols, then better hash-tables.
21:12:18
pjb
minion: memo for elderK: In both cases, if you attach attributes to symbols at compilation time, you must ensure that data is passed over to run-time if you need it at run-time. basically: (defmacro x (s) (setf (getf (symbol-plist s) 'your-attribute) 'your-value) `(progn (setf (getf (symbol-plist ',s) 'your-attribute) 'your-value) … ))
21:23:48
aeth
pjb: Using plists for an unspecified small number of elements just because they're slightly faster is a micro-optimization that this channel normally criticizes.
21:25:59
aeth
I did this: (defparameter *foo* (list :foo 42 :bar 43 :baz 44)) (defparameter *bar* (zr:hash-table* :foo 42 :bar 43 :baz 44)) (time (dotimes (i 1000000) (getf *foo* :baz))) (time (dotimes (i 1000000) (gethash :baz *bar*)))
21:27:02
aeth
I keep adding zeros and sometimes the hash table is slower (up to 4x in one case) but they're usually about the same.
21:29:26
aeth
So 25% speed improvement in niche cases for something that either way takes about as much time as +
21:35:10
aeth
SBCL doesn't appear to, but either way, I allocated one tiny object for each and was just accessing it.
21:36:07
aeth
The memory argument is a bit mixed. Conses will probably take up the least memory of a data structure for things like this, but they might not be together in memory, which is probably more important in today's software than being tiny overall.
21:36:48
aeth
fiddlerwoaroof: alist will add a bit more overhead, I think. At least, if you don't test that your plist is a proper one.
21:37:22
pjb
aeth: notice that it also depends a lot on the type of keys, and the hash function used by the hash-table.
21:40:04
aeth
plist is [key|-][value|-][key|-][value|...] and alist is [pointer-to-key-and-value-cons|-][pointer-to-key-and-value-cons|-][pointer-to-key-and-value-cons|...]
21:40:59
pjb
(print-conses '((a . 1) (b . 2))) #| ((a . 1 ) . ((b . 2 ) . ())) --> ((a . 1) (b . 2)) |#
21:44:36
aeth
Also, plists will iterate by cddr and then check car and alists will iterate by cdr and then check caar, so that's another difference.
22:06:00
aeth
pjb: Yes. I'm guessing the typical operations are (1) iterate over the entire alist/plist or (2) iterating over the alist/plist until you find a key that matches
22:07:02
aeth
pjb: so cddr iteration with car [and cadr], or cdr iteration with caar [and cdar] where the parts in brackets are only needed if either you're in case #1 or you matched in case #2
22:07:58
aeth
(Of course you'd use named accessors and I'd probably use destructuring-bind, rather than actually call those cxr functions)
22:11:25
fiddlerwoaroof
I've also thought that you could dynamically optimize an alist or plist by always prepending the last entry you looked up
22:27:34
aeth
Remember, the hash table is only 25% slower and now you're slowing down the lookup of the alist/plist.
22:57:00
aeth
Here's a fun little program to see the day's time from a different perspective. (defun countdown () (multiple-value-bind (second minute hour) (get-decoded-time) (format t "~2,'0D:~2,'0D:~2,'0D" (- 23 hour) (- 59 minute) (- 59 second))))
23:06:21
fiddlerwoaroof
Think about it this way: there's a certain number N of elements where the O(n) lookup of an alist becomes slower than the O(1) lookup of a hash-table. If you have a dataset where < N elements constitute the majority of accesses, then an alist that begins with those elements will generally be faster than a hash-table.
23:07:01
White_Flame
I think the only speedup would be if you read the head multiple times in succession. Even if you're reading just the 2nd entry of the list, you still have to shuffle or cons to shift a new head
23:08:00
fiddlerwoaroof
Well, my guess is the best way to do it would be to have a "training" period where you move accessed elements to the head and then freeze the list
23:08:40
fiddlerwoaroof
You'd probably also avoid reshuffling the list if the access is to one of the first N elements
23:19:38
pjb
Ok, in this lisp image, there are 169 symbol-value that are hash-tables. Of those, 133 have only 1 entry.
23:28:43
aeth
pjb: when you're talking about comparing two operations that are approximately as fast as +, it's still a micro-optimization
23:34:47
pjb
Here are the types of the first key in those 196 hashtables: (character (simple-base-string 45) standard-generic-function ccl:compiled-lexical-closure function (simple-base-string 2) (simple-base-string 5) standard-class (simple-base-string 11) (simple-base-string 6) hash-table pathname ccl:process cons (simple-base-string 17) (simple-base-string 22) bit readtable (simple-base-string 21) (simple-base-string 13) (simple-base-string
23:34:47
pjb
keyword nil (integer 0 1152921504606846975) symbol (simple-base-string 4) (simple-base-string 29))
23:55:46
scymtym
dim: best to ask in #clim. what i found is that i have to (setf mcclim-truetype::*truetype-font-path* #P"/usr/share/fonts/truetype/**/") after loading mcclim, but before doing anything with it. then i can use truetype fonts via (clim:make-text-style "EB Garamond Initials" "Regular" 24) where the "Regular" face must exist in the font
23:56:25
scymtym
when the *TRUETYPE-FONT-PATH* is properly set, the "Font Selector" demo should show all fonts and their respective faces