freenode/#lisp - IRC Chatlog
Search
6:23:56
beach
So here is an idea for the implementation of thread-aware breakpoints in x86. The important feature is that threads that are not being debugged should not take too big a performance hit.
6:24:00
beach
I suggest that, when code is compiled with a high value of the DEBUG quality, then the execution of each form starts and ends with a test of the DF status flag. If 0, then execution continues as normal.
6:24:01
beach
If 1, then some further action is taken to determine whether there is a breakpoint at this point in the code. The "further action" remains to be specified, but the slow version would be to interrogate a breakpoint table in the thread instance.
6:25:32
beach
The use of the DF flag means that, if the implementation uses specific instructions that depend on its value, then the value has to be saved and restored around the execution of these instructions.
6:33:16
rme
when I last looked into using the x86 direction flag, I found that manipulating it was expensive. it ended up being faster to set a flag in memory.
6:33:49
White_Flame
I'm kind of preferential to Java-style interruption, where an address is routinely read. If it's to be interrupted, the address range is made unreable. Now, you could extend this so each thread is reading a different address (just an indirection through a thread-local pointer), and/or use different addresses per potential breakpoint.
6:34:15
White_Flame
I think you have to push flags to stack to read the DF flag, only set & clear are part of the ISA
6:35:35
White_Flame
hmm, if you have ample room in your 64-bit address space, you could read thread_local_page + N*program_counter to get a unique memory-protection based interrupt per location in your code
6:36:45
White_Flame
and with future fixes for meltdown/crispr, it might even be quicker to interrupt than it is now ;)
6:37:09
rme
I'd want to look at trying to use the dr0 through dr3 debug registers on x86 for setting hardware breakpoints if possible
6:39:34
beach
rme: I haven't had the time to read up on hardware breakpoints, but that's an important subject.
6:43:30
beach
I still like the DF over an address in memory, because that address would have to depend on the thread, making it expensive to read because of indirection. But here is a compromise: at the entry of each function compiled with a high value of the DEBUG quality, move the DF flag to a register or to the local stack frame so that each further test inside the function is cheaper.
6:46:07
beach
Some simple hash-like technique could be used. Take the PC modulo (say) 256 and consult a table in the thread instance. If a bit is set there, then consult a complete hash table to determine whether this PC value actually has a breakpoint.
6:47:11
White_Flame
I suspect the Java way is one of the least intrusive, though it probably does tie up a cache line
6:47:59
rme
there's almost certainly going to be thread-local memory maintained by the lisp runtime. on ccl, %fs or %gs points to the thread-local memory block (except on highly advanced systems like macOS, where we have to burn a gpr for this).
6:48:23
White_Flame
beach: each thread reads its own address; you only deny read permissions for the memory page for one thread, in my idea
6:49:53
White_Flame
basically, each thread just reads a canary location in memory. When you want to halt a thread, play with the memory manager for a particular canary location
6:52:42
White_Flame
it's just a read through a register+offset indirection, assuming a thread-local table is in a register
6:55:11
beach
So, either way, we have at least two ways of making thread-aware breakpoints with very little cost for threads that are not being debugged, and a modest cost for a thread being debugged, but not at this particular value of PC.
6:56:11
White_Flame
"safepoints" is seemingly the jvmterm: http://blog.ragozin.info/2012/10/safepoints-in-hotspot-jvm.html
6:56:56
White_Flame
If you did a read of thread-local-canary-address + 4k*PC, then maybe you could use the same trick to interrupt at a particular PC on a particular thread
6:59:13
White_Flame
if I remember correctly, the width of the visible universe in terms of electron-diameters is on the order of 2^80
6:59:18
beach
It is not important to me at this point to know the exact solution; just that one exists. And I am now convinced that there are at least two.
7:01:15
jackdaniel
someone told that god had to be a very cruel at the time he made men – give them mind with capabilities to make almost infinitely ambitious plans and time to live which doesn't allow to excercise them :)
7:21:55
beach
White_Flame: Wait, your single per-thread address read suffers a severe performance problem when the potential breakpoint turns out not to be one.
7:23:14
beach
Here is a thread being debugged, so I arrive at some point in the code and I read my unique address.
7:23:17
White_Flame
but yeah, switching the MMU stuff to reject he read and flushing TLB cache or whatever is a pretty expensive operation, but that's all involved when you actually perform the interrupt, not exploration
7:23:44
White_Flame
ah, right, if it's just a per-thread address, and not per-thread address + 4k*pc
7:25:16
White_Flame
there's all sorts of granularity tradeoffs you could make, like assign a page offset per function, so you only hammer it when it's in the particular function the breakpoint is in
7:25:43
White_Flame
but yeah, if you're not going to have it per-PC, then it might make more sense to do manual checks, if you can do them cheaply
7:26:17
White_Flame
random thought, but are there any feature of cpu affinity that could help isolate the thread?
7:27:50
White_Flame
but really, how expensive is it to set a breakpoint, and test to see if you're in the intended thread?
7:28:47
beach
OK, final proposal that does not involve DF: At the entry of each function, read an address at a fixed offset in the thread instance. It contains a flag that indicates whether this thread is debugged.
7:28:53
beach
The contents is put in the stack frame or in a register, determined by the register allocator. Before and after the evaluation of each form, call a small local routine with PC as its argument.
7:28:59
beach
That routine starts by testing the flag. If 0 it returns normally (this is the case when the thread is not being debugged). If 1, it takes PC modulo (say) 1024 and consults a bit table in the thread instance.
7:29:08
beach
If the table contains a 0 (the usual case) then it returns normally, because it means there is no breakpoint at this PC value. If 1, it consults a hash table in the thread instance to see whether this particular PC value has a breakpoint.
7:29:09
beach
If not, it returns normally. If it does, it suspends execution and gives control to the debugger thread.
7:31:29
beach
White_Flame: I don't know the answer to your question, but this last solution of mine would work on any architecture.
7:33:19
White_Flame
hmm, it would only work for a single breakpoint, but if you could do 1/(PC - read-from-thread-local), you could trigger a divide by zero exception if that location was set to the current PC
7:34:30
beach
When the debugger sets a breakpoint, it updates the hash table and the bit table in the thread instance. When a breakpoint is being removed, it clears the bit table, traverses the hash table to set the bits in the bit table.
7:35:45
White_Flame
hmm, if you double the memory footprint of your code, you could have a flag per instuction byte
7:36:09
White_Flame
then it's a large PC-relative offset to see if that flag matches your current thread pointer
7:36:51
phoe
I don't think he wants per-instruction breakpoints though, rather one breakpoint per Lisp form.
7:40:03
White_Flame
it's nice living in the future where you can throw egregious amounts of RAM at a problem
7:44:02
beach
Hmm. If I make the bit table smaller, say 64 bits, I can skip the single bit and load this small table to the stack or to a register (subject to register allocation). The famous small routine would then immediately test a bit that is PC modulo 64. How much more expensive would that be compared to testing one single bit? I am guessing "not much".
7:46:03
beach
Maybe not. The debugger would have to know how to update the local version of the bit table. Not worth it, probably.
7:49:59
shka
White_Flame: it would be nice, if people would not throw copious amounts on RAM into web based application
7:52:58
White_Flame
but I think a good post-IDA disassembler (my big web-based application) tends to be for people willing to have big hardware footprints
8:52:46
beach
A summary of my own conclusions of the discussion on breakpoints is now in Appendix A of this document: http://metamodular.com/clordane.pdf
8:53:21
beach
I will submit it to nyef`` when he wakes up, because we have discusses such things in the past.
8:56:09
beach
Interesting related information: When I run SICL inside a host Common Lisp implementation (currently SBCL), I can implement breakpoints this way with very little effort. I can then implement Clordane so that it can debug SICL programs, even though 1. SICL doesn't really exist, and 2. SBCL doesn't support per-thread breakpoints.
8:58:29
beach
Compare this idea to generating a SICL executable sooner rather than later. Then I would lose all my usual tools and I would have to resort to GDB or something similar with absolutely no knowledge of Common Lisp.
8:58:41
schweers
beach: So its possible to add better debuggers to existing implementations without modifying said implementation?
8:59:25
beach
Not really. I am not improving the processing of native SBCL code. Only code compiled with the SICL compiler for execution inside SBCL.
9:03:51
beach
I am basically accumulating evidence that I should not be hasty when it comes to creating a native SICL executable, and even less so when it comes to creating a bootable LispOS image. Instead, I should take advantage of a safe execution environment (here SBCL) to make the system as complete as possible first.
9:23:25
beach
Strange! Most of the code exists, but I am making very slow progress with the bootstrapping phase.
9:27:07
schweers
beach: out of curiosity: what exactly does sicl do? does it compile to native code or something else?
9:28:08
beach
It will when it's done. Right now, it generates AST then HIR. The HIR code is then turned into a small subset of Common Lisp that is compiled on the host compiler for ultimate execution.
9:29:00
beach
And the Common Lisp that is submitted to the host compiler uses first-class global environments for all its lookups.
9:29:54
beach
I haven't contemplated that possibility. It is probably easier to start with HIR directly for any other backend.
11:16:19
kolb
can anyone with a working lispworks install check https://github.com/eugeneia/maxpc/issues/10 for me?
11:30:41
shka
Shinmera: why oh why you had to use table names for user that makes it impossible to use s-sql ;_;
11:31:56
Shinmera
If ssql is so broken that it can't even let you specify precise table and column names I don't know what to tell you
11:35:25
flip214
I've just seen a few C compiler switch/case fallthrough warnings and wondered - what's the Lisp way to do something similar?
11:36:01
flip214
TAGBODY needs the tags in the outermost body, so we can't GO into the THEN clause of an if...
11:36:44
flip214
so perhaps the cases need to be specified manually up front, with appropriate GOs into later parts of the TAGBODY
11:37:58
Shinmera
https://www.reddit.com/r/Common_Lisp/comments/5tvd05/duffs_device_in_common_lisp/ddqpt9v/
11:50:26
flip214
would need to accumulate TAGs if arbitrary expressions need to be allowed (as in COND)
12:57:12
Xach
khrbt: maybe we could move the github one to sharplispers and call it canon? i can check with nikodemus
13:18:27
alandipert
is there a way to specify a quicklisp disp on a per-project basis? ie http://blog.quicklisp.org/2011/08/going-back-in-dist-time.html appears to set the dist for all subsequent lisp processes on the machine; i'm curious about setting it just for a session/image
13:19:13
Xach
alandipert: you can have multiple quicklisp directories - everything is done relative to the initial setup.lisp file.
13:24:02
alandipert
Xach i'm kind of a noob, so to clarify... for the per-project way, each project would have something like setup.lisp in it, right? since i see the *quicklisp-home*/qmerge stuff at the top
13:24:44
Xach
alandipert: you would have basically an entire quicklisp directory in it (the actual name doesn't matter, just the file arrangement under it)
14:14:12
attila_lendvai
I can push to alexandria and I don't know about any pending non-controversial patches
15:16:20
pjb
For example, you could encode font attributes there, so you could write bold or italic characters with format.
15:18:25
tfb
CL originally had char-bits and other functions to deal with attributes (all coming from Symbolics I think), and char-int included the bits but char-code didn't
15:19:35
tfb
but that all went away between CLtL1 and 2 I think (might have still been there in 2, was gone by the standard)
15:52:44
Bike
it's not about code points. in old lisp implementations you could encode things like a character being italicized or having a particular color.
16:06:49
shka
Shinmera: what is the canonical way to handle errors from api (like handling incorrect user input
16:08:44
Shinmera
shka: If the error is of type api-error, then the API will automatically handle it be either outputting it as data, or redirecting back with the error GET parameter set if the browser parameter is "true"
16:13:17
Shinmera
shka: Since being able to just (error "foo") and it doing the right thing would be much more convenient
16:14:39
Shinmera
The reason I didn't do so so far is that catching all errors might leak information you don't want to leak.
16:15:52
shka
perhaps automatic logging of errors and redirecting to "Sorry, programmer is any idiot :(" page would be better
16:16:52
Shinmera
Maybe, but it still increases boilerplate a lot since it doesn't integrate well with other libraries.
16:17:18
Shinmera
So for instance if you have a thing that checks inputs or whatever for validity and uses errors, that won't automatically translate to the useful behaviour.
16:50:11
Bike
Has anyone ever used next-method-p? i have no concept of how it would be used and am curious.
16:52:56
Shinmera
Bike: https://github.com/Shinmera/qtools/blob/master/widget-convenience.lisp#L39 https://github.com/Shinmera/qtools/blob/master/examples/game/primitives.lisp#L46
16:53:31
Shinmera
Bike: Generally I find it useful if you want to do something like the append/etc method combination, but want to leave control to the method definer about whether they want to exclude superclasses or not.
16:53:36
shka
oleo: around method won't be called if you don't have primary method so what's the point?
16:55:27
oleo
clos looks for primary methods first it doesn't say there have to be primary methods in order for the rest to be looked up.....
16:57:08
oleo
and it won't call other bookkeeping stuff, so if you still want that you need the call-next-method in it
16:57:39
oleo
so before calling call-next-method you can use call-next-method-p in order to ensure there is anything to be called....
17:01:11
shka
i setfed *debugger* to nil, but then out of sudden i get messages like Handling stray condition: The variable MODULARIZE-HOOKS:NAME is unbound.
17:12:51
specbot
Built-in Method Combination Types: http://www.lispworks.com/reference/HyperSpec/Body/07_ffd.htm
17:13:07
_death
"An error is signaled if there are applicable around methods and no applicable primary methods."
17:22:17
khrbt
Xach: yes, would be good to get linedit on sharplispers. I would be happy to assist with testing and merging of patches/PRs.
17:27:27
oleo
If an around method invokes call-next-method, the next most specific around method is called, if one is applicable. If there are no around methods or if call-next-method is called by the least specific around method, the other methods are called as follows: ....
17:29:31
_death
I think there's some confusion.. if an :around method does not c-n-m, of course no next method will be called..
17:30:48
oleo
so when it has a c-n-m then it looks for a most specific :around method again, if there's none it goes to the least specific :around method and then from there to the rest of the methods
17:32:08
_death
but usually it can assume that there will be a next method, so next-method-p is not needed
17:42:04
alandipert
is anyone aware of anything like a design discussion about lisp keywords? the 'why' and history of keywords. thanks in advance for pointers
17:46:24
phoe
alandipert: keywords are just symbols that evaluate to themselves. I do not know which dialect of Lisp they originate from though. Why do you ask?
17:47:11
alandipert
i am curious about aesthetic arguments for and against them, since they're technically equivalent to quoted symbols
17:47:25
_death
phoe: that's not the whole story, of course.. the reader is also modified to make keywords convenient to use
17:48:02
Xach
Yeah, (open "foo.txt" 'direction 'output) then involves package management or string compares or deeper trickery
17:53:21
_death
alandipert: a keyword is a symbol exported from the package named "KEYWORD" that evaluates to itself.. the reader is hacked to intern keywords this way, and to accept :foo as keyword:foo
17:53:24
alandipert
Xach in what respect do you find them convenient? considering 'foo and :foo are the same number of characters to type. visual distinction, automatic package membership, combination thereof?
17:54:30
Shinmera
alandipert: You're not comparing it properly. :foo is the same as 'keyword:foo. The /point/ of keywords is that they come from one single package.
17:54:50
tfb
alandipert: if you have a lot of functions which need an 'test' keyword argument, you need a home-package for that symbol. That package could be CL, but probably should not be. So it's KEYWORD
17:56:04
_death
alandipert: you may be interested in KMP's post on "data hygiene" https://adeht.org/usenet-gems/data-hygiene.txt
17:56:20
tfb
(in fact it can't be CL because that would require you to be able to intern new symbols in CL and that would be horrible)