freenode/#lisp - IRC Chatlog
Search
1:29:31
pfdietz
I'd say not hard, although there are some parts you don't want to do first. Restarts are a bit tricky.
1:35:11
pfdietz
Restarts are a way to resume after a condition (like, an error) is signaled, without unwinding the stack.
4:01:59
Colleen
beach: drmeister said 1 hour, 17 minutes ago: concrete-syntax-tree-base.asd contains a dependency for :alexandria but doesn't use anything from the system - could you remove it? Also - I submitted a pull request for the improved fix for handling end-of-file in the sicl reader.
4:38:51
drmeister
Hey lispers - when people refer to "tooling" as being important in how they use their programming languages and IDE's - do you know what that means?
4:39:27
drmeister
I think I kind of know what it means - but I'm trying to broaden my understanding of what people think it is.
4:43:38
drmeister
http://reactkungfu.com/2015/07/the-hitchhikers-guide-to-modern-javascript-tooling/
4:45:57
drmeister
When I read responses on Hacker News to new language announcements - invariably someone says "Your new language X sucks because it lacks what my language Y has - which is the tooling".
4:47:28
drmeister
One of the promises of Clang's AST tools that hasn't really been realized afaik is that it would improve tooling for C++ syntax aware tools.
4:48:36
drmeister
syntax highlighting, finding locations that call a function, finding where functions are defined - these could be made more robust with cst-to-ast.
4:52:11
beach
Currently, most implementations implement TRACE by modifying the code. Then you can't debug the debugger.
4:52:12
drmeister
Imagine if someone makes a poor choice for a function name - or dynamic variable. grep/search/replace is brittle and error prone. A syntax aware search and replace would be much more robust.
4:54:05
beach
The good thing here is that I plan to use CST-to-AST also in Second Climacs, so that we can avoid code duplication.
4:55:11
drmeister
Can I just say that I think "Second Climacs" is an unfortunate name for an editor?
4:55:50
drmeister
I feel the same way about "Slime" - love the programming environment - hate the name.
4:57:34
drmeister
That won't make me stop using something - but I'd love to see a better "official" name.
5:02:30
beach
The point here is that our Common Lisp implementations need to make breakpoints "thread aware" so that we can have one thread run the debugger to debug another thread.
5:02:50
beach
As long as that is not happening, we are doomed to have inferior programming environments.
5:04:07
beach
If we do it that way, we can even allow the user to set breakpoints in standard functions like READ or GETHASH. Currently, that would almost certainly make the debugger stop working.
5:05:13
drmeister
I don't disagree. For things like that I currently use gdb or lldb - but they are not lisp aware.
5:07:33
drmeister
I say I think it compiles a wrapper function because I haven't investigated how it works. I inherited it from ECL and it just works.
5:08:21
aeth
Do people use issue trackers to track future features in large Lisp projects? Or do people use something else (e.g. org files?)
5:09:50
beach
drmeister: Then you are not only recompiling the function, but also modifying the FDEFINITION.
5:21:44
beach
Anyway, I suspect Clasp and ECL (just like SBCL) don't have thread-aware trace points or breakpoints.
6:23:56
beach
So here is an idea for the implementation of thread-aware breakpoints in x86. The important feature is that threads that are not being debugged should not take too big a performance hit.
6:24:00
beach
I suggest that, when code is compiled with a high value of the DEBUG quality, then the execution of each form starts and ends with a test of the DF status flag. If 0, then execution continues as normal.
6:24:01
beach
If 1, then some further action is taken to determine whether there is a breakpoint at this point in the code. The "further action" remains to be specified, but the slow version would be to interrogate a breakpoint table in the thread instance.
6:25:32
beach
The use of the DF flag means that, if the implementation uses specific instructions that depend on its value, then the value has to be saved and restored around the execution of these instructions.
6:33:16
rme
when I last looked into using the x86 direction flag, I found that manipulating it was expensive. it ended up being faster to set a flag in memory.
6:33:49
White_Flame
I'm kind of preferential to Java-style interruption, where an address is routinely read. If it's to be interrupted, the address range is made unreable. Now, you could extend this so each thread is reading a different address (just an indirection through a thread-local pointer), and/or use different addresses per potential breakpoint.
6:34:15
White_Flame
I think you have to push flags to stack to read the DF flag, only set & clear are part of the ISA
6:35:35
White_Flame
hmm, if you have ample room in your 64-bit address space, you could read thread_local_page + N*program_counter to get a unique memory-protection based interrupt per location in your code
6:36:45
White_Flame
and with future fixes for meltdown/crispr, it might even be quicker to interrupt than it is now ;)
6:37:09
rme
I'd want to look at trying to use the dr0 through dr3 debug registers on x86 for setting hardware breakpoints if possible
6:39:34
beach
rme: I haven't had the time to read up on hardware breakpoints, but that's an important subject.
6:43:30
beach
I still like the DF over an address in memory, because that address would have to depend on the thread, making it expensive to read because of indirection. But here is a compromise: at the entry of each function compiled with a high value of the DEBUG quality, move the DF flag to a register or to the local stack frame so that each further test inside the function is cheaper.
6:46:07
beach
Some simple hash-like technique could be used. Take the PC modulo (say) 256 and consult a table in the thread instance. If a bit is set there, then consult a complete hash table to determine whether this PC value actually has a breakpoint.
6:47:11
White_Flame
I suspect the Java way is one of the least intrusive, though it probably does tie up a cache line
6:47:59
rme
there's almost certainly going to be thread-local memory maintained by the lisp runtime. on ccl, %fs or %gs points to the thread-local memory block (except on highly advanced systems like macOS, where we have to burn a gpr for this).
6:48:23
White_Flame
beach: each thread reads its own address; you only deny read permissions for the memory page for one thread, in my idea
6:49:53
White_Flame
basically, each thread just reads a canary location in memory. When you want to halt a thread, play with the memory manager for a particular canary location
6:52:42
White_Flame
it's just a read through a register+offset indirection, assuming a thread-local table is in a register
6:55:11
beach
So, either way, we have at least two ways of making thread-aware breakpoints with very little cost for threads that are not being debugged, and a modest cost for a thread being debugged, but not at this particular value of PC.
6:56:11
White_Flame
"safepoints" is seemingly the jvmterm: http://blog.ragozin.info/2012/10/safepoints-in-hotspot-jvm.html
6:56:56
White_Flame
If you did a read of thread-local-canary-address + 4k*PC, then maybe you could use the same trick to interrupt at a particular PC on a particular thread
6:59:13
White_Flame
if I remember correctly, the width of the visible universe in terms of electron-diameters is on the order of 2^80
6:59:18
beach
It is not important to me at this point to know the exact solution; just that one exists. And I am now convinced that there are at least two.
7:01:15
jackdaniel
someone told that god had to be a very cruel at the time he made men – give them mind with capabilities to make almost infinitely ambitious plans and time to live which doesn't allow to excercise them :)
7:21:55
beach
White_Flame: Wait, your single per-thread address read suffers a severe performance problem when the potential breakpoint turns out not to be one.
7:23:14
beach
Here is a thread being debugged, so I arrive at some point in the code and I read my unique address.
7:23:17
White_Flame
but yeah, switching the MMU stuff to reject he read and flushing TLB cache or whatever is a pretty expensive operation, but that's all involved when you actually perform the interrupt, not exploration
7:23:44
White_Flame
ah, right, if it's just a per-thread address, and not per-thread address + 4k*pc
7:25:16
White_Flame
there's all sorts of granularity tradeoffs you could make, like assign a page offset per function, so you only hammer it when it's in the particular function the breakpoint is in
7:25:43
White_Flame
but yeah, if you're not going to have it per-PC, then it might make more sense to do manual checks, if you can do them cheaply
7:26:17
White_Flame
random thought, but are there any feature of cpu affinity that could help isolate the thread?
7:27:50
White_Flame
but really, how expensive is it to set a breakpoint, and test to see if you're in the intended thread?
7:28:47
beach
OK, final proposal that does not involve DF: At the entry of each function, read an address at a fixed offset in the thread instance. It contains a flag that indicates whether this thread is debugged.
7:28:53
beach
The contents is put in the stack frame or in a register, determined by the register allocator. Before and after the evaluation of each form, call a small local routine with PC as its argument.
7:28:59
beach
That routine starts by testing the flag. If 0 it returns normally (this is the case when the thread is not being debugged). If 1, it takes PC modulo (say) 1024 and consults a bit table in the thread instance.
7:29:08
beach
If the table contains a 0 (the usual case) then it returns normally, because it means there is no breakpoint at this PC value. If 1, it consults a hash table in the thread instance to see whether this particular PC value has a breakpoint.
7:29:09
beach
If not, it returns normally. If it does, it suspends execution and gives control to the debugger thread.
7:31:29
beach
White_Flame: I don't know the answer to your question, but this last solution of mine would work on any architecture.
7:33:19
White_Flame
hmm, it would only work for a single breakpoint, but if you could do 1/(PC - read-from-thread-local), you could trigger a divide by zero exception if that location was set to the current PC
7:34:30
beach
When the debugger sets a breakpoint, it updates the hash table and the bit table in the thread instance. When a breakpoint is being removed, it clears the bit table, traverses the hash table to set the bits in the bit table.
7:35:45
White_Flame
hmm, if you double the memory footprint of your code, you could have a flag per instuction byte
7:36:09
White_Flame
then it's a large PC-relative offset to see if that flag matches your current thread pointer
7:36:51
phoe
I don't think he wants per-instruction breakpoints though, rather one breakpoint per Lisp form.
7:40:03
White_Flame
it's nice living in the future where you can throw egregious amounts of RAM at a problem
7:44:02
beach
Hmm. If I make the bit table smaller, say 64 bits, I can skip the single bit and load this small table to the stack or to a register (subject to register allocation). The famous small routine would then immediately test a bit that is PC modulo 64. How much more expensive would that be compared to testing one single bit? I am guessing "not much".
7:46:03
beach
Maybe not. The debugger would have to know how to update the local version of the bit table. Not worth it, probably.
7:49:59
shka
White_Flame: it would be nice, if people would not throw copious amounts on RAM into web based application
7:52:58
White_Flame
but I think a good post-IDA disassembler (my big web-based application) tends to be for people willing to have big hardware footprints
8:52:46
beach
A summary of my own conclusions of the discussion on breakpoints is now in Appendix A of this document: http://metamodular.com/clordane.pdf
8:53:21
beach
I will submit it to nyef`` when he wakes up, because we have discusses such things in the past.
8:56:09
beach
Interesting related information: When I run SICL inside a host Common Lisp implementation (currently SBCL), I can implement breakpoints this way with very little effort. I can then implement Clordane so that it can debug SICL programs, even though 1. SICL doesn't really exist, and 2. SBCL doesn't support per-thread breakpoints.
8:58:29
beach
Compare this idea to generating a SICL executable sooner rather than later. Then I would lose all my usual tools and I would have to resort to GDB or something similar with absolutely no knowledge of Common Lisp.
8:58:41
schweers
beach: So its possible to add better debuggers to existing implementations without modifying said implementation?
8:59:25
beach
Not really. I am not improving the processing of native SBCL code. Only code compiled with the SICL compiler for execution inside SBCL.
9:03:51
beach
I am basically accumulating evidence that I should not be hasty when it comes to creating a native SICL executable, and even less so when it comes to creating a bootable LispOS image. Instead, I should take advantage of a safe execution environment (here SBCL) to make the system as complete as possible first.
9:23:25
beach
Strange! Most of the code exists, but I am making very slow progress with the bootstrapping phase.
9:27:07
schweers
beach: out of curiosity: what exactly does sicl do? does it compile to native code or something else?
9:28:08
beach
It will when it's done. Right now, it generates AST then HIR. The HIR code is then turned into a small subset of Common Lisp that is compiled on the host compiler for ultimate execution.
9:29:00
beach
And the Common Lisp that is submitted to the host compiler uses first-class global environments for all its lookups.
9:29:54
beach
I haven't contemplated that possibility. It is probably easier to start with HIR directly for any other backend.
11:16:19
kolb
can anyone with a working lispworks install check https://github.com/eugeneia/maxpc/issues/10 for me?
11:30:41
shka
Shinmera: why oh why you had to use table names for user that makes it impossible to use s-sql ;_;
11:31:56
Shinmera
If ssql is so broken that it can't even let you specify precise table and column names I don't know what to tell you