freenode/#lisp - IRC Chatlog
Search
21:57:54
ealfonso
what is the best way to understand which objects are consuming the most memory in the heap? is it the sbcl statistical profiler?
21:59:10
aeth
I use sb-profile and if you break things up into small enough functions it's pretty easy to see where the allocations are happening.
22:02:38
ealfonso
if I'm interpreting correctly, statistical profiler or sb-profile gives me a 'consed' column or the amount of new allocs, but does not account for gc'd memory. I have a lot of functions that seem to alloc memory temporarily. what I'd like to see is which memory has been allocated and not gc'd (e.g. to optimize long-running process slow memory leak)
22:08:35
aeth
you can diff two (rooms) to see what has changed, but the act of producing the string for (room) probably allocates. There's probably a low-level, SBCL-specific thing you can use to bypass this.
22:12:45
ealfonso
I can see dynamic space usage with (room), but not which objects or which functions are generating it. I can see a snapshot of new allocations with sb-*prof*, but it includes temporary allocations that might be gc'd. I guess I also need information about what is being gc'd
22:13:42
aeth
ealfonso: how temporary are the temporary allocations? Could you (declare (dynamic-extent foo)) them?
22:18:40
ealfonso
I don't know why a function with a single cffi:foreign-funcall sexp is showing the most 'consed'. when there should not be any allocs in the foreign function
22:20:19
aeth
I don't think so. My CFFI code doesn't cons. The overhead tends to be from poorly wrapped CFFI, like copying huge arrays (cl-opengl likes do do this, but fortunately it also has the %gl package)
22:24:39
ealfonso
https://pastebin.com/2tacbyZZ this is my wrapper that is showing the most conses... does it mean my_foreign_function must be allocating?
22:38:12
ealfonso
pjb is cffi:foreign-funcall the wrong (slower) approach, should I be using defcfun instead?
22:39:03
pjb
Using defcfun will generate at macroexpansion time the native FFI definitions that the compiler should be able to call directly. AFAIK, it's bound to be more efficient.
22:40:32
pjb
Yes, for the lisp object it's implementation dependent whether there's such a function, or even such a notion.
22:43:16
ealfonso
pjb I will try switching to defcfun. how about getting the size of a lisp class (or defstruct)
23:18:53
pjb
minion: memo for beach: apropos 9:17 https://www.youtube.com/watch?v=ZJr81DtSwUc (loop for i below 10 sum i maximizing (sin i) into maxin collect (* i i) into squares finally (print (list maxin squares))) is conforming AFAIK. it has several accumulations clauses of different types… ;-)
0:17:23
aeth
com.informatimago and xelf and ECL are officially ahead of the trend because they've been on Gitlab for forever.
0:36:58
rumbler31
pjb: regarding trust, if I held that mindset I'm not sure how I would have gotten anything done.
0:56:20
aeth
rumbler31: It definitely depends on your domain. For some domains, you have no choice but to pay thousands of dollars for some overpriced specialized software.
0:57:25
rumbler31
the discussion I meant to re-start was about debugging tools. beach mentioned something that suggested that in his experience, paid implementations have better debugging experiences and I wanted them to expound on that
1:02:58
aeth
It wouldn't surprise me. Commerical software like that tends to focus on features to justify upgrades. e.g. integrated Prologs.
3:10:55
minion
beach, memo from pjb: apropos 9:17 https://www.youtube.com/watch?v=ZJr81DtSwUc (loop for i below 10 sum i maximizing (sin i) into maxin collect (* i i) into squares finally (print (list maxin squares))) is conforming AFAIK. it has several accumulations clauses of different types… ;-)
3:12:40
beach
rumbler31: I have no experience with commercial Common Lisp implementations, but I hear they are much better than the free ones for debugging. For my own experience, I still miss using GDB back in the days when I was programming in C. I find I can't do things with our free Common Lisp implementations that I could do then.
3:15:49
beach
rumbler31: I do know what I would like to see for our free Common Lisp implementations, namely this: http://metamodular.com/clordane.pdf but it will require significant support from the implementation, so it probably won't happen with the existing free implementations.
3:48:07
jeosol
I have not gone through your linked pdf in detail, but does your design closely match gdb in c/c++ or it's different with added functionality.
3:49:38
jeosol
Also, what do you mean by "significant support from implementation", obviously man hours, but does this also mean it is likely to require lot of implementation details that it is likely to be very different for each free implementation
3:50:11
jeosol
also since SBCL is very popular, is this feasible to at least put efforts for that implementation.
3:51:42
jeosol
As the designer, you are in better place to push this because it is going to be more clearer to you, and subsequently others who take time to study the document (and who may have used other debugging systems)
3:52:30
beach
jeosol: I did not try to match GDB. I was considering something I would like for Common Lisp.
3:53:11
beach
jeosol: In particular, I won't use different processes. I want one thread to debug another thread, and the debugged thread could be the debugger.
3:54:03
beach
jeosol: I am totally confident that I would be unable to convince the maintainers of any current implementation, including SBCL, to add the support that Clordane requires.
3:55:03
dtornabene
i just finished it btw, seems like the implementation of this on sbcl wouldn't be....momentous? but then I have the barest knowledge of sbcl internals
3:56:19
beach
jeosol: I don't know SBCL well enough to give an estimate. It requires compiler support and, as I understand it, the SBCL compiler is very complicated.
3:56:23
jeosol
and possibly implement using a style with as little modification to the original code base.
3:57:13
jeosol
if it has to involve working deep in the compiler, then it will require major effort for sure
3:57:50
beach
The SBCL maintainers will likely say that it might slow down the generated code, and reject it before even considering it.
3:57:55
dtornabene
talk about a subject in need of a good book, a serious thorough book on SBCL internals would be *awesome*
3:57:59
jeosol
for my applications, I wouldn't say I have not had a need for debugger, but I ensure my systems are testing in very small units, and when CLOS objects are used, I have constructor functions for easy testing.
3:59:09
jeosol
however, my debugging probably sucks, combination of inlet and outlet statements to function (stopped this), SLDB, good old print statements, ...
4:00:03
jeosol
beach: on the issue of rejecting, it would have been nice if it was even possible to scope out requirements and what might change or be impacted in the context of SBCL
4:01:22
beach
dtornabene: You have to consider that SBCL has a long history and the internals have very likely become a bit messy over time.
4:02:53
dtornabene
beach: for sure, i spent a few hours recently just reading, taking line counts, investigating the layout. It seemed like a huge task to get on top of the system as a whole, but its something I'd like to devote a non-trivial amount of time to over the next 12 months or so
4:03:48
beach
jeosol: I totally think that we need a more modern implementation. That is why I am not willing to put any effort into SBCL. In particular, if I have to convince the current maintainers to make changes that are very unlikely to be accepted, I prefer putting my energy into something more modern.
4:05:25
beach
jeosol: One that has first-class global environments, my generic dispatch, my LOOP implementation, my garbage collector, the debugging support I want, etc, etc.
4:06:44
jeosol
Good points. I take it there is not way to implement these new functionalities or add them in existing implementation
4:08:34
beach
jeosol: There is always a way to implement them in an existing implementation, but it is a question of how much work it requires, and whether the maintainers are willing to accept it.
4:10:53
jeosol
By the way, are big companies using SBCL or some of the other free implementations, such that they could fund some of these type of projects
4:12:22
jeosol
This could be an opportunity to take CL to be more modern and add more battle tested, and proven libraries so some library class "like Boost C++ libraries" that eventually make their way into the core.
4:12:33
beach
loke: A sliding collector for the nurseries (per-thread) and a combination of mark-and-sweep and a traditional malloc/free for the global collector.
4:13:58
jeosol
I say we apply for grant, so pull some resources, especially with the merits of the new system.
4:14:17
beach
As Wilson et all showed a few decades ago, fragmentation only exists in programs that don't do anything useful.
4:14:42
beach
loke: I am planning that, but I haven't given it much thought. I want it to be correct first.
4:15:10
loke
The benefit of compaction isn't that fragmentation is sliminated per se, but that allocations can be much faster if the free memory is clustered.
4:15:40
jeosol
Or may be fork off SBCL now, just like the latter from CMUCL back then. But it will likely be more difficult if the base code is messy. For large codes, I know first hand how important it is to get the base right.
4:16:09
loke
beach: Right. That's what makes GC-language allocations so much faster than C, for example.
4:16:11
beach
loke: In the global collector, allocation is made in big blocks to promote many objects at the same time.
4:16:56
beach
jeosol: My hunch is that it would be even more work to transform SBCL into a modern implementation than it would be to create a new one.
4:17:38
jeosol
beach: I agree with you last point. Yes, especially if the design is radical. It may be better to start of a fresh base where things are layered systematically.
4:17:42
loke
akkad: Ironclad hasn't had any known vulnerabilities, AFAIK. It doesn't suffer from buffer overruns and such, and the algorithms seems to be solidly implemented.
4:18:15
beach
loke: Nothing special. I have a linked list of two-word header objects that use mark-and-sweep. When a dead header object is found, the equivalent of free() is called on the "rack" (the part that contains the data).
4:21:18
beach
Sort of, yes. The nursery could tell the global collector that it needs a bunch of objects at once.
4:22:56
beach
There are plenty of advantages of this scheme. For one thing, an object in the global heap never moves.
4:26:19
beach
I would need people who 1. are available for hire, 2. know about compiler techniques, 3. know about Common Lisp details, 4. know about garbage collection, 5. know the details of CLOS, 6. know about reusability, maintainability, portability.
4:26:59
beach
I am betting that the handful of people who fit that profile are already working on some exiting free or commercial implementation.
4:28:12
jeosol
Well, it also depends, there are people coming up the ranks that may not know everything now
4:28:48
jeosol
My code uses most CLOS but I had to read Sonja's book, and a bunch of others to get good over time.
4:29:12
jeosol
Looking at the list, I think you want people who are heavily invested in CL overall, and true some may be working on other projects now
4:29:19
beach
jeosol: Oh, I see what you are saying. I should 1. Apply for a grant, and 2. Hire you.
4:30:22
jeosol
I am not the most person qualified for this task. What I meant above is the person who has the main design and how all the parts fit, will lead the efforts early on.
4:31:22
beach
jeosol: Here is what I think. I think I should continue doing what I am doing now, i.e. working on the implementation and doing the research to invent techniques of making it modern, good, and fast. Then, people who are sufficiently knowledgeable and sufficiently interested will show up spontaneously. Like Bike, heisig, drmeister, etc. And there are already signals that some more people will join.
4:31:24
jeosol
I think it's the team that matters most as long as they believe in the final objective, agree on the common grounds, ..., off and running
4:32:06
jeosol
I didn't know you have gone far with this, hence my grant comment. I thought it is something just starting out.
4:32:09
beach
jeosol: Besides, Common Lisp has been off the radar for grant-giving institutions for the past half a century or so.
4:32:39
jeosol
If you have gone far, by any means, I would say continue, because changes are likely going to slow and impede things now.
4:33:15
beach
jeosol: The Cleavir compiler framework is already used for Clasp. And karlosz has a GSoC project to use it for CLISP. jackdaniel has plans to use it for ECL. And in Marbella, there were rumors that someone might use it to write a new SBCL compiler.
4:33:50
minion
jeosol: Cleavir: A project to create an implementation-independent compilation framework for Common Lisp. Currently Cleavir is part of SICL, but that might change in the future
4:34:40
beach
jeosol: And, obviously, the compiler is a HUGE part of a new implementation, so a big chunk of code is already done.
4:34:46
jeosol
When I started working on my code/application. I got to a point, where I was considering changing language since people start saying I should consider Clojure. I actually posted this question on Twitter and Paul Graham advised against Lisp, and only agreed when I mentioned my code is heavily CLOS-based
4:35:13
jeosol
Eventually, I stayed with it. A buddy told me to use something else as a condition to come along.
4:36:46
beach
The fact that Cleavir can adapt to totally different implementations is an indication that it is modular and maintainable.
4:37:08
jeosol
Clasp is something I planned to take a look at after discussing ML libraries and it was recommend to use clasp and link to other ML/AI libraries and make them accessible via CL
4:37:21
beach
Contrast that with the SBCL compiler that has assumptions about that particular implementation all over the code.
4:38:04
jeosol
I see your point. It is better to start fresh. For large code bases, it usually works better than way and you get more productivity as you add more layers on top.
4:39:22
beach
loke: Interestingly, the fact that the global garbage collector does not move things around, makes this ideal for interoperation with foreign code like C. But such interoperation is not one of the goals of SICL. :)
4:40:11
jeosol
beach: to echo your "reusability, maintainability, ...", point, it is very point and you can see from the use of Cleavir component in several projects.
4:41:27
beach
Right. I spend a lot of time on that stuff. I am not perfect of course, so occasionally I get it wrong.
4:41:58
karlosz
to add to beach's point, the Cleavir to CLISP interface is only about 900 lines or so
4:49:35
beach
So the goal of Cleavir is not only to provide a compiler framework, but also to include all kinds of modern compiler optimization techniques so that a client implementation can pick and choose what optimizations it wants to apply and in what situation.
4:50:53
beach
Some of these optimization techniques can be used as published. Some need to be adapted to the specific situation of Common Lisp (nested functions, threads, closures, etc). Others need to be invented from scratch (like the path replication stuff we published in ELS).
4:51:44
jeosol
beach: you got me. I am convinced and sold. Definitely not possible, so forking it wouldn't work
4:52:17
jeosol
Layered software/code is a beauty, it takes almost no effort to add new features or corner cases.
4:52:44
karlosz
one big obstacle to compilers is just visualizing the IR. sbcl still odesn't have any good tools for doing that, even after 30 years of existence
4:53:11
karlosz
whereas i can say that the cleavir visualization tools really cut down on debugging time
4:53:51
karlosz
in fact, if you want o learn about sbcl internals, its probably still better to go back to the cmucl docs
4:55:51
beach
jeosol: Since perhaps you don't fit the profile for Cleavir or SICL, you might instead want to consider a project on this list: http://metamodular.com/Common-Lisp/suggested-projects.html :)
4:58:04
jeosol
I have been trying to learn ASDF as Fare is planning to leave at some point so he told me. Beach, I can work on multiple projects. Most of the work I was doing on that application is for the backend which is mostly complete
4:59:29
loke
It processes text, and generates a sequence of glyphs and their precise location where they should be drawn
5:01:06
beach
jeosol: SICL is only one line in that list: "A Lisp system with first-class global environments"
5:04:30
loke
LdBeth: McCLIM already supports it in my Freetype branch. What Beach is talking abou tis a pure-Lisp implementation.
5:04:43
jeosol
are you considering things around data manipulation arrays (e.g., numpy, pandas as in pythong), machine learning/AI, even if down the line
5:17:15
beach
jeosol: For applications, I am only able to write a list of the ones that I personally would like to see and that I know that we don't have.
5:20:40
jeosol
people talking about optimization tend to be carried array. very impressive, I need to take a look at the stuff again.
5:21:46
jeosol
I think there are expertise littered around that can be converged which is why I mentioned the effort of the individuals involved.
5:22:19
jeosol
oh I will definitely come to ELS. I submitted an abstract for a conference in China and one due next week for a conference here. This is also the lisp work
9:44:32
beach
jmercouris: It might work with some restrictions, yes. But it would be very implementation specific.
9:44:59
shrdlu68
There are several serialization libraries, perhaps those that do binary serialization could be useful.
9:45:10
jmercouris
beach: "The pickle module implements a fundamental, but powerful algorithm for serializing and de-serializing a Python object structure. "Pickling" is the process whereby a Python object hierarchy is converted into a byte stream, and "unpickling" is the inverse operation"...
9:45:48
beach
Sure, you can serialize things. But that's not the same as getting the binary representation.
9:46:18
jmercouris
what is the difference between serializing to binary, and the binary representation?
9:47:27
beach
PRINT is a very good serializer. It produces output in the form of characters. If you convert those to UTF-8 as most Common Lisp systems would, then you serialize to binary.
9:47:39
shrdlu68
jmercouris: Yep, so does ironclad and most packages that deal with networks protocols.
9:48:04
jmercouris
beach: well, I am not necessarily interested in how it is represented internally, just in a way to serialize and deserialize the objects
9:48:18
jmercouris
and to take the serialized form, and be able to input it in some reserializer and create an "identical" object
9:48:19
beach
jmercouris: And, you can de-serialize with READ. But there are restrictions, just like there must be in Python. You can't get an EQ object back.
9:49:02
shka
cl-store works on lists, vectors, instances of classes, structs (for sbcl), hash-tables..
9:49:19
phoe
a human-readable text representation is a very specific case of a binary representation of an object
9:50:10
jmercouris
okay fine, I would then need to take the character representation and turn it into a binary stream of some text encoding and decode it
9:50:47
_death
in many cases, READ is unsuitable.. and serialization of arbitrary objects is often not a good idea anyway.. it's better to think about your external representation without reference to any particular programming language
9:51:52
jmercouris
I just wanted a simple way to get a bunch of bits for some representation of some thing to test with
9:52:34
jmercouris
I know about existing CL QR code libs, I might end up inventing a version of my own, but I just thought it was a cool idea
9:52:45
jmercouris
you could sort of stream a whole bunch of data in video format, to transfer more substantial things
9:53:12
_death
so you can encode the string to an octet vector, then use that to produce a QR code, and do the other way around on the side side
9:53:23
shrdlu68
jmercouris: See also: https://common-lisp.net/~loliveira/ediware/flexi-streams/doc/#strings
9:54:06
jmercouris
shka: yeah sure, there will have to be something in the protocol to show "start of sequence" "end of sequence"
9:54:28
jmercouris
it's effectively a list of 0s and 1s, at least that's what I got from the wikipedia article
9:56:13
jmercouris
beach: I'm a little bit confused, is there some way to do (print "some string") --> 0010101010101011?
9:57:12
beach
jmercouris: There is no standard way (Common Lisp or otherwise) to represent a string in binary form.