freenode/#lisp - IRC Chatlog
Search
21:35:21
MrtnDk[m]
* On the matrix it might be room #freenode_#common-lisp.net I'm guessing, if it be bridged.
21:40:44
etimmons
Mrtn Dk: yes (coming from a fellow Matrix user on that channel). Also, my understanding is that all of Freenode is "bridged"
22:02:22
nij
Hello! I recently started writing a personal project using emacs-lisp. It crashed me on performence so I wonder if I should switch back to common lisp.
22:02:57
nij
For now, my database is just a folder with ~7500 files. Each file contains one lisp list.
22:03:50
nij
I have never dealt with a database before, so I must be doing something wrong. But in any case, the performence is very bad. It is not clear in my head when I should read/write from/to the database, or when I should just use the variable to hold things.
22:04:30
nij
I wonder if switching back to common-lisp would solve this problem. I've heard of the power of CLOS, but have never experienced it in depth. That's my situation. So.. any advice would be appreciated.
22:18:05
phoe
also, yes, sqlite - unless you'd like to try a pure-CL solution of sorts, or try to reinvent the wheel for some personal learning
22:27:10
phoe
do you know what is a relational database, and how its data differs from object-oriented data?
22:27:27
phoe
if not, you should - and then you'll know that ORM is a toolkit that attempts to bridge that difference
22:28:22
nij
relational dbs are based on spreadsheets, and oo data are lots of classes,instances, and slots?
22:29:11
moon-child
data representation is slightly similar, and problem domain has some overlap, but the two have wildly different roles
22:29:53
moon-child
spreadsheet is end-user data flow & analysis. Relational db is relational algebra (=academia goodness) + long-term data storage & representation
22:45:16
White_Flame
nij: for something that's going to stay under 10MB for years from now, a simple persistence to log file and keeping all of it in RAM would be fine :-P
22:57:51
markasoftware
the simplest thing to do is just maintain it all in memory under a single top level variable, then use cl-marshal or similar to serialize it to a single file on disk periodically
23:26:43
jcowan
The main advantage of SQLite in this context is that it works very hard to either persist your data or fail.
23:29:25
jcowan
If you have an easy sqlite setup, then "INSERT INTO log VALUES (?)" with ? bound to the line to be logged is just about as easy as writing to a file.
23:41:46
Xach
yes, it would be much safer and more reliable. but if you go the diy route, a log is an easy step beyond stashing all the data from memory periodically.
23:43:15
Xach
and by log, i don't mean syslog things, i mean writing out data records, as in "write-ahead logs" or "commit logs"
0:00:33
White_Flame
and again, this is only if SQL is not worth learning for the scale of your project
2:07:34
aeth
postmodern (for postgresql) was the general recommendation back in the day. Idk if it still holds
2:08:59
copec
That's what I was looking at, and then looking at Mito since it is high on all my searches
4:49:23
fiddlerwoaroof
I generally think Postmodern or the sqlite wrappers are a good choice if you need a "Real DB"
4:50:06
fiddlerwoaroof
However, most programs work really well with a well-thought-out system of files for persistence
4:50:37
fiddlerwoaroof
e.g. logging updates as lisp forms and then replaying the log when your system starts up
4:51:22
fiddlerwoaroof
SQLite is useful here, because the developers of SQLite have put a lot of effort into handling the edge cases around "writing reliably to secondary storage"
4:54:24
fiddlerwoaroof
Anyways, would it be reasonable for EQL specializers to only hold a weak reference to the object they reference?
4:58:07
no-defun-allowed
From the protocol level, it makes sense. From the implementation level, I wonder how much having to remove methods which will never be called complicates things, and if having the weak-value indirection slows down things.
4:59:03
fiddlerwoaroof
Yeah, you'd probably need to make the GC aware of generic functions or something
5:01:24
no-defun-allowed
Or you could mess with the discriminator function constant vector and MMU to get similar performance.
5:02:30
fiddlerwoaroof
The reason I'm wondering is that it seems like there might be some interesting patterns around implementing protocols inside a LET or a DEFUN
5:02:45
no-defun-allowed
When the GC runs, it would have to replace any eql-specializer values that have been collected with some value - nah, we don't do anything with the value other than test it with EQL, so you couldn't use a memory trap to re-compile with dead methods.
5:04:58
fiddlerwoaroof
If the GC special-cased generic functions, it could remove the dead methods and collect the method objects
5:05:35
fiddlerwoaroof
Your method combination would have to be designed to handle dangling references, but that seems relatively easy
5:07:29
no-defun-allowed
If the function was removed, oh, there could still be mutator threads in the discriminating function, so that's true.
5:42:01
MrtnDk[m]
nij just tell us; are you raised in a Microsoft Windows environment, complete with Microsoft office, etc?
5:43:49
MrtnDk[m]
Never mind it will get off topic, and it seems you already found a solution for the db-stuff.
6:24:40
beach
ACTION wonders why people seem to systematically ignore Clobber for logging and replaying transactions, even though nobody who used it has had any complainants so far.
6:26:12
beach
It is a very simple system for logging transactions and replaying them. I mention it pretty much every time someone wants an object store, or prevalence, or a data base, or...
6:29:04
beach
I guess I am ignorant, but I don't see how to use a traditional relational database from Common Lisp.
6:29:34
beach
I mean, what if you store, say, a person (i.e. an instance of the class PERSON). How does the GC know to look in the external database for it?
6:30:23
loke[m]1
beach: That's unrelated to SQL databases though. What you're referring to is an OR-mapping.
6:30:29
beach
And if the GC doesn't find it, does it still live on in the data base? And if so, what happens when you read the object back in? Is it EQ to the object you stored?
6:31:15
loke[m]1
Typically an OR mapping is implemented using a metaclass that tracks changes to an instance and then implements a synchronisation function that writes the updates to a database.
6:32:06
loke[m]1
I implemented an OR-mapping for CouchDB once though, you can take a look at my code here:
6:32:37
beach
So that sounds really complicated. And I still don't understand how it works. Does it mean that I can't just store a person. I have to have a different kind of person that with a different metaclass?
6:33:50
no-defun-allowed
I am terrified of planning how I might handle crashes leaving my server "database" (which is backed by the filesystem) in an inconsistent state. Ideally, replication would make the stored state so small, that it'd be easier just to reload everything from the network, but that is probably not going to happen.
6:33:51
beach
But suppose I just drop the reference to my person in memory, even though it is stored in the database. Does the OR mapping know to delete the object from the data base when the GC has recycled it?
6:34:54
no-defun-allowed
The OR only deletes when instructed to. Using references would make it impossible to use datasets that are larger than the address space provided to the Lisp image.
6:36:01
no-defun-allowed
We both know that such datasets rarely exist, but that is a problem. And yes, it could be considered manual entry management.
6:36:42
fiddlerwoaroof
I tend to think of the database as a remote application like a webservice, when I'm dealing with RDBMSes
6:36:43
no-defun-allowed
I wanted to use a SQL database (e.g. Postgres) as a backend for this server, but I have an approximation of real objects to store, and the best option is to serialize into a binary format, which apparently Postgres doesn't handle so efficiently.
6:36:56
beach
So then, why on earth would a Common Lisp programmer turn to a data base even though the data set is small enough to be kept in memory?
6:37:23
fiddlerwoaroof
The main one is consistency checks that don't depend on your application code being correct
6:37:27
beach
fiddlerwoaroof: Yes, and I also don't understand remote stuff since I don't see how identity would be preserved.
6:38:17
beach
And I don't understand serialization in general either, since it too does not preserve identity.
6:38:21
fiddlerwoaroof
The DB schema usually changes much more slowly than your application code, and the cost of corrupted persistent data can be extremely high
6:39:01
beach
Clobber is such that you work in memory, and log the transactions. You can then quit your Common Lisp system, restart it, and load the log, and a "similar" object graph will be created.
6:39:17
fiddlerwoaroof
Usually identity is represented by one or more values that uniquely identify an object
6:39:24
no-defun-allowed
I am also not going to bet that the objects stored by a server will fit in memory, because truth told, I have no clue what kind of access patterns could happen, if the network is running literally any class-based object-oriented program.
6:39:42
beach
fiddlerwoaroof: So you mean you need to manually create pointers just like you do with SQL?
6:40:33
beach
I truly can't figure out why a Common Lisp programmer would be willing to sacrifice all the good parts of Common Lisp in order to use a data base.
6:41:27
fiddlerwoaroof
In distributed systems in general, the idea of "identity" is usually not really central, in the sense of reference identity
6:41:30
no-defun-allowed
Having some varying stages of going from real objects to binary blobs, I maintain a weak hash table of already-existing instances and their hashes in the server and client. Thus we can have enough object identity for things to make sense.
6:41:56
beach
fiddlerwoaroof: Exactly, which is why I don't want distributed, and I don't understand it.
6:42:32
fiddlerwoaroof
Usually you treat as identical things that are equal under an appropriately specified value of "equality"
6:42:44
beach
Abandoning Common Lisp-style identity means you are no longer programming Common Lisp really.
6:43:12
no-defun-allowed
And we continue to maintain the illusion by putting a read barrier on SLOT-VALUE-USING-CLASS, which rewrites slot values to replace "reference" objects with the real object supplied by the client.
6:43:43
no-defun-allowed
There was a technique based on "naming" objects and maintaining name translation tables in replicated object systems which I read about.
6:43:46
White_Flame
nor is a database part of your address space, nor connected to the ephemeral runtime of the lisp image
6:44:02
beach
And you have to take into account the possibility of one application object being represented by several instances. Then you have to track updates so that all instances are synchronous. That sounds like C++ programming to me.
6:44:38
White_Flame
in database applications, the database is the canonical store of information. Clients work with queries & pieces at a time, resolving all changes back into the shared database
6:45:07
fiddlerwoaroof
Yeah, no client should ever care about the state of the data in another client
6:45:12
White_Flame
any lisp runtime can come and go, working with the external data in the database, as much as little as it wants, and can do so simultaneously with others
6:46:26
no-defun-allowed
I don't think SQL has references (any more than it has indexes of rows in tables).
6:46:35
fiddlerwoaroof
I think the best way to think of a table is a relation between the primary key and 0 or more other values
6:47:17
no-defun-allowed
At the rate at which I program, it might be wiser for myself to wait for CLOSOS, and use the in-memory database implementation as a persistent database.
6:47:21
beach
Now I am beginning to see the answer to my question. It appears that the data base can store only objects that are not guaranteed to be EQ, so that creating a copy is not a problem.
6:47:23
White_Flame
and because it has no implicit references, that removes the burden fo having everything manifested at once
6:48:03
White_Flame
databases are for: 1) your data is bigger than RAM, 2) you need transactional security, 3) you have multiple users hitting your data simultaneously, 4) the data needs to persist safely
6:48:03
moon-child
if it's a concern, make your own version of eq that compares primary keys for db-sourced objects
6:49:14
White_Flame
and of course, SQL (and any query language, like prolog etc) makes traversing thorugh tons of data way simpler than imperative looping
6:49:56
no-defun-allowed
Re-reading the Postgres manual, the "limitation" that a binary value has to be materialized entirely in memory is not really a problem, as I would do that on the Lisp side briefly too.
6:50:16
fiddlerwoaroof
It's great for situations where you want to distribute work across a large number of computers
6:51:09
fiddlerwoaroof
The last system I worked on basically treated the database as a cache, and you replayed the log through all the worker nodes to rebuild or update the cache.
6:51:46
White_Flame
beach:your EQ conclusion is incorrect as well. Relational keys have very speciifc equality semantics, and those are the only things that are compared for identity
6:51:51
fiddlerwoaroof
But, in this context, the restrictions are even more severe because you have to design your data so that reprocessing the same log entry doesn't change the DB
6:52:34
fiddlerwoaroof
Because, roughly, you can't guarantee that a message in the log is only processed once (if you want to make sure that you don't lose any data)
6:52:38
beach
White_Flame: Sure, but they are separate from my Common Lisp references, so I have to program differently.
6:53:04
White_Flame
correct, they are data, they are not objects. there is no pointer or reference to them. they are stored by key
6:53:35
White_Flame
and those keys between rows can be connected in unique ways via sql to get summaries, comparisons, joins, etc in complex ways across huge datasets
6:54:18
beach
So here is a good story: My employer bought Oracle to store student information like exam results, etc. I proved to my students that 100 years of such data would fit in the RAM of the laptop I was using at the time.
6:54:29
White_Flame
actually, the key is only for fast lookup, there can be slow lookup for any match
6:55:10
White_Flame
right, you can make custom solutions for anything. Or you can have multiple machines all sharing ACID protected data in a standard way
6:55:26
White_Flame
instead of only implemented the protections you think are needed at implementation time
6:55:32
fiddlerwoaroof
White_Flame: the primary key is sort of essential to the relational model, though
6:57:03
White_Flame
any non-key column can be used for selection. just the key is unique with the table and indexed well
6:57:06
fiddlerwoaroof
e.g. your schema is largely driven by the idea that a table is a relation between the columns that compose the primary key and the rest of the columns
6:58:29
fiddlerwoaroof
I've never read Codd's book on the relational model, but my general impression is that the primary key is not just an optimization, it's part of the theory behind the model
7:01:29
fiddlerwoaroof
RDBMSes also tend to be used by multiple applicaitons simultaneously: e.g. the applications your developers make for users, the financial department's accounting software, the data science department's modeling tools, etc.
7:02:01
fiddlerwoaroof
The other big advantage is that it's a relatively rigorously specified way to represent data that lots of existing tools can take advantage of.
7:07:14
no-defun-allowed
Generally, I'm now convinced that getting a database to behave with the possibility of crashes is easier than getting a filesystem to behave.
7:09:26
fiddlerwoaroof
no-defun-allowed: this is a great read on that topic https://danluu.com/deconstruct-files/
7:11:30
fiddlerwoaroof
Some people tried testing the assumptions made by various tools that have to write files reliably.
7:11:39
fiddlerwoaroof
They discovered "every single piece of software they tested except for SQLite in one particular mode had at least one bug"
7:11:53
no-defun-allowed
Making either work correctly with all the logic, and all the stuff I keep in memory for performance, seems hard though.
7:12:24
fiddlerwoaroof
And this isn't an accident: SQLite is tested for correctness pretty thoroughly https://www.sqlite.org/testing.html
7:12:44
fiddlerwoaroof
So, my recent thought has been "don't write data to files, write it to SQLite"
7:13:30
fiddlerwoaroof
You don't even need to treat it as an actual database, just one table with a single column that you append to
7:48:03
loke[m]1
If you want to see my implementation of a object mapping for CouchDB: https://github.com/lokedhs/potato/blob/master/src/potato/metaclasses.lisp#L33
7:49:04
loke[m]1
And here is one use of it: https://github.com/lokedhs/potato/blob/master/src/potato/upload.lisp#L193
7:59:56
srandon111
hello all, is active development on common lisp and/or common lisp libraries/modules still done? or is it a lnaguage just used to mantain old software?
8:00:35
beach
srandon111: There is lots of active development of applications and system software written in Common Lisp.
8:03:15
scymtym
there is probably also some work to keep legacy common lisp software running, but my impression is that such cases often involve commercial software and the commercial implementations and are not discussed here
8:04:23
beach
srandon111: Where did you get the idea that there might not be any active development in Common Lisp?
8:10:08
beach
srandon111: I said "ignorant", but there is another category of people who will spread this kind of misinformation, namely people who have a huge amount of effort invested in some technology that might be threatened by Common Lisp.
8:12:09
beach
That last kind of people is a dangerous one. Carol Dweck calls this phenomenon a "closed mindset". And these people will sometimes do awful stuff to avoid having to abandon the technology they invested in, and learn something better.
8:13:03
beach
Often, people with a closed mindset are also very sociable and come across as leaders, which makes the problem even worse.
8:13:25
scymtym
but there is no useful answer to such questions anyway. if a single person still uses common lisp, the answer whether anybody still uses common lisp would technically be "yes", but practically "no". but at what threshold does the answer become practically "yes"?