freenode/#lisp - IRC Chatlog

3:28:35 holycow thanks ahungry

3:36:58 no-defun-allowed Maybe offtopic, but what's the best practice if I want to store upwards of 10,000 binary files (addressed by hash of contents)? Filesystems usually don't perform as well as usual when you use directories with that many tiles.

3:40:27 holycow 10k files in a folder shoudln't be a problem for ext4

3:40:47 holycow maybe millions of files and then you start to see performance degredation

3:41:02 no-defun-allowed Right.

3:42:16 holycow oh, upwards of ...

3:42:19 holycow sorry i misread

3:43:36 no-defun-allowed I'm thinking around 10^4 to 10^6 files.

3:45:09 holycow well, i have run email servers with more files than that but they have all subdivided the attachments into subdirs

3:45:21 holycow i don't know if i have ever had a single folder with that many files

3:46:05 holycow i am just waisting your time, my apologies. i don't have any useful information for you. i just misread your original question and was curious.

3:46:47 no-defun-allowed You're not wasting my time; I don't even have useless information to work with.

3:47:07 no-defun-allowed ...or can I manipulate binary data from CLSQL somehow?

3:52:54 holycow well, you know, even the base linux system has thousands if not millions of files

3:53:07 buffergn0me no-defun-allowed: You can organize the files into sub-directories named after one or more digits in the hash. Should be pretty uniform.

3:53:11 holycow my home nas machines have hundreds of million

3:53:26 no-defun-allowed buffergn0me: Sure.

3:53:39 holycow none of them seem to suffer in any way because they are all in subdirs

3:54:32 holycow i would say it does kind of depend on how big your files are and what kind of post processing you need to do. large files will force you into the obvious answer, how many small files will give you some room to play with

4:34:47 hhdave_ ** NICK hhdave

5:40:06 flip214 no-defun-allowed: while 2^20 files ain't that many, having them in a single directory is a bad idea - lookup performance and all that stuff.

5:40:18 no-defun-allowed flip214: Yeah, that's my concern.

5:40:39 flip214 I'd suggest keeping 3 or 4 levels of 8-bit divisions, that should bring the last level down to a good number.

5:40:47 no-defun-allowed (And, well, that SQL can be run over a network trivially, whereas a filesystem is a little less portable.)

5:40:51 flip214 which filesystem are you using?

5:40:51 no-defun-allowed Right.

5:41:35 no-defun-allowed btrfs, but I would expect a client to use ext4 if they use a Linux system.

5:41:45 holycow oooooohhhhh

5:41:46 holycow dude

5:41:57 flip214 no-defun-allowed: btrfs should be fine (IMO)

5:42:03 holycow did you read up on btrfs on forums?

5:42:21 holycow let me find a lnk

5:42:22 flip214 which kernel version?

5:42:24 no-defun-allowed I can't remember, I formatted /home about five years ago.

5:42:30 no-defun-allowed No, it hasn't eaten my data.

5:42:38 holycow oh it will

5:43:09 no-defun-allowed And it hasn't for the last five years?

5:44:50 holycow there is an arch page somewhere with red sections all over warning about bugs and features that don't work properly

5:44:56 no-defun-allowed ...one of them. 5.3...ish?

5:45:21 holycow just trying to find the link for you to read, i can't remember where i foudn it

5:45:45 no-defun-allowed The btrfs status page says everything I do is stable.

5:45:57 holycow zfs is stable

5:46:08 holycow btrfs shouldn't even be considered a file system

5:46:22 flip214 no-defun-allowed: I guess something recent will be good enough. I've had a few problems in the 4.<low digit> aera, but none since then.

5:47:04 holycow anyway my opinion is not important :) looking for something for you to review

5:47:11 no-defun-allowed I do lament fallocate, but my experience with btrfs is that's a load of shite and it's been rock solid.

5:49:12 andrzejku hejo

5:49:51 no-defun-allowed holycow: And I did read that page five years ago and did an analysis. Still sounds quite stable.

5:49:59 holycow oh, aha!

5:50:04 holycow okay then :)

5:51:18 no-defun-allowed I don't use some of the advanced features like checkpointing or RAID; but is even more off topic.

5:52:40 holycow i used to replicate datacenters geographically using zfs. i can't go back to anything less and reading that btrfs arch writeup scared teh crap out of me. i could never use anything like that.

5:56:40 no-defun-allowed Offsite backups are probably excessive for my /home filesystem.

5:56:46 holycow looks like synology and suse use btrfs

5:56:46 holycow huh

5:57:18 holycow i could never ever use a file system where i've read users loowing entire datasets using simple file system management tools

5:58:43 holycow yeah, no, you are actually correct. you have to match the solution to your needs and risk comfort level

6:28:56 beach Hello andrzejku.

6:29:54 beach andrzejku: Are you new here? I don't recognize your nick.

6:51:34 flip214 COMPILE returns a second and third value for warnings and errors; can I get the conditions that are signalled directly? Something like *debugger-hook*?

6:52:50 flip214 ah, HANDLER-CASE does the trick

7:07:16 andrzejku beach, yes

7:09:26 hhdave_ ** NICK hhdave

7:18:06 beach andrzejku: Great! What brings you to #lisp?

7:21:27 andrzejku beach, emm good question, Lisp is in my learning plans

7:21:39 andrzejku or lets say guile I wanted to join guile more

7:21:57 beach Ouch. This group is dedicated to Common Lisp. Sorry!

7:22:19 andrzejku yes, I know

7:22:30 andrzejku had mistaken

7:24:02 beach I think if you learn Common Lisp, you can apply that knowledge to Guile.

9:16:52 phoe the lisp dialects are similar in the beginning, though guile is a scheme and therefore much more functional-programming-oriented

9:17:01 phoe whereas CL is multi-paradigm

9:18:08 phoe once you know the basics of e.g. how S-expressions work, the evaluation model, lexical environments and lambdas, however, you'll be able to apply to any Lisp dialect with none or few modifications

9:19:18 beach phoe: andrzejku left.

9:21:21 beach phoe: Also, be careful with that kind of information. As I recall, some of the first Scheme standards did not define its operators in terms of the result of READ, but instead as surface syntax, much like other languages. So then (x . (y)) was not equal to (x y).