freenode/#shirakumo - IRC Chatlog
Search
13:28:54
Shinmera
I can't come up with anything other than just "file-system" or "file-systems", but that's not a great name for obvious reasons
13:37:09
SAL9000
Shinmera: when you say "file system API", are you thinking of something like UIOP, or...?
13:38:10
Shinmera
I'm thinking of a general protocol that describes systems that keep a set of "files", objects with several attributes and a central payload.
13:39:14
Shinmera
The primary idea is to provide a protocol that allows you to address stuff stored on the OS FS or within a "file FS" the same way.
13:42:07
Shinmera
I have a progenitor of this in Kandria to allow compressing parts of it when deploying, but not needing separate addressing.
13:42:36
Shinmera
But, while I'm ratifying it, might as well turn it into something more worthwhile.
13:44:19
Shinmera
For instance one thing I'm aiming for is to allow stuff like non-hierarchical file-systems, where hierarchical file-systems are actually described as a set of nested file-systems -- each directory is both a "file" and a "file system".
13:47:11
Shinmera
another important part is that it can be extended to in-memory stuff that's just exposed through the same protocol so you could also load stuff in and then address it the same way, still.
13:48:00
Shinmera
anyway, it just boils down to a generic protocol in the end, one implementation of which is the os file system.
15:11:16
SAL9000
Shinmera: no clue... tbh I've long given up on that kind of stuff in favour of a fileserver box running ZFS + CIFS
17:45:37
Shinmera
SAL9000: I have traumatic memories of SMB and CIFS and all those network file systems
17:46:14
Shinmera
Anyway, converting ext4 to btrfs was painless after installing a new version of btrfs-tools
17:46:21
SAL9000
Yeah, I'm aware. In my experience they suck *less* than Win native drivers for things
17:48:30
Shinmera
And I'd like to have as few partitions as possible so I don't have to arbitrarily split my shit or have dupes
17:49:22
Shinmera
not because of the driver, that's fine these days, but the user/permissions mapping and speed.
17:49:55
Shinmera
I'm hoping for WinBTRFS now. Installation was painless and it seems pretty robust except for advanced features I'm not intending on using anyway (compression/encryption/snapshotting)
17:51:32
SAL9000
let me know how that goes? it seems a cool idea but a bit too alpha for my taste despite the age of the project
17:52:38
SAL9000
point being, remember to take into account the technical competence (or lack thereof) of your target audience
17:53:10
Shinmera
My mom is mostly just living on her android phone. If I can set it up to sync a folder automatically it'll be just fine.
17:53:40
SAL9000
the impression I have is that it's designed to piggyback onto existing cloudy things and back them up in One Place
17:54:17
SAL9000
if you want turnkey-ish sync, consider Syncthing -- admittedly, I haven't had much success with running the phone-app in background sync mode.
17:54:45
SAL9000
on phone I've fallen back to explicitly opening it when I want to sync, and closing it afterward
17:57:32
SAL9000
I'd welcome a sync solution built with your usual level of simplicity & thoughtfulness, but that involves diving into the Void of phone apps and I'd like to keep you sane :-)
17:57:56
Shinmera
too late Seqread: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
17:57:58
Shinmera
Seqwrite: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
17:58:00
Shinmera
512Kread: (g=2): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1
17:58:02
Shinmera
512Kwrite: (g=3): rw=randwrite, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1
17:58:04
Shinmera
4kQD32read: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
17:58:06
Shinmera
4kQD32write: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
17:58:38
Shinmera
bw ( KiB/s): min=329045, max=365909, per=100.00%, avg=334263.00, stdev=10622.51, samples=21
17:59:18
Shinmera
bw ( KiB/s): min=357717, max=407552, per=99.90%, avg=370774.75, stdev=12170.05, samples=20
17:59:58
Shinmera
bw ( KiB/s): min=271018, max=325632, per=100.00%, avg=298973.21, stdev=11833.20, samples=24
18:00:40
Shinmera
bw ( KiB/s): min=329728, max=358400, per=100.00%, avg=341140.43, stdev=6902.89, samples=21
18:01:22
Shinmera
bw ( KiB/s): min=391496, max=396058, per=100.00%, avg=394567.67, stdev=1308.75, samples=18
18:02:04
Shinmera
bw ( KiB/s): min=219904, max=309520, per=100.00%, avg=280498.67, stdev=13917.66, samples=36
18:02:28
Shinmera
READ: bw=326MiB/s (342MB/s), 326MiB/s-326MiB/s (342MB/s-342MB/s), io=5000MiB (5243MB), run=15317-15317msec
18:02:32
Shinmera
WRITE: bw=362MiB/s (380MB/s), 362MiB/s-362MiB/s (380MB/s-380MB/s), io=5000MiB (5243MB), run=13795-13795msec
18:02:36
Shinmera
READ: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=5000MiB (5243MB), run=17139-17139msec
18:02:40
Shinmera
WRITE: bw=333MiB/s (349MB/s), 333MiB/s-333MiB/s (349MB/s-349MB/s), io=5000MiB (5243MB), run=15021-15021msec
18:02:44
Shinmera
READ: bw=385MiB/s (404MB/s), 385MiB/s-385MiB/s (404MB/s-404MB/s), io=5000MiB (5243MB), run=12989-12989msec
18:02:48
Shinmera
WRITE: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=5000MiB (5243MB), run=18301-18301msec
18:02:50
Shinmera
linus@linuzz ~ fio --loops=5 --size=1000m --filename=/run/media/fastchive/data.tmp --stonewall --ioengine=libaio --direct=1 \
18:03:04
Shinmera
Seqread: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
18:03:06
Shinmera
Seqwrite: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
18:03:08
Shinmera
512Kread: (g=2): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1
18:03:10
Shinmera
512Kwrite: (g=3): rw=randwrite, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1
18:03:12
Shinmera
4kQD32read: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
18:03:14
Shinmera
4kQD32write: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
18:03:46
Shinmera
bw ( KiB/s): min=629166, max=925696, per=100.00%, avg=711380.22, stdev=106519.95, samples=9
18:04:28
Shinmera
bw ( KiB/s): min=741376, max=1028101, per=100.00%, avg=853379.62, stdev=104903.38, samples=8
18:05:10
Shinmera
bw ( KiB/s): min=451711, max=788165, per=100.00%, avg=562305.08, stdev=132057.22, samples=12
18:05:19
gingerale
SAL9000: Shinmera says he's tried to leave and return but it's throttled by the server.
18:05:52
Shinmera
bw ( KiB/s): min=651264, max=937869, per=99.46%, avg=722899.10, stdev=84908.16, samples=10
18:08:12
SAL9000
okay so bracketed paste is an extra mode, which can be enabled by the application (i.e. weechat)
18:08:13
|3b|
ACTION thinks the "irc will kick you if you paste a bunch, so lets paste it really slowly" feature is somewhat a bad idea :p
18:13:51
Shinmera
Anyway, back to android: I've already been tainted by the pain of android development https://github.com/shirakumo/ocelot
18:13:51
Colleen
github.com/shirakumo/ocelot Website (HTML), Title: GitHub - Shirakumo/Ocelot: An Android client for Lichat
18:15:13
SAL9000
I recommend trying the following: start a new ssh session (w/o tmux), enable bracketed paste (see above), cat --show-all, then try to paste and see what your terminal is sending.
18:15:28
SAL9000
that will help you confirm whether (your version of) xfce4-terminal supports bracketed paste
18:16:53
SAL9000
Shinmera: there's also the ersatz solution -- install the 'multiline.pl' plugin for weechat, then your paste should in theory become a mulitline input which you can cancel.
18:19:33
SAL9000
as for Syncthing and mobile sync/background stuff in general... basically, you're stuck between a rock and a hard place, from what I can tell
18:19:57
SAL9000
on the one side, you have the battery management police trying it's best to find an excuse to kill your background process
18:20:36
SAL9000
from what I can tell anything that doesn't use Google's push-notification API (or alarms/timers, I guess?) is basically screwed over by those two aspects :(
18:28:24
SAL9000
Shinmera: you can either use Ctrl-U line by line, or unset plugins.var.perl.multiline.modify_keys so then weechat treats the whole multiline as one line, and Ctrl-U will delete the whole lot...
18:29:04
SAL9000
alternatively, you can use Ctrl-Down to return to an empty prompt immediately (weechat command history trick)
18:29:26
SAL9000
there's some settings for paste detection, but I can't test those myself because I'm running bracketed
19:10:11
SAL9000
Shinmera: something with "cabinet", maybe? although that might evoke windows' .cab file horrors...
19:10:50
Shinmera
but both packets and cabinets have the issue of implying it's about stuff within a logical file.
19:19:33
SAL9000
Perforce is (apparently) widely used in the games industry due to being relatively good at handling large binary blobs
19:20:37
SAL9000
it's very much a BigCorp thing, not something indie devs would want to use, if only because non-free clients
19:20:59
Shinmera
right, well, I'm not concerned about the name clash from a lisp library that implements a different concept.
19:24:21
SAL9000
so long as we don't end up, later, with a trivial-version-control compat library trying to abstract over legit/svn/perforce /s
19:30:39
Shinmera
Well, here's an initial draft. https://github.com/Shinmera/depot/blob/master/protocol.lisp
19:30:39
Colleen
github.com/Shinmera/depot/b... Website (HTML), Title: depot/protocol.lisp at master · Shinmera/depot · GitHub
19:43:05
Shinmera
Not sure how best to facilitate switching protocols when encountering an entry that could be handled more specifically
19:43:32
Shinmera
Meaning if you access an entry that's a zip file, it should be transformed into something that's also a zip-file-depot.
19:47:28
SAL9000
Shinmera: not sure where you're going with the transaction stuff... not all fs have transactional semantics?
19:48:37
SAL9000
as for zip files -- does depot itself know that entry is a zip-file? seems to me like that is a job for a magic(5) library
19:49:58
Shinmera
they don't, but it can be emulated and or it'll just write anyway and "succeed" every commit.
19:50:43
SAL9000
might be cleaner to emulate transactions as the depot library deferring everything until commit occurs?
19:50:59
SAL9000
so long as it's clear to the API user that it doesn't guarantee atomicity, of course
19:51:15
Shinmera
sure, my point is just that it's more general to do it like this than to present a raw streams interface.
19:51:48
SAL9000
point being that you're setting yourself up for feature-flags or real-transactions-p or SOMETHING
19:51:48
Shinmera
as for your other question, detecting whether an entry is denoting a certain type is a separate problem
19:53:49
SAL9000
maybe depot can have backends or hooks -- e.g. magic v.s. file extension -- which allows CHANGE-CLASSing the depot:entry into any of its subclasses?
19:54:48
Shinmera
that's what I'm asking. I can see three avenues: 1) a function that takes a "type" and returns a class to use. 2) a function that encapsulates the entry in another instance 3) a set of functions that can decide to "accept" the entry and change-class it
19:55:41
SAL9000
#1 seems quite limited. Does it matter for depot whether the caller does change-class or encapsulation or mixins...?
19:55:44
Shinmera
My issue with 1 and 3 is that you might need to multiplex the type depending on the "origin depot"
19:56:33
Shinmera
My issue with 2 is that you need a separate call to convert it to the depot that is denoted by the entry.
19:57:08
SAL9000
caller could theoretically do encapsulation + delegation so you don't need a separate call
19:58:09
SAL9000
(i.e. the wrapper object quacks like a depot:entry for all intents and purposes, without necessarily satisfying DEPOT:ENTRY-P)
19:58:18
Shinmera
Wait, no, 2) is bad because you could not transparently traverse through depots, since now you need to request depot conversion.
19:59:07
Shinmera
One idea of the system is that you can go like (entry* os-depot "home" "linus" "some.zip" "my-file.dat")
19:59:31
Shinmera
or, more transparently, "some" instead of "some.zip" where "some" could be a zip or a directory, or whatever.
20:02:38
Shinmera
though it would be more like (entry* http-depot '("org" "wikipedia" "en") "wiki" "common lisp')
20:05:44
Shinmera
another thing to consider is that, for instance if you open a depot that's a zip archive, from within another zip archive depot, it needs to know to "extract" that inner file first to be able to present it.
20:07:39
Shinmera
the issue is how to do it efficiently. The primitive way would be to read the entry into memory completely, then treat it as a zip archive.
20:08:17
Shinmera
but for zip depots that are backed by a standard fs that has streams and whatever, it's far more reasonable to use the underlying fs to access it so you /don't/ have to read it into memory.
20:09:04
Shinmera
another layer that should be broken is if you have an in-memory octet vector representation that the zip could be parsed from without having to copy the sequence.
20:11:50
Shinmera
what I'm gathering here is that we need a way to multiplex depending on the backing storage. so whatever converts an entry to another depot (or more specific other type, whatever) needs to be able to dispatch based on parent depot.
20:12:36
Shinmera
so the zip "plugin" would have a method for a directory depot and a method for a sequence-depot (that's sequence-backed) and a method for a generic depot that just reads the entry data in.
20:12:48
SAL9000
hold on, what if you model it as a set of "desirable" properties, which some depots cannot conserve?
20:14:48
Shinmera
Seems really annoying to work with. I'd much rather have some mixins that allow you to extend the protocol
20:16:06
SAL9000
right... but if the directory-depot is inside a zip file, you've lost zero-copy with respect to the backing store already...
20:16:32
Shinmera
you wouldn't have a directory-depot in a zip file, it'd have to be a zip-directory-depot.
20:18:07
Shinmera
I think what I'm seeing is that stuff that's not compressed can nest arbitrarily quite naturally, and stuff that is will "automatically" flatten the hierarchy back to an in-memory sequence anyway and continue from there.
20:19:20
SAL9000
I think there's some compressions where you can start at a keyframe-like thing and go from there
20:20:01
SAL9000
also some archive types (zip?) don't force you to decompress all files, just the files you want to read... right?
20:20:13
Shinmera
also I think it would be worthwhile to do the change-class/conversion as a separate method that's automatically invoked as soon as you try to open the entry but no earlier, to avoid stuff like listing entries opening tons of zip files.
20:20:55
Shinmera
When I say "decompress" here I don't mean when you open the zip depot, but when you access another zip in a zip.
20:21:34
SAL9000
right, which would (unless you go crazy with the layer-breaking) force you to decompress the whole 2nd-level zip file from the 1st-level one
20:23:14
SAL9000
yes, but it's still a zip-in-zip and "default" behaviour still "decompresses" it into a sequence
20:23:46
Shinmera
well the zip backend can make use of the compression attribute to figure out what to do.
20:24:15
Shinmera
or actually, requesting an entry from the depot will automatically figure out what to do
20:25:25
SAL9000
child tries to do zero-copy read, if parent is "store" it works, if parent is compressed it "fails" by making a new sequence...
20:26:43
SAL9000
parent constructs new sequence, stores in a cache slot, which is then used to serve future zero-copy calls.
20:26:58
Shinmera
zip1 itself can't be compressed since it has to be decompressed to see what it contains. zip2 in zip1 can be compressed and will only decompress if needed.
20:28:13
SAL9000
"fails" in that it ends up having to do more work -- decompress the whole zip -- than the child may immediately need
20:29:29
SAL9000
if zip2 (child) is not compressed, zip1 (parent) can read direct from backing store just those bytes that child wants
20:30:45
Shinmera
zip1 would not do anything. zip2 is an entry that contains a stream and an offset in that case.
20:31:43
SAL9000
in your draft protocol, WRITE-TO/READ-FROM start/end parameters are referring to the sequence, not the entry, right?
20:34:53
Shinmera
this entry/depot knows it's coming from a directory-depot and as such internally has a file-stream to do its stuff with
20:35:54
Shinmera
when you request a list of entries, it parses the zip file structure and generates zip-entry-depots, each of which know they're coming from a zip file, so they have the corresponding zip entry structure in them.
20:35:54
SAL9000
so... you end up with a zip-file-depot-from-directory-depot, zip-file-depot-from-zip-file-depot, etc.?
20:37:23
Shinmera
so that entry is a zip-file-depot with a file-stream that starts at a certain position
20:37:55
SAL9000
Maybe I'm looking at it wrong, but I'm seeing that file-stream as a (severe) layering violation
20:38:43
SAL9000
Why does the child zip-file-depot need to know that it comes from a real file-system?
20:41:30
Shinmera
in your story the depot now also needs to have a table to store where entries start/end or whatever rather than that info and whatever else just more naturally being part of the entry.
20:42:30
Shinmera
well now you're forcing every entry to have a start/end index even when there's no need to have one
20:43:31
Shinmera
my point is, why not just store all the info that's related to how to get the entry data in the entry itself
20:45:40
Shinmera
yes because in the above scenario we have depots that are connected to file-streams.
20:45:56
SAL9000
so you either need to duplicate the code -- file-stream version vs sequence version
20:46:32
Shinmera
you don't want to do the latter because it's slow and would just lead to more copying.
20:46:55
Shinmera
having two variants is not much more effort and is far more efficient because now you can create sub-entries that just address sub-sequences.
20:47:31
SAL9000
...would it make sense to provide something that quacks like a sequence but is backed by a file-stream?
20:48:28
Shinmera
in zippy I have an "io" structure that has its own API and has two implementations for vectors or for streams.
20:49:08
Colleen
github.com/Shinmera/zippy/b... Website (HTML), Title: zippy/io.lisp at master · Shinmera/zippy · GitHub
20:51:31
Shinmera
it's still not "very fast" because each individual call of these functions will dispatch, rather than dispatching once and then the rest knows.
20:52:07
Shinmera
the defmethod is because size is used elsewhere and I was too lazy to separate them.
20:53:16
SAL9000
...right, you still don't win unless you have two (identical) copies of the body of the caller, under an etypecase
20:54:19
Shinmera
having some magic that eliminates further dispatchers once it knows the type would be nice.
20:56:27
Shinmera
but usually emitting typecase is "good enough" and while slow at compiling, SBCL should eliminate the branching if it knows the variable type.