freenode/lisp - IRC Chatlog
Search
9:24:26
beach
p_l: Yes, but many data sets now fit in the RAM of a reasonable PC. For instance a complete social-security database for a country, the register of all automobiles in a country, the bank transactions of all customer of a particular bank, etc.
9:25:54
beach
p_l: But I often see inexperienced project leaders and developers that are unable (or too lazy) to make this back-of-the-envelope calculation, so they will pay way too much money for something like Oracle without actually needing many of its features.
9:26:57
beach
p_l: Furthermore, when I do the calculation for them, they get *really* angry, as if, deep down, they didn't want to know the truth.
9:26:59
p_l
beach: dunno about other countries, but in many cases the issue is not "will it fit in naive memory implementation" but "will it not eat data, is tested to hell and back, can survive direct aircraft strike (not joking)
9:28:06
TMA
beach: that's not entirely accurate. Given that there are several billion transactions even in a small bank, just storing their ID would eat a good chunk of RAM
9:28:40
p_l
UV3000 is not a "reasonable PC", and one of its marked use cases is running single-instance DB for single ERP instance
9:31:17
TMA
beach: if you consider that each transaction is several hundred bytes worth of data, that makes the total several hundred gigs -- that's not a "reasonable pc" anymore
9:32:52
p_l
there are specialized banking DBs used for decades, often with data storage systems tailored for particular bank
9:33:24
p_l
Oracle, SAP HANA, etc. etc. shows up in analysis applications or various non-account parts
9:34:16
p_l
for example, in a project I work with, we use distributed in-memory store from Oracle (it was acquisition) for various "funds" data (it's called even "General Funds Manager")
9:34:42
beach
Despite what shka says, if I ever get the time, I should read up on all these issues and then form an opinion about the current state of things.
9:35:20
p_l
some of the most costly (in licensing and hw) systems are on the other hand low-memory (comparatively) but very highly available and more concerned with huge I/O rates as they talk all over the world
9:36:29
p_l
beach: generally, there's a lot of in-memory stores in play in OLAP, while many OLTP tasks are low-memory intensive but need disk for persistence and I/O for comms
9:36:49
p_l
meanwhile with OLAP it's all about "load as much data as you can so we can analyse it in all directions"
9:38:06
p_l
shka: well, yes, but I am not talking about lazy programmers who go with "what is easiest without considering drawbacks"
9:38:48
shka
what could be done and should be done is distributed, P2P, transactional, in-memory storage
9:47:10
shka
well, until big business will be able to swap developers in their lisp project, you won't see lisp projects in big business
10:26:58
p_l
shka: now consider that some of those machines require 2x the memory vs. what you can use, due to RAIM in mirror mode
10:27:25
p_l
phoe: "on the street" is the pricing without "enterprise markup" (actually an extra guarantee among other things, important when dealing with that kind of money)
10:28:38
p_l
and if you deal with things like z-series mainframes you have a machine that has 2.5 the amount of memory you can maximally spec, because mirror RAIM and that .5 for internal use by machine itself
10:31:10
p_l
I heard anecdotes of a big z-Series losing whole I/O drawer under max load and it having not affected the application
10:33:47
flip214
well, we've got a customer who didn't notice that his RAID controller was dead for two weeks, because the data was accessible via the software-mirror (DRBD)
10:35:23
p_l
flip214: this is more a case where the I/O of the system, which was under load (including things like mirroring to another machine) was unaffected
10:36:06
p_l
IO drawer is where things like Ethernet, Infiniband and FC (plus occassional ancient ESCOM) cards sti
10:37:31
p_l
reliable in-memory systems tend to be expensive as well, especially when you don't want to directly deal with disk (either huge expensive NUMA systems, or RDMA)
10:53:52
shka
p_l: RDMA is getting cheaper, and besides, even without RDMA you can build reasonable system
10:56:16
p_l
shka: yes, but the latencies make it very far from "let's just think of memory instead of multiple completely disparate systems"
10:57:00
p_l
and in my experience, companies balk when I suggest "ok, let me hit ebay, I'll have the 56GBit low-latency fabric ordered by tomorrow on the cheap" ;)
10:59:19
shka
anyway, those would be different beast, i agree, but this approach has some advantage over ultra expensive huge IO mainframes
10:59:56
p_l
most companies that lease mainframes have very specific reasons for doing so (combination of RAS requirements, power, software base, etc.)
11:01:05
flip214
you can get 56gbit IB, 2 dual-port adapters and 2 cables (so two machines can talk 112gbps) for < €1000
11:03:05
flip214
so all of the data needs to have been seen before the first data byte can be transmitted
11:04:25
p_l
shka: there are some low-level details that impact how fast you can go with Ethernet without changing it into non-Ethernet, including how it's designed for Store-and-Forward data
11:05:25
p_l
shka: IB doesn't have store-and-forward inside subnet (most sites anyone sees are single subnet)
11:08:17
p_l
Netherlands used Myrinet on mixed Myrinet/10GbE links, in poland I think we did something shitty with plain IP... ;)
11:08:45
p_l
flip214: IB uses IPv6 *addressing*, and intra-subnet packets have 16bit address for wormhole routing
11:09:32
p_l
the 16bit addresses are internal affair of the subnet that software stack is not supposed to bother itself with (outside of management software for subnet itself)
11:11:54
p_l
it's much harder to do wormhole switching when you have to match complex 128bit addressing to ports ;)
19:27:33
rpg
I suspect that CL-DOT cannot handle subgraphs, but I'm not 100% sure. Can anyone confirm?