Sack - sharding memory hash in perl

Main design goal is to have interactive environment to query perl hashes which are bigger than memory on single machine.

It's implemented using TCP sockets between perl processes. This allows horizontal scalability both on multi-core machines as well as across the network to additional machines.

Reading data into hash is done using any perl module which returns perl hash and supports offset and limit to select just subset of data (this is required to create disjunctive shards).

Views are small perl snippets which are called for each record on each shard with $rec. Views create data in $out hash which is automatically merged in output.

Tags

cloud hash memory perl sack sharding