I am looking for a key value store to use as a LRU cache for my application. It needs to hold a lot of data (100G - 1000G) which is why I say disk based. Things I have found via Google seem to be focussed on being in memory.

Other requirements:

  • Automatically evict the least recently accessed keys when full
  • Highly available (data replicated across 3 nodes, service must work if 2 nodes are up)
  • 100k writes, 300k reads per hour
  • Values are binary (1k-100k more or less)
  • On premise installation on Ubuntu
  • Low maintenance (this is for a side-project)
  • Cheap/free (this is for a side-project)
  • Reliable (reloading the data from source will be very slow, so the cache losing its contents is bad)

Currently I have all of the data in Apache Cassandra. I want to move the data to Backblaze and keep only the hot data local. On a cache miss I will fetch from Backblaze and add to the cache. New/updated keys will write to the cache and later to Backblaze (it might be down). Old data is very rarely accessed.

Cassandra is working very well but I need to add nodes more frequently than I would like as the cluster fills up (currently have 6). The values stored at Backblaze will contain a collection of approx 20 related keys.

Any suggestions? Thanks.

  • Little confused with the terms cache and disk based in the same context. When an item gets evicted where does it go, being we're already at the disk level? How does it return? Why not store the entirety of the data on disk, especially when you say "Reliable (reloading the data from source will be very slow, so the cache losing its contents is bad)".
    – J.D.
    Apr 18 at 0:02
  • I instead to keep all the data at Backblaze (like S3 but much cheaper). So the "disk based cache" is for the hot recently accessed part of the data set. Most of this data is cold. Apr 18 at 3:31


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.