2

I have a semi-large (around 30 million records) key/value list which I want to load into redis.

The data is stored in PostgreSQL (as master data) and we want to distribute this to clients for faster (local) lookups.

The keys are Belgium telephone numbers (national format, so strings between 9 and 10 digits). The values are 5 letter strings.

There would be a redis-server on each client (around 10 clients per customer). The full dump will distributed once or maybe twice a year, while the updates are between 100-1000 a day (only business days between 08:00 and 18:00).

Is this an efficient way to do this?

Side remark: if redis isn't the correct choice, other suggestions are also welcome.

  • That depends on what you mean by efficient... Are you looking to have a local Redis instance on each client with the entire dataset or a single shared one? Are you pushing the entire thing in one batch, or are the incremental updates as well? Are the any updates or is the data immutable? – Itamar Haber Apr 20 '15 at 22:50
1

An RDB would be the fastest way to replace/create an entire dataset. You can prepare the bi-annual forklift update beforehand at HQ at your convenience and ship it out to the customers. Do note that loading from RDB requires downtime.

As for the 0.1-1K updates/day per client-local Redis database - some of the real questions are:

  1. How many updates are done to the master database?
  2. Are all updates applied to all client/local databases?
  3. Can the updates be batched?
  4. Is there a latency budget for updates to be shipped to clients?
  5. 10 clients/database per customer, but how many customers overall?

I assume there are many possible solutions to this challenge, but I'm for KISSing it pimple :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.