0

I'm working on a geo-replicated web platform, composed of an ecosystem of microservices. We need to improve and rework the user activity tracing pipeline and I'm looking for "the best" database to achieve that.

Our platform relies entirely on kubernetes, for that reason we would exclude every technology that is not compatible with this approach.

Each log is quite simple and potentially made of the following data:

  • timestamp
  • user_id
  • action_type
  • description
  • some metadata, their format can be adapted to choice of the database (json, key-value, and so on)

Goals

  • high available write operation
  • excellent write-scale capability
  • excellent storage scalability

Plus

  • geo-replication
  • cloud native
  • hot-warm architecture/some kind of rotation

Non goals

  • complex data aggregation
  • complex search query
  • batch processing

Based on my knowledge, researches and experience, good candidates would be:

  • cassandra: should satisfy all the goal and geo-replication
  • cockroach: I've never used it before, but based on the documentation all the goals should be satisfied + geo-replication and is cloud native
  • influxDB: Not sure about that, I've been using influxdb for a while and though it should satisfy all the goals and all the plus, maybe is not the best choice for this kind of data

what I would not choose:

  • elasticsearch: it does a lot of things I don't need, is tricky to be maintained and set up, is very resource-consuming
  • mongodb: the write scalability can be achieved only with sharding, this configuration is hard to be maintained and evolved, the shard key is tricky to be changed. Not fully HA due to the master-slave election mechanism
  • all the classic SQL with a single master

UPDATE:
good candidate for functionality but is a monster (and I don't know how it works with kubernetes)

  • HDFS
2
  • I hate to say it, but this might be a case for... HDFS? – bbaird Jul 31 '20 at 12:59
  • And I hate to hear that :) Anyway it could be, I really don't have any experience with HDFS + kubernetes. If it works good, I'll consider that monster :) Thanks for your opinion (I'll update the question adding HDFS) – Federico Bevione Jul 31 '20 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.