1

I have made a chat application where for now I am storing complete history of chat of all the users.

I am using django as backend and postgres as the database. I am nearing 100k daily active users which makes around 1 million messages per day.

So I am wondering how to scale the postgres data horizontally? I have heard that sharding is not simple in SQL databases and also they have limit to scaling. Like I have heard that Google's big table can scale to 100 of petabytes while postgres is hard to scale to that level. Is it true? If not, how to scale at moment? Also, how to tackle with messages history, they will eventually get too big to handle?

Another question is should I shift to another dataset to handle scaling like mongodb or Cassandra or anything else, because it makes me fear that eventually I will have to scale to billions of messages per month level and if I can shift now that would be better. I don't want to over think or over analyse but just want to get perspective of how to go about it?

closed as primarily opinion-based by mustaccio, Colin 't Hart, Marco, kevinsky, LowlyDBA May 6 at 16:05

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Interesting candidate if you are looking for scalable postgres: citusdata.com/product/community. We are considering to use it in our own project. – Rémi Bonnet May 4 at 6:24
  • @RémiBonnet Should we consider some more scalable databases like Hbase or Cassandra? Can there be problem at later stage to shift from relational databases like Postgresql to databases like Cassandra? – hardik24 May 4 at 10:09
  • You could put "open source chat software" into your favourite search engine and see what they do. I would second @RémiBonnet 's recommendation of some sort of PostgreSQL - it has classic RDBMS, JSONB and H-Store (Key-Value). – Vérace May 4 at 10:12
  • @Vérace I completely agree and I searched online but I am unable to find any relevant information in regard to database – hardik24 May 4 at 10:34
  • This is the part that's going to require some work! :-( You're going to have to dig into the code and find out what they use - take a look at the schemas and make up your own mind. I know that Orafaq uses FUDforum and it seems to work great as a chatroom - a bit like here but exclusively for Oracle. You could probably adapt it to work with PostgreSQL? There's a shedload more here. HTH. p.s. welcome to the forum! :-) – Vérace May 4 at 10:39