2

I realise this question may have be answered in some form but I would like to gather the thoughts and opinions of actual users with experience in this field of Database design and development. I have a 'large' financial time series database, close to 150GB and growing at a rate of about 4GB per day. Currently I am using MySQL as my database engine, however I am concerned about scalability and performance in future. I have read conflicting reports on alternative solutions (MySQL, Postgres, Cassandra, Datomic, KDB+ etc etc), and with the 100s of Database technologies available one is left somewhat unsure. That's why personal experience of users I think would be invaluable.

Before anyone mentions the obvious such as KDB+, the issue I have with that is that it is predominantly an 'in-memory' database - not ideal for me. I understand it would be great for high frequency traders but that is not the nature of my research.I would prefer a scalable, on disk, SQL solution? Happy to hear thoughts and discuss.

closed as primarily opinion-based by RolandoMySQLDBA, Kin Shah, Max Vernon, Colin 't Hart, Vérace Jul 22 '15 at 16:25

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    Your big question is whether you can/wish to pay for a solution. The best Open Source SQL database is PostgreSQL with Firebird second and MySQL third - amazingly enough give MySQL's market share - does Betamax ring a bell? – Vérace Jul 22 '15 at 16:31
  • I would happily consider paying for solutions if its feasible. I've used MS SQL Server in the past, not a big fan of microsoft products, found them to be very buggy and clunky in design. The DB should also be compatible with the linux environment. – Greg Jul 23 '15 at 2:45
  • Anyone have comments with regards to MariaDb, heard very positive stuff lately also. Read its compatible with MySQL. – Greg Jul 23 '15 at 3:01
  • I'm with Axibase. We have a product called Axibase Time-Series Database that is quite efficient in terms of time-series storage, 4-6 bytes per sample: millisecond time and float value. It's not geared toward financial series though, there is no support for money datatype etc. Feel free to give it a try: axibase.com/products/axibase-time-series-database. It uses HDFS/HBase for raw storage. – Sergei Rodionov Jul 23 '15 at 8:43
  • @Greg - MariaDB aims to be an improved MySQL... PostgreSQL is already a vast improvement on MySQL. I'd go with Firebird before I'd use a MySQL (based) database. This is my opinion. – Vérace Jul 23 '15 at 16:49
1

In my opinion you must be, or you need, a DBA.
Not a single rdbms can solve problems with big data out of the box.

In your case partitions can be useful for example. So data compression, etc...

I can just suggest 3 alternatives:

  • SQL Server Enterprise Edition
  • Oracle Enterprise Edition
  • Postgresql

And the skills to manage data.

  • Not a DBA by trade, but its one aspect of what I am trying to do. Not sure what is defined as 'Big data' these days, but growing by 4GB per day for 5 days of the week for 12 months would be around 1 TB annually. Not sure what your experience is with mysql, could you comment on the performance vs postgressql for large datasets? – Greg Jul 23 '15 at 2:55
  • I work with large dataset, and I can tell you: everything is bad unless you don't go to optimize. In your case it is relevant how data is inserted and read. Anyway yes, postgresql can handle it, but you must start to look for inerithance and indexing. (all free with postgresql). Mysql , in my poor experience can be fast, but not reliable. – user_0 Jul 23 '15 at 8:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.