1

I have decided to improve my well-being and in case being successful write a note about this and share it with my peers for free and try to help them improve there well-being as well. But on this road, I need foundational advice from the database practitioners.

Background Description:

In my research, I need to collect a lot of different types of time-series (GDP, Real GDP, Nominal GDP, Consumption, Investment, Percapita GDP, Number of Hours Work, Unit Labor Cost and many more). Right now, I am following and collecting about 155 time-series. Until now, I have been updating them manually as soon as an update was published on statistical web pages. I have been doing this using Excel spreadsheet (like, downloading new excel and then copying and pasting the necessary data point into my excel). However, this is a very daunting task, and it is prone to a lot of errors. Checking and finding the error is another nightmare. So, in total, maintaining this Excel database (updating checking and documenting) takes about 10-12 hours per week approximately.

My Task:

  1. I want to create a database where I will have all these and many more variables (I want each variable to have a name, description, source description (ideally with URL included), date of last update etc.) stored.
  2. I also want to be able to update the database automatically from downloaded Excel or CSV file (These files come in different formats from various webpages. Some are structure vertically some horizontally).
  3. I also want to have some error checkers for easier debugging the base.
  4. The series are of different frequencies (some are daily, some weekly, some monthly and some quarterly and yet some annual). I want to be able to easily convert from daily to monthly or from monthly to quarterly or vice versa using my own predetermined formula. (being it average or median or whatever the current task requires)
  5. I also want to then later to be able to easily query any variable or several of the variables and create a dashboard in various programs like MATLAB or Python or Julia to visualize dynamics and use them for regression or model estimations in these programs.

Question

For all the tasks described above which database management program would you recommend (ideally, it would be better to be it free and open source)

P.S. I tried writing MATLAB script for automatic updates, but it's very inconvenient.

P.P.S This is a cross-post from Cross Validated in hope to receive the answer somewhere.

Thanks, Giorgi.

As an example of how my data set looks like please see the following screenshot:

enter image description here

2
  • @EvgenyKuznetsov & G.T.: Cross-posting is discouraged in the SE network. Please choose one community and stick with it. Alternatively you can edit each copy of the question to make it tailored to that specific community rather than seeking for the same answer in both. Thank you.
    – Andriy M
    Dec 17 '20 at 9:01
  • @Andriy M Thanks. As the author explains the question was asked on Cross Validated site in hope to find someone who has already thought and tried database from statisticians. And it is highly possible. I suggested to the author to post here too. There are moments when sticking to the one site only is worse then cross-posting in hope to find better answer. Dec 17 '20 at 9:08
2

Your root question is subjective and it's a little hard to fully envision what your data looks like without some examples, but it sounds like a Relational Database Management System (RDBMS) could work for you based on the type of querying you'll be doing.

If you solely want something free than PostgreSQL. If you want something with the most features built into it out of the box and are ok with paying then Microsoft SQL Server. Both can house your data, have Views (and other types of objects) which can save your queries (predetermined formulas) for conversion, and support importing data from CSVs and other files if needed as well (though I recommend you do as much of your data gathering and importing into the database via a non-database language like Python).

Your additional requirements for creating dashboards and other ad-hoc querying will be supported by either of these databases as well.

4
  • Thanks for your answer. I have add screenshot from my excel file. Please check the question again.
    – G.T.
    Dec 17 '20 at 12:36
  • @G.T. Thanks, yes your data looks relational to me, so I think my answer should suffice. I personally like Microsoft SQL Server for all the features it has built into it, but PostgreSQL is a great free alternative.
    – J.D.
    Dec 17 '20 at 12:59
  • 1
    I checked it and it seems great. Will dive into it On Saturday. There are two other alternatives I just found out InfluxDB and kudu. What do you think of these two?
    – G.T.
    Dec 17 '20 at 13:20
  • 1
    @G.T. Unfortunately I don't have a lot of experience with those database systems but I have heard of InfluxDB at least. I would probably choose that over Kudu if those were my only choices. Keep in mind the more popular a system is, the more likely it'll be easier to accomplish your goals from the perspective that it already offers an out of box solution or there's a strong enough community around it for guiding you how to solve your problems. That's why I generally prefer enterprise solutions like Microsoft SQL Server, I haven't hit a problem it couldn't solve yet (including big data goals).
    – J.D.
    Dec 17 '20 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.