I have a timeseries pandas dataframe which dynamically increases the columns every minute as well as adds a new row:
Initial:
timestamp 100 200 300
2020-11-01 12:00:00 4 3 5
Next minute:
timestamp 100 200 300 500
2020-11-01 12:00:00 4 3 5 0
2020-11-01 12:01:00 14 3 5 4
The dataframe has these updated values and so on every minute.
so ideally, I want to design a database solution that supports such a dynamic column structure. The number of columns could grow to over 20-30k+ and since it's one minute timeseries, it will have 500k+ rows per year.
I've read that relational db's have a limit on the number of columns so that might not work here, but also, since I am setting the data for new columns and assigning a default value(0) to previous timestamps, I lose out on the DEFAULT param that's there on MySQL.
Eventually, I will be querying data for 1 day, 1 month to get the data for the columns and their values.
Please suggest a suitable database solution for this type of dynamic row and column data.