Return to Answer

I believe this would be possible with a relational database but throughput will be an issue.

The best form of storing the data will be trade off between simplicity and storage efficiency.

Given the sheer volume of data, If the observations were taken at the same second. I think it would make sense to save each engine in its own column. This would result in 25,000 rows rather than 500,000 rows for 1 seconds worth of data.

Edit: However as the observation time will vary then it would make more sense to store each series in its own row. Although this will result in a massive volume of data, Storing each observed value in its own field will make reporting and analysis much easier. However long term I don't think this would be feasible.

The next challenge is then being able to make use of this data. Realistically a human can't interpret that volume of raw data. And so Aggregating data would make sense, So for a given sample record the max, min, avg, std dev etc.

I believe this would be possible with a relational database but throughput will be an issue. SQL Server In Memory Optimised Tables could be very useful for this.

The best form of storing the data will be trade off between simplicity and storage efficiency.

Given the sheer volume of data, If the observations were taken at the same second. I think it would make sense to save each engine in its own column. This would result in 25,000 rows rather than 500,000 rows for 1 seconds worth of data.

Edit: However as the observation time will vary then it would make more sense to store each series in its own row. Although this will result in a massive volume of data, Storing each observed value in its own field will make reporting and analysis much easier. However long term I don't think this would be feasible.

The next challenge is then being able to make use of this data. Realistically a human can't interpret that volume of raw data. And so Aggregating data would make sense, So for a given sample record the max, min, avg, std dev etc.

I believe this would be possible with a relational database but throughput will be an issue.

The best form of storing the data will be tradeoff between simplicity and storage efficiency.

Normally i would save each series in its own row but in this case (given the sheer volume of data) I think it would make sense to save each engine in its own column. This would result in 25,000 rows rather than 500,000 rows for 1 seconds worth of data.

The above is assuming all observations were taken at the same second. If the observations of each engine are at random and unpredictable intervals then this wouldnt be of much use.

Storing each observed value in its own field will make reporting and analysis much easier.

The next challenge is then being able to make use of this data. If aggregation is of use then an OLAP cube could be very useful.

I believe this would be possible with a relational database but throughput will be an issue.

The best form of storing the data will be trade off between simplicity and storage efficiency.

Given the sheer volume of data, If the observations were taken at the same second. I think it would make sense to save each engine in its own column. This would result in 25,000 rows rather than 500,000 rows for 1 seconds worth of data.

Edit: However as the observation time will vary then it would make more sense to store each series in its own row. Although this will result in a massive volume of data, Storing each observed value in its own field will make reporting and analysis much easier. However long term I don't think this would be feasible.

The next challenge is then being able to make use of this data. Realistically a human can't interpret that volume of raw data. And so Aggregating data would make sense, So for a given sample record the max, min, avg, std dev etc.

I believe this would be possible with a relational database but throughput will be an issue.

The best form of storing the data will be tradeoff between simplicity and storage efficiency.

Normally i would save each series in its own row but in this case (given the sheer volume of data) I think it would make sense to save each engine in its own column. This would result in 25,000 rows rather than 500,000 rows for 1 seconds worth of data. (Assuming all observations were taken at the same second)

Storing each observed value in its own field will make reporting and analysis much easier.

The next challenge is then being able to make use of this data. If aggregation is of use then an OLAP cube could be very useful.

I believe this would be possible with a relational database but throughput will be an issue.

The best form of storing the data will be tradeoff between simplicity and storage efficiency.

Normally i would save each series in its own row but in this case (given the sheer volume of data) I think it would make sense to save each engine in its own column. This would result in 25,000 rows rather than 500,000 rows for 1 seconds worth of data.

The above is assuming all observations were taken at the same second. If the observations of each engine are at random and unpredictable intervals then this wouldnt be of much use.

Storing each observed value in its own field will make reporting and analysis much easier.

The next challenge is then being able to make use of this data. If aggregation is of use then an OLAP cube could be very useful.