Timeline for What type of datastore should I use for high volume time series data?
Current License: CC BY-SA 3.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 24, 2012 at 7:19 | vote | accept | harm | ||
| Apr 22, 2012 at 14:28 | answer | added | ConcernedOfTunbridgeWells | timeline score: 5 | |
| Mar 23, 2012 at 9:39 | answer | added | Stephen Senkomago Musoke | timeline score: 1 | |
| Mar 23, 2012 at 9:20 | comment | added | harm | @ssmusoke I don't need the graphing to keep up. It would be cool but certainly not a requirement, if I had to choose I would like querying but better. How would would you load the data in two steps and in what DB? | |
| Mar 23, 2012 at 9:18 | comment | added | harm | @MyrddinEmrys That is correct. The write portion (not read) of the DB needs to keep up. | |
| Mar 23, 2012 at 7:06 | comment | added | Stephen Senkomago Musoke | @harm Are you also stating the graphing has to be instantaneous as the data comes in? If not then you need a two step process for loading the data into the database | |
| Mar 22, 2012 at 23:25 | comment | added | Myrddin Emrys | So your answer is that batch processing isn't a good choice because the measurements won't ever stop, correct? They will be processed constantly, so the read portion of the database needs to keep up? | |
| Mar 22, 2012 at 16:22 | comment | added | harm | I would be willing to do it in two steps. But wouldn't this lead to a lagging database? There are 50k writes coming in, if the db writes them slower then that it would introduce an ever bigger lag. | |
| Mar 22, 2012 at 15:08 | comment | added | FrustratedWithFormsDesigner | Would you be willing to do this in two stages? First stage is just to write the >50k records to a file, and then the second stage would be to process and load that to a database, but possibly at a rate slower than 50k/s. Basically, does it need to go to the database immediately? | |
| Mar 22, 2012 at 13:45 | comment | added | harm | Querying would be nice (eg select everything above a certain int). But initially having a list of numbers would be cool. | |
| Mar 22, 2012 at 12:29 | history | edited | Joel Brown |
edited tags
|
|
| Mar 22, 2012 at 9:59 | comment | added | Philᵀᴹ | Is graphing the data your only requirement, or do you want to do some kind of querying on the data? | |
| Mar 22, 2012 at 9:48 | history | asked | harm | CC BY-SA 3.0 |