Timeline for What type of datastore should I use for high volume time series data?
Current License: CC BY-SA 3.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 22, 2012 at 13:22 | comment | added | ConcernedOfTunbridgeWells | @Myrddin Emrys - If the OP's application can cope with batched writes than bulk inserts will be much faster. | |
| Mar 24, 2012 at 6:48 | comment | added | Stephen Senkomago Musoke | @MyrddinEmrys In my experience this method is an improvement, since it allows you to batch and throttle the inserts of the data. If the database slows down for some reason you do not lose data from the inserts. Also you can have indexes on the table then disable index updates as you load the data files and enable them after. | |
| Mar 23, 2012 at 16:12 | comment | added | Myrddin Emrys | That still doesn't answer the question of whether this method is an improvement over a direct connection to the database and normal inserts, particularly since the user presumably must have indexing or they will be unable to get any useful information back out of the database... this is 129 billion entries per month at this rate. Do you want to query a 129 billion line table with no index? | |
| Mar 23, 2012 at 16:04 | comment | added | Stephen Senkomago Musoke | When you use loading you can queue the files to be loaded as fast the database can handle them, but the files will be loaded. If you remove keys and index definitions from the tables you may be able to get that level of performance. | |
| Mar 23, 2012 at 15:56 | comment | added | Myrddin Emrys | Can MySQL and/or PostgreSQL handle a 50k line LOAD DATA INFILE once per second? Is that command faster than direct writes to the database? | |
| Mar 23, 2012 at 9:39 | history | answered | Stephen Senkomago Musoke | CC BY-SA 3.0 |