I have an application that is very write heavy. It saves GPS data every second. The way I handle it at the moment is to store the data locally and upload it to s3 every 5 minutes. The database (dynamoDB) then only contains the metadata about the gps file.
Now I would like to get all the gps data into the db rather than just the metadata. I thought about a aws lambda function which automatically reads the gps file and saves the entries to the db, but this has one big problem.
DynamoDB is priced by write capacity (i.e. about 5$ per month for every 10 writes/s). So I either have to pay a lot of money to get the necessary capacity (5min = 300 entries/writes, multiplied by the amount of simultaneously uploaded files) or throttle the db writes to slowly process all the entries (this is not really an option since I need the file contents to get in the DB close to realtime).
Therefore I consider switching to AWS RDS postgresql since it does not charge per request and also allows multi item inserts. Does anybody have experience regarding the performance on a micro instance? Can it handle potentially hundreds of incoming multi row insert statements (about 300-1000 items each).
Another short question. I use csv to store the gps data since it uses much less space than json and therefore saves bandwith. What does the result of a select statement look like? Is it something like:
firstname,lastname steve,jobs michael,jackson
firstname: steve, lastname: jobs firstname: michael, lastname: jackson