Here is my situation:
- I have a WebSocket connection that feeds me data real-time
- I have an async callback function that gets the data and inserts into a queue
- I have another thread that reads from the queue and stores it into the Postgres database, one row at a time (using python's psycopg2 library)
The problem is the real-time data comes in faster than the time it takes to insert into the database, and in a couple of hours, the server runs out of memory. (Is this because psycopg2 library is slow?)
A simple solution is to create more threads that insert into the database; however, this will cause the data to be out of order. Is there a database that sorts the data? Other suggestions would be very much appreciated.