I'm running a large ETL operation on > 100,000 jsonl files. I'm using RabbitMQ to insert filepaths into a queue and then using 25 workers to transform the files and perform batch inserts into a number of Postgres tables.
I've noticed that if I crank the number of workers up, I may end up with less data at the end, but no errors pop up.
Is it possible to "overload" the database instance and lose data?
- At the end, the largest table should have 2B rows
- Average table size is 250M rows
- Using the npm package
node-postgresto handle the inserts