Questions tagged [batch-processing]

Processing an operation in chunks rather than the whole set in one operation. Typically applied to DELETE, UPDATE, and INSERT commands to improve concurrency and reduce resource usage.

Filter by
Sorted by
Tagged with
0
votes
1answer
26 views

how to call lo_unlink(lo) in batches

In Postgres 9.5 I am tying to reproduce the postgres tool vacuumlo to get rid of unreferenced large object to free disk space. References are stored in text type columns by hibernate so using the tool ...
0
votes
1answer
32 views

How to assign GUID to batches of data based on row counts and tunable batch size

I need to assign batches of rows a common GUID id (for consumption by an external process). Below is a simplified setup describing what I'm looking for: BASE_TABLE represents one of pre-existing ...
6
votes
1answer
541 views

How to avoid Table Lock Escalation?

I've got a task to update 5 million rows in a production table, without locking down the whole table for extended time So, I used approach that helped me before many times - updating top (N) rows at a ...
3
votes
3answers
158 views

what files should I exclude/add/copy mysql dbs (without mysqldump)

I need to have an exact copy of my databases in a clone VM of my server (just in case the server goes down, have the VM ready with the dbs). I know that mysqldump is the best alternative to migrate, ...
0
votes
0answers
17 views

Drop or truncate a temporary table during LOAD DATA?

I'm using LOAD DATA to batch csv content into a TEMPORARY table, with deactivated auto-commit. Within the same transaction, I then INSERT... ON DUPLICATE KEY UPDATE the content from temp table into ...
0
votes
1answer
14 views

What is the safe way to update production data if data found inconsistent

I am in a small company and require to fix a bunch of data in production which are inconsistent. I have written the script to handle the fix. I understand that writing sql to fix data is way more ...
2
votes
1answer
219 views

PostgreSQL Copy command internals

I need to know some internals about the copy command in PostgreSQL 10. In particular I need to know if the copy from into a remote database makes batch insertions. If so, what is the default ...
1
vote
2answers
846 views

Mysql innodb table dead lock while batch update in multiple thread

I've two tables of innoDB type and I'm updating one table using batch statement from 10 thread (one batch statement per thread) but some time I'm getting dead lock. java.sql.BatchUpdateException: ...
0
votes
4answers
180 views

Dynamically generate batched data for import

I need to execute a test in which I have to simulate 20 years' historical data in PostgreSQL (and TimescaleDB) DB. My problem is that I have to generate .sql files and ingest them (using psql-client) ...
2
votes
4answers
8k views

Delete from table in chunks

I want to cleanup one of my tables which has 4 million. So I prepare this query to clean it. But it's throwing an error, can anyone please help me to fix this? DELIMITER $$ DROP PROCEDURE IF EXISTS ...