I manage an application which has a very large (nearly 1TB of data with more than 500 million rows in one table) Oracle database back end. The database doesn't really do anything (no SProcs, no triggers or anything) it's just a data store.
Every month we are required to purge records from the two of the main tables. The criteria for the purge varies and is a combination of row age and a couple of status fields. We typically end up purging between 10 and 50 million rows per month (we add about 3-5 million rows a week via imports).
Currently we have to do this delete in batches of about 50,000 rows (ie. delete 50000, comit, delete 50000, commit, repeat). Attempting to delete the entire batch all at one time makes the database unresponsive for about an hour (depending on the # of rows). Deleting the rows in batches like this is very rough on the system and we typically have to do it "as time permits" over the course of a week; allowing the script to run continuously can result in the a performance degradation that is unacceptable to the user.
I believe that this kind of batch deleting also degrades index performance and has other impacts that eventually cause the performance of the database to degrade. There are 34 indexes on just one table, and the index data size is actually larger than the data itself.
Here is the script that one of our IT people uses to do this purge:
BEGIN LOOP delete FROM tbl_raw where dist_event_date < to_date('[date]','mm/dd/yyyy') and rownum < 50000; exit when SQL%rowcount < 49999; commit; END LOOP; commit; END;
This database must be up 99.99999% and we've only got a 2 day maintenance window once a year.
I'm looking for a better method for removing these records, but I've yet to find any. Any suggestions?