How to simplify schema for multi database / hundreds of millions of rows data set

Question

We currently have a MySQL data set that was started about 10 years ago that aggregates a log of data, and accumulates millions of rows each month. The people that developed our system started separating the data into months, however they did this by creating an entirely new database with that month name (e.g. jan2001, mar2009, aug2012), and one table in each database named "detail". Each month a cronjob kicks off and spins up and entirely new database, with the table "detail", and the application starts writing the data to said table.

At this point the combined number of rows in all the "detail" tables is 448,231,722. This data is only used for analytics purposes at this point, so the speed of the query returning the data does not need to be instantaneous. One assumption we have is that the data will never be deleted.

I am wondering if you could suggest a better method of condensing or consolidating the data into a more manageable schema?

Answer 1

When it comes to databases, less is more.

You are doing the right thing by archiving off old data as this will speed up the performance and reduce the maintenance time of the active database.

The archived data needs to be considered. Keeping it in a separate database instance will be beneficial because backups/restoration and other maintenance activities will be completed separately - reducing the effect on the live data.

You should consider keeping the archived data on a physically different disk to the live data - this will avoid conflicts if both are being used at the same time. Also consider the type of disk this the archive is on as this has cost implications - does it really need to be on a raid type array or is a single disk and backup tape/dvd all that is needed? Storage may be relatively cheap, but it is still a cost in $, IO, time and network.

Next take a look at the archived data - do you really need to keep all the records? do you really need to keep all the columns of data? Would changing the structure of the data result in a smaller disk footprint/faster read query time? Can you summarize the data? Archived data for data mining does not always need to have the same structure as live data.

Keeping separate months data in separate databases may be a good way to go, but there other options that may make the analysis easier/quicker. Your options here include keeping the data in one database instance, but with table per month. Another is to have one set of archive tables and use table partitioning (read the manuals).