we are a small internet provider and we are storing all our clients traffic data in a Mysql database. We insert data every few minutes in three tables like this:
capture_table
----------
client_id
date
time
data1
...
datan
---------
The data is always searched by date (Network stats).
After the tables were getting too big, we decided to use several tables to divide the data. We created three PHP scripts that ran every few minutes dividing the original data into 3 tables per month and creating an index table.
index_table
data_table_2014-01-1
data_table_2014-01-2
data_table_2014-01-3
...
...
Currently we have and about 2.5 million of rows per table for our current number of clients, and growing (15GB of data in 3 years).
The queries are done by joining the data from all the tables involved between two dates, using UNION ALL.
First question: Could have done it better? I'm thinking in table partitioning for example.
Now, we are going to move all the database to a new server, to use this new server only for these kind of data. We are thinking in using N servers, as we were doing with the tables. So the application should ask a server_index table to see which server stores that data, and then a table_index table to see which table. And finaly, UNION ALL the data.
As we have not started yet to prepare the servers or the scripts for the migration...
Second question: could we do it better?
Thanks a lot.