Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I have a very large log table from which I want to copy rows, putting them into the same table structure on a new server. I don't want to copy everything, only old rows, so that the table on the main server stays small-ish. So I have to SELECT the data I want and only move (and delete) that.

Keep in mind that there is a lot of data, and I don't want to copy it all with a mysqldump if I don't need to, especially if it's going to lock the table while it's being copied.

The best I've come up with is a PHP script, which I will post as a tentative answer, although I'm certain it's not the best option.

share|improve this question
add comment

3 Answers

up vote 4 down vote accepted

You are looking for pt-archiver, part of the Percona Toolkit. The tool can purge rows from a table; write rows from a table to file; purge rows from a table while copying them to another table (possibly another server)...

It does so in chunks, so like 1,000 rows at a time, so that there is no high load on your server.

share|improve this answer
add comment

Please bear with me as I am kind of sluggish with PHP

//Connect to both databases, which are set up the same
$localDB = new mysqli("localhost", $username, $password, "db"); 
$remoteDB = new mysqli("remotehost.com", $username, $password, "db");
$maxDate = //some timestamp in the past

//Set up query to get the data
$getQuery = "SELECT message FROM `log` WHERE timestamp < ?";
$getStmt = $remoteDB->prepare($getQuery);
$getStmt->bind_param('i',$maxDate);
$getStmt->execute();
$getStmt->bind_result($message);

//Set up query to insert the data 20 at a time
$commitCount = 0;
$commitLimit = 20;
$putQueryPrefix = "INSERT INTO `log` (message) VALUES ";
$putQuerySuffix = "";
$comma = "";

//Read many rows and insert them in bulk
while ($getStmt->fetch()) {
  $putQuerySuffix .= "$comma('$message')";
  $comma = ",";
  $commitCount++;
  if ( $commitCount == $commitLimit ) {
    $putQuery = $putQueryPrefix . $putQuerySuffix;
    $putStmt = $localDB->prepare($putQuery);
    $putStmt->execute();
    $commitCount = 0;
    $putQuerySuffix = "";
    $comma = "";
  }
}
if ( $commitCount > 0 ) {
  $putQuery = $putQueryPrefix . $putQuerySuffix;
  $putStmt = $localDB->prepare($putQuery);
  $putStmt->execute();
}

//And we're done
$getStmt->close();
$putStmt->close();

//Now delete it from the old server
$result = $remoteDB->query("DELETE FROM `log` WHERE timestamp < $maxDate");
$result->free();

$remoteDB->close();
$localDB->close();

The general idea is this: mysqldump preps multiple rows for a single INSERT statement. This is known as an extended INSERT. My example shows how to create a group of 20 rows to perform such an INSERT. If you want to increase $commitLimit to something greater than 20, you may have to increase max_allowed_packet on the Remote Server.

share|improve this answer
add comment
//Connect to both databases, which are set up the same
$localDB = new mysqli("localhost", $username, $password, "db"); 
$remoteDB = new mysqli("remotehost.com", $username, $password, "db");
$maxDate = //some timestamp in the past

//Set up query to get the data
$getQuery = "SELECT message FROM `log` WHERE timestamp < ?";
$getStmt = $remoteDB->prepare($getQuery);
$getStmt->bind_param('i',$maxDate);
$getStmt->execute();
$getStmt->bind_result($message);

//Set up query to insert the data
$putQuery = "INSERT INTO `log` (message) VALUES (?)";
$putStmt = $localDB->prepare($putQuery);
$putStmt->bind_param('s',$message);

//Read each row and insert it
while ($getStmt->fetch()) {
  $putStmt->execute();
  $putStmt->fetch();
}

//And we're done
$getStmt->close();
$putStmt->close();

//Now delete it from the old server
$result = $remoteDB->query("DELETE FROM `log` WHERE timestamp < $maxDate");
$result->free();

$remoteDB->close();
$localDB->close();
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.