2

I have a very large log table from which I want to copy rows, putting them into the same table structure on a new server. I don't want to copy everything, only old rows, so that the table on the main server stays small-ish. So I have to SELECT the data I want and only move (and delete) that.

Keep in mind that there is a lot of data, and I don't want to copy it all with a mysqldump if I don't need to, especially if it's going to lock the table while it's being copied.

The best I've come up with is a PHP script, which I will post as a tentative answer, although I'm certain it's not the best option.

3 Answers 3

6

You are looking for pt-archiver, part of the Percona Toolkit. The tool can purge rows from a table; write rows from a table to file; purge rows from a table while copying them to another table (possibly another server)...

It does so in chunks, so like 1,000 rows at a time, so that there is no high load on your server.

1

Please bear with me as I am kind of sluggish with PHP

//Connect to both databases, which are set up the same
$localDB = new mysqli("localhost", $username, $password, "db"); 
$remoteDB = new mysqli("remotehost.com", $username, $password, "db");
$maxDate = //some timestamp in the past

//Set up query to get the data
$getQuery = "SELECT message FROM `log` WHERE timestamp < ?";
$getStmt = $remoteDB->prepare($getQuery);
$getStmt->bind_param('i',$maxDate);
$getStmt->execute();
$getStmt->bind_result($message);

//Set up query to insert the data 20 at a time
$commitCount = 0;
$commitLimit = 20;
$putQueryPrefix = "INSERT INTO `log` (message) VALUES ";
$putQuerySuffix = "";
$comma = "";

//Read many rows and insert them in bulk
while ($getStmt->fetch()) {
  $putQuerySuffix .= "$comma('$message')";
  $comma = ",";
  $commitCount++;
  if ( $commitCount == $commitLimit ) {
    $putQuery = $putQueryPrefix . $putQuerySuffix;
    $putStmt = $localDB->prepare($putQuery);
    $putStmt->execute();
    $commitCount = 0;
    $putQuerySuffix = "";
    $comma = "";
  }
}
if ( $commitCount > 0 ) {
  $putQuery = $putQueryPrefix . $putQuerySuffix;
  $putStmt = $localDB->prepare($putQuery);
  $putStmt->execute();
}

//And we're done
$getStmt->close();
$putStmt->close();

//Now delete it from the old server
$result = $remoteDB->query("DELETE FROM `log` WHERE timestamp < $maxDate");
$result->free();

$remoteDB->close();
$localDB->close();

The general idea is this: mysqldump preps multiple rows for a single INSERT statement. This is known as an extended INSERT. My example shows how to create a group of 20 rows to perform such an INSERT. If you want to increase $commitLimit to something greater than 20, you may have to increase max_allowed_packet on the Remote Server.

0
//Connect to both databases, which are set up the same
$localDB = new mysqli("localhost", $username, $password, "db"); 
$remoteDB = new mysqli("remotehost.com", $username, $password, "db");
$maxDate = //some timestamp in the past

//Set up query to get the data
$getQuery = "SELECT message FROM `log` WHERE timestamp < ?";
$getStmt = $remoteDB->prepare($getQuery);
$getStmt->bind_param('i',$maxDate);
$getStmt->execute();
$getStmt->bind_result($message);

//Set up query to insert the data
$putQuery = "INSERT INTO `log` (message) VALUES (?)";
$putStmt = $localDB->prepare($putQuery);
$putStmt->bind_param('s',$message);

//Read each row and insert it
while ($getStmt->fetch()) {
  $putStmt->execute();
  $putStmt->fetch();
}

//And we're done
$getStmt->close();
$putStmt->close();

//Now delete it from the old server
$result = $remoteDB->query("DELETE FROM `log` WHERE timestamp < $maxDate");
$result->free();

$remoteDB->close();
$localDB->close();

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.