Preamble:
We use RDS at AWS to manage our MySQL database. I know that RDS creates binary logs used for replication.. I want to harvest these binary logs and store them on S3, so that at any time, i can restore to an off-site database and replay the actions.
The Problem:
I span up a new RDS instance, and tried the following.
root@aws_test_server:/root# mysqlbinlog -h testing.xxxxxxxx.eu-west-1.rds.amazonaws.com -u myuser -p --read-from-remote-server -t mysql-bin-changelog.000002
Enter password:
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
ERROR: Got error reading packet from server: Could not find first log file name in binary log index file
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
root@aws_test_server:/root#
So the binary log retrieves fine, however see the Error within the binary log.. What does this mean?
I googled for about 2 days and could find no mention of this error [anywhere].
My questions:
- Is this a good idea? (backing up binary logs to S3..)
- Can it be done in a better way? (~infinite point-in-time db recovery)
- What can I do about the error?
PS: this is happening for the production DB and this test DB
PS2: I note that this is not directly a programming question, however is related to it. Please do not move/close from SO to somewhere like serverfault. Thanks
Update 2/4/2014:
SHOW BINARY LOGS;
mysql> show binary logs;
+----------------------------+-----------+
| Log_name | File_size |
+----------------------------+-----------+
| mysql-bin-changelog.000002 | 1097 |
| mysql-bin-changelog.000003 | 469 |
| mysql-bin-changelog.000004 | 469 |
| mysql-bin-changelog.000005 | 594 |
| mysql-bin-changelog.000006 | 715 |
| mysql-bin-changelog.000007 | 412 |
+----------------------------+-----------+
6 rows in set (0.00 sec)
SHOW MASTER STATUS;
mysql> show master status;
+----------------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+----------------------------+----------+--------------+------------------+-------------------+
| mysql-bin-changelog.000007 | 412 | | | |
+----------------------------+----------+--------------+------------------+-------------------+
1 row in set (0.01 sec)
Command: mysqlbinlog -R -h $HOST -u ttc -p -j 0 --stop-position=412 -r file
Result:
Enter password:
Warning: option 'start-position': unsigned value 0 adjusted to 4
mysqlbinlog Ver 3.3 for debian-linux-gnu at x86_64
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Dumps a MySQL binary log in a format usable for viewing or for piping to
the mysql command line client.
Usage: mysqlbinlog [options] log-files
-?, --help Display this help and exit.
--base64-output[=name]
Determine when the output statements should be
base64-encoded BINLOG statements: 'never' disables it and
works only for binlogs without row-based events;
'decode-rows' decodes row events into commented SQL
statements if the --verbose option is also given; 'auto'
prints base64 only when necessary (i.e., for row-based
events and format description events); 'always' prints
base64 whenever possible. 'always' is deprecated, will be
removed in a future version, and should not be used in a
production system. --base64-output with no 'name'
argument is equivalent to --base64-output=always and is
also deprecated. If no --base64-output[=name] option is
given at all, the default is 'auto'.
--character-sets-dir=name
Directory for character set files.
-d, --database=name List entries for just this database (local log only).
--debug-check Check memory and open file usage at exit .
--debug-info Print some debug info at exit.
--default-auth=name Default authentication client-side plugin to use.
-D, --disable-log-bin
Disable binary log. This is useful, if you enabled
--to-last-log and are sending the output to the same
MySQL server. This way you could avoid an endless loop.
You would also like to use it when restoring after a
crash to avoid duplication of the statements you already
have. NOTE: you will need a SUPER privilege to use this
option.
-F, --force-if-open Force if binlog was not closed properly.
(Defaults to on; use --skip-force-if-open to disable.)
-f, --force-read Force reading unknown binlog events.
-H, --hexdump Augment output with hexadecimal and ASCII event dump.
-h, --host=name Get the binlog from server.
-l, --local-load=name
Prepare local temporary files for LOAD DATA INFILE in the
specified directory.
-o, --offset=# Skip the first N entries.
-p, --password[=name]
Password to connect to remote server.
--plugin-dir=name Directory for client-side plugins.
-P, --port=# Port number to use for connection or 0 for default to, in
order of preference, my.cnf, $MYSQL_TCP_PORT,
/etc/services, built-in default (3306).
--protocol=name The protocol to use for connection (tcp, socket, pipe,
memory).
-R, --read-from-remote-server
Read binary logs from a MySQL server.
-r, --result-file=name
Direct output to a given file.
--server-id=# Extract only binlog entries created by the server having
the given id.
--set-charset=name Add 'SET NAMES character_set' to the output.
-s, --short-form Just show regular queries: no extra info and no row-based
events. This is for testing only, and should not be used
in production systems. If you want to suppress
base64-output, consider using --base64-output=never
instead.
-S, --socket=name The socket file to use for connection.
--start-datetime=name
Start reading the binlog at first event having a datetime
equal or posterior to the argument; the argument must be
a date and time in the local time zone, in any format
accepted by the MySQL server for DATETIME and TIMESTAMP
types, for example: 2004-12-25 11:25:56 (you should
probably use quotes for your shell to set it properly).
-j, --start-position=#
Start reading the binlog at position N. Applies to the
first binlog passed on the command line.
--stop-datetime=name
Stop reading the binlog at first event having a datetime
equal or posterior to the argument; the argument must be
a date and time in the local time zone, in any format
accepted by the MySQL server for DATETIME and TIMESTAMP
types, for example: 2004-12-25 11:25:56 (you should
probably use quotes for your shell to set it properly).
--stop-position=# Stop reading the binlog at position N. Applies to the
last binlog passed on the command line.
-t, --to-last-log Requires -R. Will not stop at the end of the requested
binlog but rather continue printing until the end of the
last binlog of the MySQL server. If you send the output
to the same MySQL server, that may lead to an endless
loop.
-u, --user=name Connect to the remote server as username.
-v, --verbose Reconstruct SQL statements out of row events. -v -v adds
comments on column data types.
-V, --version Print version and exit.
--open-files-limit=#
Used to reserve file descriptors for use by this program.
Variables (--variable-name=value)
and boolean options {FALSE|TRUE} Value (after reading options)
--------------------------------- ----------------------------------------
base64-output (No default value)
character-sets-dir (No default value)
database (No default value)
debug-check FALSE
debug-info FALSE
default-auth (No default value)
disable-log-bin FALSE
force-if-open TRUE
force-read FALSE
hexdump FALSE
host ttc-testing.cz6c92dfxwoh.eu-west-1.rds.amazonaws.com
local-load (No default value)
offset 0
plugin-dir (No default value)
port 3306
read-from-remote-server TRUE
server-id 0
set-charset (No default value)
short-form FALSE
socket /var/run/mysqld/mysqld.sock
start-datetime (No default value)
start-position 4
stop-datetime (No default value)
stop-position 412
to-last-log FALSE
user ttc
open-files-limit 64
