I need to setup replication via SSL. 2 nodes master/master for H/A. Then 2 slave nodes for backup. Node 3 will slave off of node 1, and node 4 will slave off of node 2 for redundant backups. Nodes 3 & 4 will store backups to a directory mapped to S3 via S3FS. All nodes are physical.
We're using mariadb-5.5.41 server on CentOS 7 hosts.
It's a neat plan!
However I am unable to get authentication to work with ssl required of the user.
We already have a CA certificate and key established in the environment. And we used those to generate the cert and key to be used with mariadb.
This is the process we used to generate the cert/key:
openssl genrsa -des3 -out db1.example.com.key 4096
openssl req -new -key db1.example.com.key -out db1.example.com.csr
openssl x509 -req -days 3650 -in db1.example.com.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out db1.example.com.crt
Using those keys I put together this my.cnf file. Haven't gotten around to configuring replication yet, as I have yet to get ssl logins to work.
[root@db1:~] #cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd
ssl
ssl-ca=/opt/mysql/ca.crt
ssl-cert=/opt/mysql/db1.example.com.crt
ssl-key=/opt/mysql/db1.example.com.key
[mysqld_safe]
general_log_file=/var/log/mariadb/mariadb.log
general_log=1
log-error=/var/log/mariadb/mariadb_error.log
pid-file=/var/run/mariadb/mariadb.pid
log_slow_queries=/var/log/mysql/mysql-slow.log
long_query_time=2
log-queries-not-using-indexes
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
Made sure mariadb could read the cert files:
[root@db1:~] #ls -ld /opt/mysql/ /opt/mysql/*
drwx------. 2 mysql mysql 86 Jul 20 06:20 /opt/mysql/
-r--------. 1 mysql mysql 2212 Jul 20 05:14 /opt/mysql/ca.crt
-r--------. 1 mysql mysql 1956 Jul 20 05:17 /opt/mysql/db1.example.com.crt
-r--------. 1 mysql mysql 3247 Jul 20 05:15 /opt/mysql/db1.example.com.key
Verified that the ssl libraries were linked against the mysqld binary
[root@db1:~] #ldd $(which mysqld) | grep ssl
libssl.so.10 => /lib64/libssl.so.10 (0x00007f79669ba000)
And restarted mariadb. And if I take a look at my SSL variables in my mysql command line, everything is looking good.
MariaDB [mysql]> show variables like '%ssl%';
+---------------+---------------------------------+
| Variable_name | Value |
+---------------+---------------------------------+
| have_openssl | YES |
| have_ssl | YES |
| ssl_ca | /opt/mysql/ca.crt |
| ssl_capath | |
| ssl_cert | /opt/mysql/db1.example.com.crt |
| ssl_cipher | |
| ssl_key | /opt/mysql/db1.example.com.key |
+---------------+---------------------------------+
7 rows in set (0.00 sec)
If I create one replication user without SSL required, he is able to log in from node 2 to node 1 no problem:
MariaDB [mysql]> grant replication slave on *.* to 'slave1'@'db2.example.com' identified by 'secret';
Query OK, 0 rows affected (0.00 sec)
[root@db2:~] #mysql -uslave1 -p -h db1.example.com
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 15
Server version: 5.5.41-MariaDB MariaDB Server
Copyright (c) 2000, 2014, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
If I show grants on this user I can confirm that SSL is not required
MariaDB [mysql]> show grants for 'slave1'@'db2.example.com';
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| Grants for slave1@db2.example.com |
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| GRANT REPLICATION SLAVE ON *.* TO 'slave1'@'db2.example.com' IDENTIFIED BY PASSWORD '*somelongpasswordhash' |
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
If I create the second slave user with the SSL requirement, and even flush privileges, I can not log into node 1 from node 2
MariaDB [mysql]> grant replication slave on *.* to 'slave2'@'db2.example.com' identified by 'test' require ssl;
Query OK, 0 rows affected (0.00 sec)
MariaDB [mysql]> flush privileges;
Query OK, 0 rows affected (0.00 sec)
[root@db2:~] #mysql -uslave2 -p -h db1.example.com
Enter password:
ERROR 1045 (28000): Access denied for user 'slave2'@'db2.example.com' (using password: YES)
And if I look at the grants for this user I able to tell that ssl is required for him to be able to login:
MariaDB [mysql]> show grants for 'slave2'@'db2.example.com;
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Grants for slave2@db2.example.com |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| GRANT REPLICATION SLAVE ON *.* TO 'slave2'@'db2.example.com' IDENTIFIED BY PASSWORD '*somelongpasswordhash' REQUIRE SSL |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
If I take a look at the error log:
[root@db1:~] #grep error /etc/my.cnf
log-error=/var/log/mariadb/mariadb_error.log
[root@db1:~] #tail /var/log/mariadb/mariadb_error.log
150720 5:18:24 InnoDB: Initializing buffer pool, size = 128.0M
150720 5:18:24 InnoDB: Completed initialization of buffer pool
150720 5:18:24 InnoDB: highest supported file format is Barracuda.
150720 5:18:24 InnoDB: Waiting for the background threads to start
150720 5:18:25 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 120637807
150720 5:18:25 [Note] Plugin 'FEEDBACK' is disabled.
150720 5:18:25 [Note] Server socket created on IP: '0.0.0.0'.
150720 5:18:25 [Note] Event Scheduler: Loaded 0 events
150720 5:18:25 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
I see absolutely NO indication as to why the login for the 'slave2' user (that requires SSL) is failing.
And I notice that if I do a \s it tells me that ssl is not in use:
MariaDB [(none)]> \s
--------------
mysql Ver 15.1 Distrib 5.5.41-MariaDB, for Linux (x86_64) using readline 5.1
Connection id: 36
Current database:
Current user: root@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server: MariaDB
Server version: 5.5.41-MariaDB MariaDB Server
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: latin1
Db characterset: latin1
Client characterset: utf8
Conn. characterset: utf8
UNIX socket: /var/lib/mysql/mysql.sock
Uptime: 22 hours 9 min 2 sec
Threads: 1 Questions: 311 Slow queries: 0 Opens: 39 Flush tables: 3 Open tables: 24 Queries per second avg: 0.003
--------------
From the above output, there's this line:
SSL: Not in use
Why would that be the case, given the way I've configured this? How can I correct this?
Also 1) how to I bump up the verbosity on the logs so I can get an indication as to why this is failing? and 2) what is the best way to troubleshoot this?
OK so question 2 may seem a little redundant to question 1. But I am truly stumped.
Any help would be appreciated.
