1

I need to setup replication via SSL. 2 nodes master/master for H/A. Then 2 slave nodes for backup. Node 3 will slave off of node 1, and node 4 will slave off of node 2 for redundant backups. Nodes 3 & 4 will store backups to a directory mapped to S3 via S3FS. All nodes are physical.

We're using mariadb-5.5.41 server on CentOS 7 hosts.

It's a neat plan!

However I am unable to get authentication to work with ssl required of the user.

We already have a CA certificate and key established in the environment. And we used those to generate the cert and key to be used with mariadb.

This is the process we used to generate the cert/key:

openssl genrsa -des3 -out db1.example.com.key 4096
openssl req -new -key db1.example.com.key -out db1.example.com.csr
openssl x509 -req -days 3650 -in db1.example.com.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out db1.example.com.crt

Using those keys I put together this my.cnf file. Haven't gotten around to configuring replication yet, as I have yet to get ssl logins to work.

[root@db1:~] #cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd
ssl
ssl-ca=/opt/mysql/ca.crt
ssl-cert=/opt/mysql/db1.example.com.crt
ssl-key=/opt/mysql/db1.example.com.key

[mysqld_safe]
general_log_file=/var/log/mariadb/mariadb.log
general_log=1
log-error=/var/log/mariadb/mariadb_error.log
pid-file=/var/run/mariadb/mariadb.pid
log_slow_queries=/var/log/mysql/mysql-slow.log
long_query_time=2
log-queries-not-using-indexes

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

Made sure mariadb could read the cert files:

[root@db1:~] #ls -ld /opt/mysql/ /opt/mysql/*
drwx------. 2 mysql mysql   86 Jul 20 06:20 /opt/mysql/
-r--------. 1 mysql mysql 2212 Jul 20 05:14 /opt/mysql/ca.crt
-r--------. 1 mysql mysql 1956 Jul 20 05:17 /opt/mysql/db1.example.com.crt
-r--------. 1 mysql mysql 3247 Jul 20 05:15 /opt/mysql/db1.example.com.key

Verified that the ssl libraries were linked against the mysqld binary

[root@db1:~] #ldd $(which mysqld) | grep ssl
        libssl.so.10 => /lib64/libssl.so.10 (0x00007f79669ba000)

And restarted mariadb. And if I take a look at my SSL variables in my mysql command line, everything is looking good.

MariaDB [mysql]> show variables like '%ssl%';
+---------------+---------------------------------+
| Variable_name | Value                           |
+---------------+---------------------------------+
| have_openssl  | YES                             |
| have_ssl      | YES                             |
| ssl_ca        | /opt/mysql/ca.crt               |
| ssl_capath    |                                 |
| ssl_cert      | /opt/mysql/db1.example.com.crt |
| ssl_cipher    |                                 |
| ssl_key       | /opt/mysql/db1.example.com.key |
+---------------+---------------------------------+
7 rows in set (0.00 sec)

If I create one replication user without SSL required, he is able to log in from node 2 to node 1 no problem:

MariaDB [mysql]> grant replication slave on *.* to 'slave1'@'db2.example.com' identified by 'secret';
Query OK, 0 rows affected (0.00 sec)

[root@db2:~] #mysql -uslave1 -p -h db1.example.com
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 15
Server version: 5.5.41-MariaDB MariaDB Server

Copyright (c) 2000, 2014, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

If I show grants on this user I can confirm that SSL is not required

MariaDB [mysql]> show grants for 'slave1'@'db2.example.com';
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| Grants for slave1@db2.example.com                                                                                               |
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| GRANT REPLICATION SLAVE ON *.* TO 'slave1'@'db2.example.com' IDENTIFIED BY PASSWORD '*somelongpasswordhash' |
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

If I create the second slave user with the SSL requirement, and even flush privileges, I can not log into node 1 from node 2

MariaDB [mysql]> grant replication slave on *.* to 'slave2'@'db2.example.com' identified by 'test' require ssl;
Query OK, 0 rows affected (0.00 sec)

MariaDB [mysql]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

[root@db2:~] #mysql -uslave2 -p -h db1.example.com
Enter password:
ERROR 1045 (28000): Access denied for user 'slave2'@'db2.example.com' (using password: YES)

And if I look at the grants for this user I able to tell that ssl is required for him to be able to login:

MariaDB [mysql]> show grants for 'slave2'@'db2.example.com;
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Grants for slave2@db2.example.com                                                                                                           |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| GRANT REPLICATION SLAVE ON *.* TO 'slave2'@'db2.example.com' IDENTIFIED BY PASSWORD '*somelongpasswordhash' REQUIRE SSL |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

If I take a look at the error log:

[root@db1:~] #grep error /etc/my.cnf
log-error=/var/log/mariadb/mariadb_error.log

    [root@db1:~] #tail /var/log/mariadb/mariadb_error.log
150720  5:18:24 InnoDB: Initializing buffer pool, size = 128.0M
150720  5:18:24 InnoDB: Completed initialization of buffer pool
150720  5:18:24 InnoDB: highest supported file format is Barracuda.
150720  5:18:24  InnoDB: Waiting for the background threads to start
150720  5:18:25 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 120637807
150720  5:18:25 [Note] Plugin 'FEEDBACK' is disabled.
150720  5:18:25 [Note] Server socket created on IP: '0.0.0.0'.
150720  5:18:25 [Note] Event Scheduler: Loaded 0 events
150720  5:18:25 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server

I see absolutely NO indication as to why the login for the 'slave2' user (that requires SSL) is failing.

And I notice that if I do a \s it tells me that ssl is not in use:

MariaDB [(none)]> \s
--------------
mysql  Ver 15.1 Distrib 5.5.41-MariaDB, for Linux (x86_64) using readline 5.1

Connection id:          36
Current database:
Current user:           root@localhost
SSL:                    Not in use
Current pager:          stdout
Using outfile:          ''
Using delimiter:        ;
Server:                 MariaDB
Server version:         5.5.41-MariaDB MariaDB Server
Protocol version:       10
Connection:             Localhost via UNIX socket
Server characterset:    latin1
Db     characterset:    latin1
Client characterset:    utf8
Conn.  characterset:    utf8
UNIX socket:            /var/lib/mysql/mysql.sock
Uptime:                 22 hours 9 min 2 sec

Threads: 1  Questions: 311  Slow queries: 0  Opens: 39  Flush tables: 3  Open tables: 24  Queries per second avg: 0.003
--------------

From the above output, there's this line:

SSL: Not in use

Why would that be the case, given the way I've configured this? How can I correct this?

Also 1) how to I bump up the verbosity on the logs so I can get an indication as to why this is failing? and 2) what is the best way to troubleshoot this?

OK so question 2 may seem a little redundant to question 1. But I am truly stumped.

Any help would be appreciated.

2

According to chapter 6.3.9.3 Configuring MySQL to Use SSL Connections of the MySQL 5.5 Reference Manual your client has to be invoked with the --ssl-ca option.

shell> mysql --ssl-ca=ca.pem

If the user account is required to provide a client certificate, then you need to invoke the client also with --ssl-cert and --ssl-key.

shell> mysql --ssl-ca=ca.pem \
             --ssl-cert=client-cert.pem \
             --ssl-key=client-key.pem

To check if you're connected via SSL, the Ssl_cipher variable has to be nonempty

mysql> SHOW STATUS LIKE 'Ssl_cipher';

You might also take a look at chapter 6.3.10 Creating SSL Certificates and Keys Using openssl.

Follow up:

And since you wanna realize replication via SSL 17.3.7 Setting Up Replication Using SSL also applies.

  • The ssl options for the client can also be specified in the my.cnf. – Kevin Bott Sep 12 '17 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.