Why has my slave server stopped connecting to master in MySQL replication?

Question

We have MySQL replication setup and it has been running without issue for some time. This server acts as a slave and a master (it replicates through to another server). This morning I received an alert to say it has stopped:

mysql> SHOW SLAVE STATUS\G;
*************************** 1. row ***************************
               Slave_IO_State: Connecting to master
                  Master_Host: 123.456.78.90
                  Master_User: replication
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: binary-log.000006
          Read_Master_Log_Pos: 225898044
               Relay_Log_File: mysqld-relay-bin.000018
                Relay_Log_Pos: 4
        Relay_Master_Log_File: binary-log.000006
             Slave_IO_Running: Connecting
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table: mysql.%
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 225898044
              Relay_Log_Space: 107
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: Yes
           Master_SSL_CA_File: /var/lib/mysql/ssl/ca-cert.pem
           Master_SSL_CA_Path:
              Master_SSL_Cert: /var/lib/mysql/ssl/client-cert.pem
            Master_SSL_Cipher:
               Master_SSL_Key: /var/lib/mysql/ssl/client-key.pem
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 2026
                Last_IO_Error: error connecting to master '[email protected]:3306' - retry-time: 60  retries: 86400
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 0
1 row in set (0.00 sec)

ERROR:
No query specified

If I try connect from the same machine:

[user@host~]# mysql -u replication -p -h 123.456.78.90 Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 35069260 ...

And permissions from the master:

mysql> SHOW GRANTS;
+-------------------------------------------------------------------------------------------------------------------------------------+
| Grants for [email protected]                                                                                               |
+-------------------------------------------------------------------------------------------------------------------------------------+
| GRANT REPLICATION SLAVE ON *.* TO 'replication'@'123.456.78.90' IDENTIFIED BY PASSWORD '*E2F700F7365XXXXXXXXXX9F3A4C5C17C' |
+-------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

The error log on the slave shows:

150709 20:59:00 [ERROR] Slave I/O: error connecting to master '[email protected]:3306' - retry-time: 60  retries: 86400, Error_code: 2026

It was at that point I remembered last night updates were run:

Updating:
 mysql55w         x86_64    5.5.44-1.w6
 mysql55w-libs    x86_64    5.5.44-1.w6
 mysql55w-server  x86_64    5.5.44-1.w6

But I've not been able to find any related information to this particular update. The error, 2026, is the following:

Error: 2026 (CR_SSL_CONNECTION_ERROR)
Message: SSL connection error: %s

The certificates haven't changed and the paths are specified correctly on the slave as show at the start of the question. I found if I run:

[user@host ~]# mysql --ssl-ca=/var/lib/mysql/ssl/ca-cert.pem --ssl-cert=/var/lib/mysql/ssl/client-cert.pem --ssl-key=/var/lib/mysql/ssl/client-key.pem -u replication -p -h 123.456.78.90
Enter password:
ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)

So it appears that, for some reason, the SSL certificates have suddenly stopped working, though they haven't changed for months on both the master and slave.

Where should I turn next to figure out the problem?

Answer 1

this could be because of openssl update. take a look at the following

https://github.com/iuscommunity-pkg/mysql55/issues/4

a notable bug that occurs with the latest version of OpenSSL. This issue prevents secure connections from being made due to the default hardcoded cipher no longer being supported in the newer version of OpenSSL. Below is an example of the error:

[root@server newrscerts2]$ mysql -u rs -p'lamepassword' \
                            --ssl-ca=/etc/mysql/newrscerts/ca.pem --ssl
ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)

You can work around this issue by specifying a different cipher manually.

[root@server ~]$ mysql -u rs -p'lamepassword' \
                  --ssl-ca=/etc/mysql/newrscerts2/ca-cert.pem \
                  --ssl-cipher=AES128-SHA

Welcome to the MySQL monitor. 
Commands end with ; or \g.
Your MySQL connection id is 10057
Server version: 5.5.43-log Distributed by The IUS Community Project
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Further details can be found at https://bugzilla.redhat.com/show_bug.cgi?id=1228755.