Server responds with empty packet during session negotiation resulting in client giving a malformed packet error

Question

I'm attempting to connect to a remote mysql server. This happens 100% of the time.

client: mysql Ver 14.14 Distrib 5.7.12, for Win32 (AMD64)
server: 5.0.95

This is the error I get:

C:\>mysql -h example.com -P 3306 -D prod_rcadb -u username -p
Enter password: **********
ERROR 2027 (HY000): Malformed packet

same error with mysqladmin:

C:\>mysqladmin -h example.com -P 3306 -u username -p version
Enter password: **********
mysqladmin: connect to server at '10.106.24.79' failed
error: 'Malformed packet'

So I took a pcap, just to get an idea of what that conversation looked like.

TCP handshake:

1              2016-04-14 11:18:48.910690         0.000000              137.69.150.80                     10.106.24.79       TCP        66                51157→3306 [SYN] Seq=0 Win=8192 Len=0 MSS=1428 WS=256 SACK_PERM=1   8192
2              2016-04-14 11:18:49.019893         0.109203              10.106.24.79                       137.69.150.80     TCP        66                3306→51157 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=256           5840
3              2016-04-14 11:18:49.019893         0.000000              137.69.150.80                     10.106.24.79       TCP        54                51157→3306 [ACK] Seq=1 Ack=1 Win=65536 Len=0          256

Mysql connection negotiation:

4              2016-04-14 11:18:49.144696         0.124803              10.106.24.79                       137.69.150.80     MySQL  110        Server Greeting proto=10 version=5.0.95            23
5              2016-04-14 11:18:49.144696         0.000000              137.69.150.80                     10.106.24.79       MySQL  119         Login Request user=bigdata   256<br>
6              2016-04-14 11:18:49.144696         0.000000              10.106.24.79                       137.69.150.80     TCP        60                3306→51157 [ACK] Seq=57 Ack=66 Win=5888 Len=0        23
7              2016-04-14 11:18:49.316301         0.171605              10.106.24.79                       137.69.150.80     MySQL  60           Response                23

So client connects at the TCP level, server greets us with supported options and status:

Server Capabilities: 0xa22c
.... .... .... ...0 = Long Password: Not set
.... .... .... ..0. = Found Rows: Not set
.... .... .... .1.. = Long Column Flags: Set
.... .... .... 1... = Connect With Database: Set
.... .... ...0 .... = Don't Allow database.table.column: Not set
.... .... ..1. .... = Can use compression protocol: Set
.... .... .0.. .... = ODBC Client: Not set
.... .... 0... .... = Can Use LOAD DATA LOCAL: Not set
.... ...0 .... .... = Ignore Spaces before '(': Not set
.... ..1. .... .... = Speaks 4.1 protocol (new flag): Set
.... .0.. .... .... = Interactive Client: Not set
.... 0... .... .... = Switch to SSL after handshake: Not set
...0 .... .... .... = Ignore sigpipes: Not set
..1. .... .... .... = Knows about transactions: Set
.0.. .... .... .... = Speaks 4.1 protocol (old flag): Not set
1... .... .... .... = Can do 4.1 authentication: Set
Server Language: latin1 COLLATE latin1_swedish_ci (8)

Client requests a login:

.... .... .... ...1 = Long Password: Set
.... .... .... ..0. = Found Rows: Not set
.... .... .... .1.. = Long Column Flags: Set
.... .... .... 0... = Connect With Database: Not set
.... .... ...0 .... = Don't Allow database.table.column: Not set
.... .... ..0. .... = Can use compression protocol: Not set
.... .... .0.. .... = ODBC Client: Not set
.... .... 1... .... = Can Use LOAD DATA LOCAL: Set
.... ...0 .... .... = Ignore Spaces before '(': Not set
.... ..1. .... .... = Speaks 4.1 protocol (new flag): Set
.... .0.. .... .... = Interactive Client: Not set
.... 0... .... .... = Switch to SSL after handshake: Not set
...0 .... .... .... = Ignore sigpipes: Not set
..1. .... .... .... = Knows about transactions: Set
.0.. .... .... .... = Speaks 4.1 protocol (old flag): Not set
1... .... .... .... = Can do 4.1 authentication: Set
Extended Client Capabilities: 0x81be
.... .... .... ...0 = Multiple statements: Not set
.... .... .... ..1. = Multiple results: Set
.... .... .... .1.. = PS Multiple results: Set
.... .... .... 1... = Plugin Auth: Set
.... .... ...1 .... = Connect attrs: Set
.... .... ..1. .... = Plugin Auth LENENC Client Data: Set
.... .... 1... .... = Session variable tracking: Set
1000 0001 .0.. .... = Unused: 0x0204

Server acknowledges, then sends a response, which is essentially empty…

Packet Length: 1
Packet Number: 2
EOF marker: 254

And that immediately causes client to FIN and close out the socket:

8              2016-04-14 11:18:49.316301         0.000000              137.69.150.80                     10.106.24.79       TCP        54                51157→3306 [FIN, ACK] Seq=66 Ack=62 Win=65536 Len=0            256
9              2016-04-14 11:18:49.332901         0.016600              10.106.24.79                       137.69.150.80     TCP        60                3306→51157 [ACK] Seq=62 Ack=67 Win=5888 Len=0        23
10           2016-04-14 11:18:49.391904         0.059003              10.106.24.79                       137.69.150.80     TCP        60                3306→51157 [FIN, ACK] Seq=62 Ack=67 Win=5888 Len=0              23
11           2016-04-14 11:18:49.391904         0.000000              137.69.150.80                     10.106.24.79       TCP        54                51157→3306 [ACK] Seq=67 Ack=63 Win=65536 Len=0     256

So the connecting client didn’t like that empty packet the server sent.

I don't get this against another mysql server of the same version from the same client. Here is the response packet to the login request from server 2:

Packet Length: 7
Packet Number: 2
Affected Rows: 0
Server Status: 0x0002
.... .... .... ...0 = In transaction: Not set
.... .... .... ..1. = AUTO_COMMIT: Set
.... .... .... .0.. = More results: Not set
.... .... .... 0... = Multi query - more resultsets: Not set
.... .... ...0 .... = Bad index used: Not set
.... .... ..0. .... = No index used: Not set
.... .... .0.. .... = Cursor exists: Not set
.... .... 0... .... = Last row sent: Not set
.... ...0 .... .... = database dropped: Not set
.... ..0. .... .... = No backslash escapes: Not set
.... .0.. .... .... = Session state changed: Not set
.... 0... .... .... = Query was slow: Not set
...0 .... .... .... = PS Out Params: Not set

I don't administer this db myself, but if you have suggestions for things to look for in logs, I can request that information.

Any thoughts on why I'm getting an empty packet back from the server?

Answer 1

The packet that I thought was empty was actually not, it was an Old Auth Switch Request:

Payload
1     [fe]
Fields
status (1) -- 0xfe
Returns
Protocol::AuthSwitchResponse with old password hash
Example
01 00 00 02 fe

The 1, 2, 254 that wireshark parsed is actually 01 00 00 02 fe if you look at the actual byte strings.

So it isn't that there is a misunderstanding, the server understands completely and correctly responds and the client correctly terminates because it cannot negotiate down. The protocol is not too old as it was changed in 4.1, so both 5.0 and 5.7 understand each other perfectly. This should be a clearer error message.

The client is too new to use --skip-secure-auth (reference). They purposefully removed the ability to negotiate down prior to 5.7.

So my options are to allow new passwords on the server (which is a user specific configuration, not global server option) or to use an older binary.

The specific configuration issue is based on the username I was given. At some point in the past, someone using this username was using an older client and they changed the password method:

B.5.2.4 Client does not support authentication protocol

The current implementation of the authentication protocol uses a password hashing algorithm that is incompatible with that used by older (pre-4.1) clients. Attempts to connect to a 4.1 or newer server with an older client may fail with the following message:

shell> mysql
Client does not support authentication protocol requested
by server; consider upgrading MySQL client

To deal with this problem, the preferred solution is to upgrade all client programs to use a 4.1.1 or newer client library. If that is not possible, use one of the following approaches:

    To connect to the server with a pre-4.1 client program, use an account that still has a pre-4.1-style password.

    Reset the password to pre-4.1 style for each user that needs to use a pre-4.1 client program. This can be done using the SET PASSWORD statement and the OLD_PASSWORD() function. As of MySQL 5.6.6, it is also necessary to first ensure that the authentication plugin for the account is mysql_old_password:

    mysql> UPDATE mysql.user SET plugin = 'mysql_old_password'
    mysql> WHERE User = 'some_user' AND Host = 'some_host';
    mysql> FLUSH PRIVILEGES;
    mysql> SET PASSWORD FOR
        -> 'some_user'@'some_host' = OLD_PASSWORD('new_password');

Answer 2

I got this error when trying to connect to an older MySQL server with @@old_passwords enabled. I am using mysql-client v5.7.19 and Server version: 5.1.73 .

I fixed the problem by manually creating a MySQL SHA1 password hash (using a program I wrote for the purpose) and then I ran this command on the server:

SET PASSWORD FOR 'nagios'@'10.10.10.201' = 'xxxx';

(Where xxxx was actually the hash.)