2

I have a web application which uses a MySQL 5.7.13 server for storing its data. MySQL is running on Windows Server 2008 R2. A couple of month ago I started seeing exceptions like the following being thrown by the MySQL Connector/Net:

System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
   at MySql.Data.MySqlClient.NativeDriver.StartSSL()
   at MySql.Data.MySqlClient.NativeDriver.Open()
   at MySql.Data.MySqlClient.Driver.Open()
   at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)
   at MySql.Data.MySqlClient.MySqlPool..ctor(MySqlConnectionStringBuilder settings)
   at MySql.Data.MySqlClient.MySqlPoolManager.GetPool(MySqlConnectionStringBuilder settings)
   at MySql.Data.MySqlClient.MySqlConnection.Open()

There is nothing in the servers error log corresponding to the exception above.

These exceptions occour relatively unfrequently but often enough to cause actual problems for the system (some naive tests indicate that about 1 in 1000 connections fail like this).

When the problems began I was using MySQL Connector/Net 6.6.5. The problems prompted me to upgrade to 6.9.8 (latest version at the time) but that hos not improved the situation.

I also believe I can rule out faulty hardware in the server as my system uses a replication slave (running on a completely different server) and I have tested to switch master and slave and the problem still persists.

While I cannot pinpoint the exact date the problems started, two things changed about the time of the problem starting

  • Servers were moved to new hosting (with new/different networking, infrastructure, etc.)
  • I upgraded from MySQL 5.6.16 to 5.6.30 (I have since upgraded to 5.7.13 but the problem persists).

While I suppose this could be related to, for example, a faulty network switch cause packets to be dropped this seems somewhat unlikely as the current hosting solution is generally of much higher quality that the previous hosting solution.

Is it possible a change in MySQL's SSL implementation causes an incompability with my SSL configuration, leading to these intermittent problems? I use SSL to connect to the server using the following connection string:

server=<obfuscated>;port=3303;database=<obfuscated>;user id=<obfuscated>;pwd=<obfuscated>; allow zero datetime=true; Protocol=TCP; SslMode=Required; Compress=false; Pooling=true; Min Pool Size=10; Max Pool Size=100; Connection Lifetime=30; Connection Timeout=10;functions return string=true;Respect Binary Flags=false;

The [mysqld] section of the my.ini file contains the following:

[mysqld]
basedir="C:/Program Files/MySQL/MySQL Server 5.7/"
binlog-format = MIXED
character-set-server=utf8
datadir="E:/MySqlData/"
default-storage-engine=INNODB
innodb_buffer_pool_instances=8
innodb_buffer_pool_size=16G
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=7M
innodb_log_file_size=123M
innodb_thread_concurrency=34
key_buffer_size=316M
log-bin = "G:/mysql-log/bin-log"
log-slave-updates=0
expire_logs_days=14
max_allowed_packet=32MB
max_binlog_size = 1073741824
max_connect_errors = 100000
max_connections = 800
myisam_max_sort_file_size=100G
myisam_sort_buffer_size=339M
port=3306
query_cache_size=201M
read_buffer_size = 128K
read_rnd_buffer_size = 8MB
relay-log="E:/MySqlRelaylog/axbyte_relay_bin"
server-id = 1
slow-query-log
sort_buffer_size = 8M
sql-mode=""
ssl-ca="E:/certs/sha-256/ca-cert.pem"
ssl-cert="E:/certs/sha-256/cert.pem"
ssl-key="E:/certs/sha-256/key.pem"
sync_binlog=1
sync_master_info=10000
sync_relay_log_info=10000
sync_relay_log=10000
thread_cache_size=8
tmp_table_size=205M
tmpdir = F:/mysql-temp
skip_slave_start
ft_min_word_len=2
log_warnings = 2

I might add that the SSL certificate the server uses does not match the server's actual address (the certificate is issued to a real domain but I use an internal DNS name to connect to the MySQL server).

Any suggestions regarding what might cause this problem?

  • Do you have at least 20GB of RAM? (Otherwise, some settings are too high.) – Rick James Sep 28 '16 at 23:53
  • Yes, I have 32GB of RAM. – David Nordvall Sep 29 '16 at 11:07
  • we are experiencing the exact same issue, although in our case happens less frequently. We are a bit lost as to why it happens. Did you happen to find the solution to the problem? – Enric May 10 '18 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.