0

I have a process which runs via Pentaho on a Unix server against MySQL 5.6.19. and using 5.1.32 of the MySQL jdbc driver. We actually found this driver helped with this error in some cases compared to 5.1.26

The process is an extremely long running heavy load analytical process. Last night I had a failure, and I can see from the system stats that there was a large amount of "iowait" on the system which started ~15mins before this failure.

The actual error is pasted below. I've lived with this error over the years and usually it is simply due to the net_write_timeout. So that has already been set at 86,400 or 24 hours. For good measure i set net_read_timeout to the same. I also had a situation where after 8 hours we hit the "wait_timeout" with exactly the same error so that is now also at 86,400.

So I don't really understand the error. Last packet sent was 15 seconds ago, last packet received was 142ms ago. so this cannot be a timeout? It must be something else? Yet googling around for this error every time people say "its the timeout". I've also set tcpKeepAlive=true for good measure.

The MySQL and Pentaho process are both working on the same local box. No errors in /var/log/messages or in the MySQL error log.

It's hard to know if this should be posted here, or on the sysadmin site, or elsewhere.

Here's the full error:

org.pentaho.di.core.exception.KettleDatabaseException: Couldn't get row from result set Communications link failure

The last packet successfully received from the server was 142 milliseconds ago.  The last packet sent successfully to the server was 14,468,141 milliseconds ago.

        at org.pentaho.di.core.database.Database.getRow(Database.java:2420)
        at org.pentaho.di.core.database.Database.getRow(Database.java:2389)
        at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:145)
        at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 142 milliseconds ago.  The last packet sent successfully to the server was 14,468,141 milliseconds ago.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1137)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3697)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3586)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4131)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:925)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:2033)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:408)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:387)
        at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7008)
        at org.pentaho.di.core.database.Database.getRow(Database.java:2408)
        ... 4 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 45 bytes, read 4 bytes before connection was unexpectedly lost.
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3143)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3653)
        ... 12 more
  • Why not add a secondary read only server geographically closer to the analytical process you are running? – Craig Efrein Sep 12 '14 at 9:00
  • As I said above the analytical process is running on the same box as the database, so this is purely a localhost connection. – Codek Sep 12 '14 at 10:34
  • Its the subject that's confusing because if both MySQL and Pentaho are on the same box, there shouldn't be any sort of communications or tcp issues. Have you run show processlist on your MySQL server to see what its doing? – Craig Efrein Sep 12 '14 at 12:10
  • yes, we have queries running - quite a lot of them. There is still "communication" though, it's just over the loopback interface rather than actually going over the network. If I change the job so that it does nothing other than execute the queries it works. So it's the additional work i'm trying to do, slowing down the read, and therefore causing something to go awry. – Codek Sep 12 '14 at 13:36
  • IMHO, if there is anything going awry here, it has nothing to do with communication. There is no communication in a loopback device. A loopback simply makes a file accessible as a block device. I would lean more towards locking issues. Are your tables in MyISAM or InnoDB? – Craig Efrein Sep 12 '14 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.