Last Friday one of two instances in our 10.2.0.5 RAC cluster was non-responsive after logging the following messages. We ended up rebooting the node and it came up successfully. While I will likely end up creating an SR for this, I would like to know if anyone else has had a similar issue. The mmon process was taking up 100% of the CPU, and I could not connect to the database even as sysdba. The metalink notes that I saw point to AWR as the culprit, but they were getting return code 5, not return code 2.
alert log Fri May 30 03:55:52 EDT 2014 MMNL absent for 5619 secs; Foregrounds taking over Fri May 30 03:55:55 EDT 2014 trace file *** 2014-05-30 03:55:55.790 *** SERVICE NAME:(SYS$BACKGROUND) 2014-05-30 03:55:55.790 *** SESSION ID:(536.1) 2014-05-30 03:55:55.790 *** KEWRRFC: Remote Flush slave failed, return Code: 2.