Oracle Wallet Error in Red Hat High Availability Resource

Question

Environment: Running Red Hat 7.2, using Pacemaker 1.1.13-10.el7_2.2 and Corosync 2.3.4-7.el7_2.1 to implement cluster failover of an Oracle 12c Enterprise Edition Release 12.1.0.2.0 Database.

Background: When I start the database outside the cluster, everything works correctly and outside queries from other hosts can execute. The database has an encryption wallet which is set to auto-open, and which worked seamlessly before this problem cropped up.

Problem: When I add the oracle instance to the High Availability resource group, the instance starts up without any errors (that I can see). However, when outside connections are attempted from other hosts, the database responds

ERROR at line 1:
ORA-28365: wallet is not open

If I connect to the database on the host server and execute any queries, no errors are reported at the console, and the queries from remote hosts suddenly start to succeed again. The cluster then works fine until the resource fails over to another node, then the problem reappears with the same symptoms / temporary solution. The rejected queries cause trace files to be generated for the database SID containing:

kcbtse_get_tbskey: decrypting encrypted key for pdb 0 tablespace 6 without opening the wallet
kcbtse_get_tbskey: wallet is not opened (ts 0/6)
kcbtse_encdec_tbsblk: DIAG DUMP tsn 0/6 rdba 25165987, afn 6, mode 4

Has anyone seen this sort of issue before? What is missing from the HA environment that is present outside of HA?

Answer 1

In case anyone is interested in the solution that was identified:

The sqlnet.ora in question contained the following:

ENCRYPTION_WALLET_LOCATION =
   (SOURCE =
     (METHOD = FILE)
       (METHOD_DATA =
        (DIRECTORY = ${ORACLE_BASE}/admin/${ORACLE_SID}/wallet)
        )
   )

Which worked fine with the oracle user profile on the server. However, when running inside the HA resource, the environment variables were overridden, and ORACLE_BASE was not defined. Thus when oracle started, the system attempted to locate the "/admin/${ORACLE_SID}/wallet" directory, and upon not finding it, defaulted to a "wallet closed" state. Queries about the wallet when the database is in this state returned a null record.

When we attempted to troubleshoot the issue, we logged into the host server as oracle, got the user profile from the shell, and thus when we started sqlplus and ran queries it was able to resolve the ENCRYPTION_WALLET_LOCATION and populate the record in the table. Thus all local queries worked, and suddenly all remote queries "started working" again with little indication in the logs of what had taken place behind the scenes.