12

Problem summary

When browsing to the Oracle 12c EM Express webinterface, the page times-out when using a URL like:

  • http(s)://hostname:5500/em
  • http(s)://hostname.domain:5500/em

When using a url containing the IP address, or localhost, the page loads, but very very slowly (up to 5 minutes).

  • http(s)://ip-address/em
  • http(s)://localhost/em

When using the hostname, the listener log shows errors (multiple of these):

03-DEC-2015 11:25:33 * http * (ADDRESS=(PROTOCOL=tcps)(HOST=<ip address>%14)(PORT=52925)) * handoff * http * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12560: TNS:protocol adapter error

When using the IP address or localhost, the listener log shows no errors:

03-DEC-2015 11:23:39 * http * (ADDRESS=(PROTOCOL=tcps)(HOST=<ip address>)(PORT=52902)) * handoff * http * 0

This problem shows up all over the web (see references in footer), and nobody seems to be able to find the cause of this. I've tried -every- suggestion i could find, and listed the results below. I'll let this great community have a crack at it, and then it's off to Oracle support...

Platform information

  • Windows Server 2012 R2 x64
  • Oracle 12.1.0.2.0
  • A multitenant container database (CDB) with two Pluggable databases (PDB)
  • Automatic Storage Management (ASM) is not being used
  • Performing tests locally on the server, with firewall completely disabled

Tests already performed

emctl
A lot of people would suggest checking "emctl" out of habit. This does not exist for 12c, as it uses EM Express and not Database control.

http/https
I've set up, and tested this for http and https. No difference.

Browser
I've tested this in vanilla installations of IE 11, chrome and firefox. No add-ons enabled.

Database access
Both the CDB and PDB's can be accessed and queries through SQLPlus

Telnet
Opening telnet connections to the ports (e.g. 5500) works.

Alert log
The Oracle alert log does not contain any error messages.

My hypotheses

  • Something related to hostname lookup, since it works -slighty- when using IP address instead of hostname.
  • Something related to the certificate, although the same problem shows up for http.
  • Something related to the configuration of the XDB or listener service.
  • Something related to the domain part of the hostname being included or not. As this is added in most of the configuration files, but maybe not everywhere?

Detailed platform information

Note: In these snippets, i've always replaced the actual hostname with "HOSTNAME" and the actual domain with "domain". If "HOSTNAME" is used, it means it was not trailed by the domain name. The case represents the actual case the hostname is displayed in.

SYS_CONTEXT('USERENV','SERVER_HOST')
hostname

hostname variable in CMD
HOSTNAME

SYS_CONTEXT('USERENV','DB_DOMAIN')
domain

dbms_xdb_config.gethttpsport()
5500

dbms_xdb_config.gethttpport()
2200

"lsnrctl status" output

    LSNRCTL for 64-bit Windows: Version 12.1.0.2.0 - Production on 03-DEC-2015 15:33:12

    Copyright (c) 1991, 2014, Oracle.  All rights reserved.

    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hostname.domain)(PORT=1521)))
    STATUS of the LISTENER
    ------------------------
    Alias                     listener
    Version                   TNSLSNR for 64-bit Windows: Version 12.1.0.2.0 - Production
    Start Date                03-DEC-2015 12:27:51
    Uptime                    0 days 3 hr. 5 min. 21 sec
    Trace Level               off
    Security                  ON: Local OS Authentication
    SNMP                      OFF
    Listener Parameter File   C:\ORACLE\HOME\network\admin\listener.ora
    Listener Log File         C:\ORACLE\diag\tnslsnr\HOSTNAME\listener\alert\log.xml
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=HOSTNAME.domain)(PORT=1521)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(PIPENAME=\\.\pipe\EXTPROC1521ipc)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=HOSTNAME.domain)(PORT=2200))(Presentation=HTTP)(Session=RAW))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=HOSTNAME.domain)(PORT=5501))(Security=(my_wallet_directory=C:\ORACLE\admin\ecoomdb\xdb_wallet))(Presentation=HTTP)(Session=RAW))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=HOSTNAME.domain)(PORT=5500))(Security=(my_wallet_directory=C:\ORACLE\admin\ecoomdb\xdb_wallet))(Presentation=HTTP)(Session=RAW))
    Services Summary...
    Service "CLRExtProc" has 1 instance(s).
      Instance "CLRExtProc", status UNKNOWN, has 1 handler(s) for this service...
    Service "bibliodb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
    Service "ecoomdb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
    Service "ecoomdbXDB.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
    Service "technodb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
    The command completed successfully

"lsnrctl services" output

    LSNRCTL for 64-bit Windows: Version 12.1.0.2.0 - Production on 03-DEC-2015 15:34:22

    Copyright (c) 1991, 2014, Oracle.  All rights reserved.

    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=HOSTNAME.domain)(PORT=1521)))
    Services Summary...
    Service "CLRExtProc" has 1 instance(s).
      Instance "CLRExtProc", status UNKNOWN, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:0 refused:0
             LOCAL SERVER
    Service "bibliodb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:9 refused:0 state:ready
             LOCAL SERVER
    Service "ecoomdb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:9 refused:0 state:ready
             LOCAL SERVER
    Service "ecoomdbXDB.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
        Handler(s):
          "D000" established:8 refused:21 current:0 max:1022 state:ready
             DISPATCHER <machine: HOSTNAME, pid: 4040>
             (ADDRESS=(PROTOCOL=tcp)(HOST=HOSTNAME.domain)(PORT=49179))
    Service "technodb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:9 refused:0 state:ready
             LOCAL SERVER
    The command completed successfully

listener.ora

    SID_LIST_LISTENER =
      (SID_LIST =
        (SID_DESC =
          (SID_NAME = CLRExtProc)
          (ORACLE_HOME = C:\ORACLE\HOME)
          (PROGRAM = extproc)
          (ENVS = "EXTPROC_DLLS=ONLY:C:\ORACLE\HOME\bin\oraclr12.dll")
        )
      )

    LISTENER =
      (DESCRIPTION_LIST =
        (DESCRIPTION =
          (ADDRESS = (PROTOCOL = TCP)(HOST = HOSTNAME.domain)(PORT = 1521))
          (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
        )
      )

init.ora (Trimmed, also contains other settings...)

db_domain=domain
db_name="ecoomdb"
dispatchers="(PROTOCOL=TCP) (SERVICE=ecoomdbXDB)"
local_listener=LISTENER*

netstat -a
(relevant selection)

 TCP    0.0.0.0:1521           ECONECOOMDB1:0         LISTENING
TCP    0.0.0.0:5500           ECONECOOMDB1:0         LISTENING
TCP    0.0.0.0:5501           ECONECOOMDB1:0         LISTENING
TCP    [::]:5500              ECONECOOMDB1:0         LISTENING
TCP    [::]:5501              ECONECOOMDB1:0         LISTENING

windows hosts file
(this file is empty)

Environment variables

  • PATH contains C:\ORACLE\HOME\bin
  • ORACLE_HOME = C:\ORACLE\HOME
  • ORACLE_SID = ecoomdb

Registry HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\KEY_OraDB12Home1

  • ORACLE_HOME = C:\ORACLE\HOME
  • ORACLE_SID = ecoomdb

Related and duplicate (unanswered) questions/documents

Online resources

Stackexchange questions

Reddit threads

Oracle community

  • Is the hostname for your machine in DNS, or have you added it to your hosts file? – Philᵀᴹ Dec 3 '15 at 16:24
  • The hosts file is empty. Not sure how exactly the hostname is managed, but the network uses an ActiveDirectory server where the server is registered with its hostname, and the DNS servers recognise it as such. – Wouter Dec 4 '15 at 8:34
  • It all sounds suspiciously like DNS issues/timeouts. – Philᵀᴹ Dec 4 '15 at 8:40
  • Any idea on how to debug, track the exact cause? – Wouter Dec 4 '15 at 12:40
2

What do you get when you run these commands on db host

from command prompt ping host ping ipaddress(host ipaddress) tnsping tnsalias

and run lsnrctl services and see if any connection(s) are refused

if oracle client is installed on client machine(from where you are accessing oracle em express)

run from command prompt 1)ping ipaddress(db host ipaddress) -->> if you get error add ipaddress and domain name to hosts file 2)tnsping tnsalias

Edit:

Dispatcher D000 shows 21 connections refused.You have network problem.

Try to restart listener and connect to em express and see the lsnrctl services output ideally refused connections should be zero.

  • I'm running all tests on the db host. Pinging hostname and ipaddress, and tnsping to CDB and PDB's works fine. Nice catch about the refused connections! I've restarted the listener, and used the hostname-EM-URL: same problem "webpage not available", and lsnerctl shows no entry yet for XDB. When using the ip-EM-URL EM loads (slowly) and I get an XDB entry, but no refused connections. I have no idea what to make of this, but maybe it does to you? – Wouter Dec 11 '15 at 8:14
  • 1
    what is the output of these two commands from container db? SELECT UTL_INADDR.get_host_address from dual; SELECT UTL_INADDR.get_host_name from dual; – user82397 Dec 11 '15 at 21:24
  • host_address returns an ipv6 address, the host returns the hostname in uppercase, without the domain attached. I've also ran an RDA report, and that showed "DNS Lookup - FAILED". Also, Cloud Control fails to install when using the hostname, but works when using the IP address. I'm now certain this is some kind of DNS issue, I'm just clueless on how to debug/analyse... – Wouter Dec 14 '15 at 8:57
1

After contacting Oracle support, I've -kinda- been able to figure out what was going on. This server has 4 network adaptors, 1 adaptor connected to the network, and 3 unconfigured adaptors connected to an iSCSI storage. The problem seems to be caused by the web-client, or EM server (not sure here) trying to send the webpage request to the wrong adaptor.

For anyone facing the same issue, you can check this by browsing to the URL (using the hostname.domain) and then checking the listener.log file (-oracle_base-/diag/tnslsnr/-instance-/listener/trace/listener.log). For me, in the error message a different IP address than the servers main network adaptors IP address was shown. The IP address of one of the network adaptors connected to the iSCSI where shown. When omitting the IP addresses in my original post, I didn't spot this IP address difference.

18-DEC-2015 14:17:04 * http * (ADDRESS=(PROTOCOL=tcps)(HOST=<NOT SERVER IP address!>%14)(PORT=62119)) * handoff * http * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12560: TNS:protocol adapter error

When disabling the additional network adaptors, everything works fine.

Additionally, when going to "chrome://net-internals/#dns" in chrome, the IP addresses of all four network adaptors showed up for my hostname.

Apparently, this problem -does not- happen, when browsing to the webpage from another host. Also, nslookup returns only the correct IP address. So it seems this is not a DNS issue, but a local issue. I think this is not a full answer to the problem, as disabling network adaptors can't be a final solution, but I hope anyone facing this issue in the future has "more to go on" now...

Where does the discrepancy between IP address returned by the DNS server, and the IP addresses used by web browsers come from?

I've also made some additional changes that might have influenced the end-solution, so for completeness I'll include them. Not sure if relevant...:

  • Increased processes parameter of the database from 300 to 1000
  • Altered the "dispatchers" parameter by adding "(DISPATCHERS=5)"
0

Try using the IP Address:

http(s)://192.168.1.100:5500/em

This way you will be sure to hit the network card binding to the ip.

  • I assume you mean, try adding the port number to the URL? In my post I said I tried http(s)://ip-address/em, yet, I suspect that is a typo and I actually tried it with the port number. I don't think it would load at all over the default port (80). Either way, I have no way of testing this now, as we have switched from Oracle to SQL Server, and will never ever be looking back. Oracle just doesn't make sense in anything except colossal environments. – Wouter Jan 17 '17 at 8:28
0

I had the same problem after installing Oracle 12.2.0.1 on Windows 2016.

First I found that TCP/IPv6 was on so I turned it off.

Then tried following Oracle Note Number (Doc ID 1608258.1) - 12c EM Express shows Invalid Certificate Error in web browser.

Also didn't work.

I fixed it by old school way ... :)

I discovered that the Windows user Oracle_DBA (that logged on the PC and used to install Oracle 12c) don't have any permissions on the 2 files under the folder "xdb_wallet".

So:

1- I changed the ownership for the folder "xdb_wallet" from whatever it was to Oracle_DBA :

enter image description here

2- changed the ownership of the 2 files ( ewallet.p12 , cwallet.sso ) under the folder "xdb_wallet" from whatever it was to Oracle_DBA :

enter image description here

3- I did step 1 again , and checked that ownership changed for the 2 file.

The website https://hostname:5500/em worked fine on FireFox , because IE11/Win2016 has funny issues with Flash Player.

I don't know from security point of view what that means (steps 1-3), or what threats might happens due to those changes.

Please advice if you know.

-1

Verify the ip address for hostname in /etc/hosts file

for example, following is my hosts file

127.0.0.1       localhost
10.148.180.115  tom

tom is my host name, and 10.148.180.115 is the ip address for my host

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.