1

An application intermittently fails with generic ODBC error. The client application is using ODBC to connect to SQL cluster. Enabling ODBC tracing shows this error:

StorageOnlineOp 46ec-44a8       ENTER SQLDriverConnectW 
                              HDBC                0x012E2FD8
                              HWND                0x00000000
                              WCHAR *             0x6C228B34 [      -3] "******\ 0"
                              SWORD                       -3 
                              WCHAR *             0x6C228B34 
                              SWORD                       -3 
                              SWORD *             0x00000000
                              UWORD                        0 <SQL_DRIVER_NOPROMPT>

StorageOnlineOp 46ec-44a8       EXIT  SQLDriverConnectW  with return code -1 (SQL_ERROR)
                              HDBC                0x012E2FD8
                              HWND                0x00000000
                              WCHAR *             0x6C228B34 [      -3] "******\ 0"
                              SWORD                       -3 
                              WCHAR *             0x6C228B34 
                              SWORD                       -3 
                              SWORD *             0x00000000
                              UWORD                        0 <SQL_DRIVER_NOPROMPT>

                              DIAG [IM005] [Microsoft][ODBC Driver Manager] Driver's SQLAllocHandle on SQL_HANDLE_DBC failed (0) 

According to Microsoft the error is explained:

(DM) During SQLConnect, the Driver Manager called the driver's SQLAllocHandle function with a HandleType of SQL_HANDLE_DBC and the driver returned an error.

Any recommended steps for further narrowing down the problem?

According to ODBC logs version information is:

03.80.0000 = SQLGetInfoW (SQL_ODBC_VER) 03.52 = SQLGetInfoW (SQL_DRIVER_ODBC_VER)

1

In this case I linked StorageOnlineOp in ODBC log to process name StorageOnlineOpns.exe. When issue occurred I used Task Manager to right click process and "Create Dump File"

An analysis shows the process was a 32-bit process, nearing it's 2GB address space limit, and was unable to allocate memory for the database connection.

Opening the dmp file in WinDbg, used the following commands to identify problem:

    0:011> lmvm StorageOnlineOpns
Browse full module list
start    end        module name
00170000 002b9000   StorageOnlineOpns   (export symbols)       StorageOnlineOpns.exe
    Loaded symbol image file: StorageOnlineOpns.exe
    Image path: D:\Program Files (x86)\Enterprise Vault\StorageOnlineOpns.exe
    Image name: StorageOnlineOpns.exe
    Browse all global symbols  functions  data
    Timestamp:        Wed May  1 11:23:18 2013 (51806E86)
    CheckSum:         0014CE8C
    ImageSize:        00149000
    File version:     10.0.3.1188
    Product version:  10.0.3.0
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Symantec Corporation
    ProductName:      Enterprise Vault
    InternalName:     STORAGEONLINEOPNS
    OriginalFilename: STORAGEONLINEOPNS.DLL
    ProductVersion:   10, 0, 3, 0
    FileVersion:      10.0.3.1188
    FileDescription:  StorageOnlineOpns Module
    LegalCopyright:   Copyright (c) 2013 Symantec Corporation. All rights reserved. Use of this product is subject to license terms.

This process is 32-bit so is limited to 2GB address space, this is close to being exhausted:

0:011> !heap -s                                    
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Heap                                    669          6a04c000 (   1.657 GB)  86.08%   82.83%
<unknown>                               397           85b5000 ( 133.707 MB)   6.79%    6.53%
Image                                   712           58ea000 (  88.914 MB)   4.51%    4.34%
Free                                    406           4d72000 (  77.445 MB)            3.78%
Stack                                   153           3300000 (  51.000 MB)   2.59%    2.49%
Other                                    46             5f000 ( 380.000 kB)   0.02%    0.02%
TEB                                      51             33000 ( 204.000 kB)   0.01%    0.01%
PEB                                       1              1000 (   4.000 kB)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                            1201          73d79000 (   1.810 GB)  94.06%   90.50%
MEM_IMAGE                               741           5b38000 (  91.219 MB)   4.63%    4.45%
MEM_MAPPED                               87           19cd000 (  25.801 MB)   1.31%    1.26%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_COMMIT                             1605          6f65a000 (   1.741 GB)  90.45%   87.03%
MEM_RESERVE                             424           bc24000 ( 188.141 MB)   9.55%    9.19%
MEM_FREE                                406           4d72000 (  77.445 MB)            3.78%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          902          68c29000 (   1.637 GB)  85.06%   81.85%
PAGE_EXECUTE_READ                       135           3e04000 (  62.016 MB)   3.15%    3.03%
PAGE_READONLY                           382           2960000 (  41.375 MB)   2.10%    2.02%
PAGE_WRITECOPY                           83            1fc000 (   1.984 MB)   0.10%    0.10%
PAGE_READWRITE|PAGE_GUARD               103             d1000 ( 836.000 kB)   0.04%    0.04%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Heap                                         dd10000            fd0000 (  15.813 MB)
<unknown>                                   7f0e0000            f00000 (  15.000 MB)
Image                                       6a611000            e70000 (  14.438 MB)
Free                                        72f4a000            1c6000 (   1.773 MB)
Stack                                        1980000             fd000 (1012.000 kB)
Other                                       7efb0000             23000 ( 140.000 kB)
TEB                                         7ef19000              1000 (   4.000 kB)
PEB                                         7efde000              1000 (   4.000 kB)

The database connection code is failing due to inability to allocate more memory:

0:011> k
 # ChildEBP RetAddr  
00 031ac570 73b217cd ntdll!NtWaitForSingleObject+0x15
01 031ac5b0 73b27696 mswsock!SockWaitForSingleObject+0x3a
02 031ac62c 764b689f mswsock!WSPRecv+0x2af
03 031ac674 739616c7 ws2_32!recv+0x87
04 031ac6c4 5510b4a2 dbnetlib!ConnectionRead+0x429
05 031ac700 5514ceb2 sqloledb!CDataSource::ConnectionRead+0x82
06 031ac754 550f9056 sqloledb!CDBConnection::GetBytesFromWire+0x228
07 031ac7a4 551888e0 sqloledb!CDBConnection::ProcessTDSStream+0x1d6
08 031ac858 55121a4f sqloledb!CStmt::SQLExecRPC+0x61b
09 031ac894 550f88cb sqloledb!CCommand::ExecuteHelper+0x232
0a 031ac920 550ff5c6 sqloledb!CCommand::Execute+0x127c
0b 031ac960 57e5a335 sqloledb!CImpICommandText::Execute+0x16f
0c 031ac9a0 57e52c8b msado15!CConnection::Execute+0x9c
0d 031acbd8 57e5573b msado15!_ExecuteAsync+0x1e1
0e 031acbf4 57e556f7 msado15!ExecuteAsync+0x6f
0f 031acd24 57e54ecf msado15!CQuery::Execute+0x10f5
10 031acd88 57eabd39 msado15!CCommand::_Execute+0x192
11 031acfe0 57ead171 msado15!CCommand::ExecuteWithModeFlag+0x8da
12 031ad00c 5acb4406 msado15!CCommand::Execute+0x5e
WARNING: Stack unwind information not available. Following frames may be wrong.
13 031ad28c 5aca7a1f AdoInterface!CADODataAccess::ExecuteSQLCommand+0x266
14 031ad2b4 5853ff29 AdoInterface!CADODataAccess::ExecuteStoredProcedure+0x1f
15 00000000 00000000 EVFingerprintCatalogueAccessor!CRegistryHelper::IsOpen+0x15bb9
0:011> !gle
LastErrorValue: (Win32) 0 (0) - The operation completed successfully.
LastStatusValue: (NTSTATUS) 0xc0000017 - {Not Enough Quota}  Not enough virtual memory or paging file quota is available to complete the specified operation.

In addition there is was a lock blocking 4 threads. The blocking function was a heap function. This is commonly found in heap corruption issues.

0:004> k
 # ChildEBP RetAddr  
00 026eed5c 77a1ebae ntdll!NtWaitForSingleObject+0x15
01 026eedc0 77a1ea92 ntdll!RtlpWaitOnCriticalSection+0x13e
02 026eede8 77a1ece9 ntdll!RtlEnterCriticalSection+0x150
03 026eeed4 77a02bd5 ntdll!RtlpFreeHeap+0x96
04 026eeef4 75b06e8a ntdll!RtlFreeHeap+0x142
05 026eef08 7671d9f6 ole32!CRetailMalloc_Free+0x1c [d:\w7rtm\com\ole32\com\class\memapi.cxx @ 687] 
06 026eef1c 7671daad oleaut32!_SafeArrayFreeData+0x1f
07 026eef2c 766dfdf0 oleaut32!_SafeArrayReleaseData+0x37
08 026eef4c 766dfd5c oleaut32!_SafeArrayDestroyData+0x81
09 026eef68 766f0fc3 oleaut32!_SafeArrayDestroy+0xa0
0a 026eef8c 766f133c oleaut32!LPSAFEARRAY_UserFree+0x10e
0b 026eefa0 759f7c1e oleaut32!VARIANT_UserFree+0xae
0c 026eefd8 759f4256 rpcrt4!NdrUserMarshalFree+0x37
0d 026eeff8 75a7083e rpcrt4!NdrPointerFree+0xa8
0e 026ef01c 75a7072a rpcrt4!NdrpFreeParams+0x145
0f 026ef030 75a706e0 rpcrt4!NdrStubCall2+0x4eb
10 026ef428 75bee7e6 rpcrt4!NdrStubCall2+0x42a
11 026ef470 75bee876 ole32!CStdStubBuffer_Invoke+0xb6 [d:\w7rtm\com\rpc\ndrole\stub.cxx @ 1590] 
12 026ef4b8 75beedd0 ole32!SyncStubInvoke+0x3c [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 1187] 
13 026ef504 75b08a6b ole32!StubInvoke+0xb9 [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 1396] 
14 026ef5e0 75b08960 ole32!CCtxComChnl::ContextInvoke+0xfa [d:\w7rtm\com\ole32\com\dcomrem\ctxchnl.cxx @ 1262] 
15 026ef5fc 75b09532 ole32!MTAInvoke+0x1a [d:\w7rtm\com\ole32\com\dcomrem\callctrl.cxx @ 2105] 
16 026ef628 75beeccd ole32!STAInvoke+0x46 [d:\w7rtm\com\ole32\com\dcomrem\callctrl.cxx @ 1924] 
17 026ef65c 75beeb41 ole32!AppInvoke+0xab [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 1086] 
18 026ef740 75bef1fd ole32!ComInvokeWithLockAndIPID+0x372 [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 1724] 
19 026ef768 75b0938f ole32!ComInvoke+0xc5 [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 1469] 
1a 026ef77c 75b0934e ole32!ThreadDispatch+0x23 [d:\w7rtm\com\ole32\com\dcomrem\chancont.cxx @ 298] 
1b 026ef7c0 760f62fa ole32!ThreadWndProc+0x161 [d:\w7rtm\com\ole32\com\dcomrem\chancont.cxx @ 654] 
1c 026ef7ec 760f6d3a user32!InternalCallWinProc+0x23
1d 026ef864 760f77c4 user32!UserCallWinProcCheckWow+0x109
1e 026ef8c4 760f788a user32!DispatchMessageWorker+0x3b5
1f 026ef8d4 001cc7d4 user32!DispatchMessageW+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
20 026ef938 001cc8a9 StorageOnlineOpns+0x5c7d4
21 026ef974 752634c7 StorageOnlineOpns+0x5c8a9
22 026ef980 7650336a msvcr90!_endthreadex+0xd8
23 026ef98c 77a09902 kernel32!BaseThreadInitThunk+0xe
24 026ef9cc 77a098d5 ntdll!__RtlUserThreadStart+0x70
25 026ef9e4 00000000 ntdll!_RtlUserThreadStart+0x1b

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.