I make a web application which has experienced heavy growth in the past week and it appears my database is seizing up because of it.
You can see that everything seemed to be OK until about 3AM this morning and then all of my requests started timing out. This isn't the first time this has happened, but after I push new changes to the server it starts up OK again. It doesn't seem to happen under low load, but has also ran at high load for hours perfectly fine. I don't think 3AM is an especially high load time.
Things I have done / to note:
Dedicated server, MSSQL and I'm using NHibernate as an ORM to generate my queries.
The server is pretty much just a data store. 99% of the queries are very basic insert/selects. I have one bulk insert for creating a large amount of playlist items.
I have batching enabled. It is set to 50 by default and 200 if >1000 playlist items are about to be written to the database. This reduced the spikes in my DB's response times.
I have my command timeout set to 5 seconds. It was set at the default, 30 seconds, and I was still experiencing the issue. The problem seems to have stayed away for a longer period of time after setting it to 5 seconds.
I have tried running
exec sp_updatestatsas described here: http://stackoverflow.com/questions/8602395/timeout-expired-the-timeout-period-elapsed-prior-to-completion-of-the-operation# which did show some indexes being updated, but did not result in any real differences.
I have set my databases connection isolation level to ReadUncommitted:
I specify the default DB for NHibernate performance as described here:
What on earth should I be looking at to try and get a grip on what is causing my database to lockup? It's showing only 1% CPU usage so I don't think it is overly taxed. It shouldn't be a long running query if I have 5 second caps on everything. It's not the indexes. I'm running out of ideas.
In terms of queries -- it's all SELECT user statements. It's just a bunch of people trying to login and access their account. I'm getting about 1000 requests / 30 minutes for SELECT user.
Here's some more info:
Microsoft SQL Server 2008 R2 (SP2) - 10.50.4270.0 (X64) Nov 30 2012 17:11:43 Copyright (c) Microsoft Corporation Web Edition (64-bit) on Windows NT 6.1 (Build 7601: Service Pack 1) (Hypervisor)