Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a piece of code that performs inserts into highly denormalized tables. The tables have numbers of columns ranging from ~100 to 300+. This is SQL Server 2008 R2, running on Windows Server 2008.

Each insert consists of inserting to a number of tables under the same transaction. Some inserts are batched by NHibernate, but some cannot be, but they are all under the same transaction nonetheless.

When I perform inserts for say 500 times by repeatedly calling a piece of code that performs the insert, I get an average of ~360 ms.

The weird bit is, when I run the test code simultaneously using 4 processes (the same exe run from 4 different command prompts under windows server 2008), the insertion performance per call gets much better. I see bursts that go as fast as 90 ms (almost X4 faster). I'm measuring the insertion time from the code.

Since the 4 processes know nothing about each other, I'm assuming that this has something to do with SQL Server, but I have absolutely no clue why. I'd like to know why this is happening and if there is any configuration that would allow me to get the same performance when the inserts are not that frequent.

Suggestions regarding SQL Server monitoring methods to understand what is going on at the db level are equally welcome.

share|improve this question
up vote 11 down vote accepted

One possible reason is that four concurrent processes generate a more favourable pattern of log flushes - typically meaning that each log flush writes more data than is the case with a single executing process.

To determine if transaction log throughput/flush size is a factor, monitor:

Look for internal limits being reached. In SQL Server 2008 R2, there can be a maximum of 32 outstanding (asynchronous) log flush I/Os per database on 64-bit versions (only 8 on 32-bit). There is also a total size limit on the outstanding IOs of 3840KB.

More information and further reading:

share|improve this answer

Everything @PaulWhite says, plus...

If you have foreign keys in place, every insert will require a check to be done on each table referenced. It sounds to me like you are, as you're only getting 360ms, which feels slow to me.

Anyway, checking those tables is massively helped by having that data in RAM already, rather than having to load it into disk.

It sounds to me like loading the data into RAM is a significant part of your execution, and that it only needs to happen once.

It could also be effective plan caching, and that your queries need to be compiled the first time, with subsequent calls being able to avoid that phase.

share|improve this answer
    
Thanks Rob. My performance problem is associated to the high number of tables used during an insert. There are no foreign keys, I removed them for performance reasons and my model and domain requirements allow me to do that. I'm not loading data to RAM, and my inserts are shaped dynamically by incoming requests, which are changing all the time. I'm basically misusing a star/snowflake(ish) schema for OLTP and trying to get away with the best performance I can. – mahonya Feb 22 at 11:40
2  
@mahonya, even though you are not explicitly loading data into RAM, SQL Server must first read the needed index and data pages into buffer cache before performing the insert operation. Concurrent insert threads may have the effect of warming the cache such that one thread incurs the read overhead and the other access the data in cache. – Dan Guzman Feb 22 at 12:27
    
Thanks @DanGuzman - and yes, mahonya , there is a strong chance your cache is being nicely warmed. I'd be checking your waits to see if it's physical I/O causing your bottleneck. – Rob Farley Feb 22 at 13:11
    
Thanks @DanGuzman Agreed, the db index cache speedup is something that I'm used to seeing in postgres I probably misunderstood Rob's input. – mahonya Feb 22 at 14:31

some servers / cpus / os's remember the patterns. like cache.

Since you doing the same thing 4 times, i'm sure there are ways it can cut corners, What i am guessing is that the first way you do it, it thinks of it as one long proccess (example1) but in the second way is seens the reused code and runs it like cache (example2) or it could be the first proccess is to big to fit it all in the (ram example3).

example1: 0111110000110111110000111011111000011110111110000

example2: 0111110000|11|0111110000|111|0111110000|1111|0111110000

example3: 0111110000011111000001111100000111110000 example3: loop: 0111110000

I know ubuntu server does this with repeated mysql queryies. I can save them in cache, although really the only difference in time is 10-40mms but it adds up. When i was in school there was classes that showed you have to make programs (perl / php) use that cache to be faster.

But, it might depend on the program, what language is it, what is it compiled in or how it was programmed.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.