Return to Question

Update - Down goes the CIFS server
Today the identical and functioning CIFS server started exhibiting IO requests hanging. This server wasn't running SQL Server, just plain Windows Web Server 2008 R2 serving shares over CIFS. As soon as I disabled TOE on it as well, everything was back to running smooth.

Just confirms I won't ever be using TOE on Broadcom NICs again, if I can't avoid the Broadcom NICs at all, that is.

Update - Wtf, Broadcom?
Based on Mark Storey-Smiths suggestions and Kyle Brandts previous experiences with Broadcom NICs, I decided to do some experimentation. As we've got multiple active paths, I could relatively easily change the configuration of the NICs one by one without causing any outages.

Disabling TOE and Large Send Offload yielded a near perfect run:

Processed 1064672 pages for database 'XXX', file 'XXX' on file 1.
Processed 21 pages for database 'XXX', file 'XXX' on file 1.
BACKUP DATABASE successfully processed 1064693 pages in 58.533 seconds (142.106 MB/sec).

So which is the culprit, TOE or LSO? TOE enabled, LSO disabled:

Didn't finish the backup as it took forever - just as the original problem!

TOE disabled, LSO enabled - looking good:

Processed 1064680 pages for database 'XXX', file 'XXX' on file 1.
Processed 29 pages for database 'XXX', file 'XXX' on file 1.
BACKUP DATABASE successfully processed 1064709 pages in 59.073 seconds (140.809 MB/sec).

And as a control, I disabled both TOE and LSO to confirm the issue was gone:

Processed 1064720 pages for database 'XXX', file 'XXX' on file 1.
Processed 13 pages for database 'XXX', file 'XXX' on file 1.
BACKUP DATABASE successfully processed 1064733 pages in 60.675 seconds (137.094 MB/sec).

In conclusion it seems the enabled Broadcom NICs TCP Offload Engine caused the problems. As soon as TOE was disabled, everything worked like a charm. Guess I won't be ordering any more Broadcom NICs going forward.

Update - Wait stats
I did three tests to gather some wait stats. Wait stats are queried using Glenn Berry / Paul Randals script. And just to confirm - the backups are not being done to tape, but to an iSCSI LUN. Results are similar if done to local disk, with results similar to the NUL backup.

Cleared stats. Ran for 10 minutes, normal load:

Cleared stats. Ran for 10 minutes, normal load + normal backup running (didn't complete):

Cleared stats. Ran for 10 minutes, normal load + NUL backup running (didn't complete):