Operation System error 665 with VLDB backup (The requested operation could not be completed due to a file system limitation)

Question

Has anyone seen the following error while doing backups? We have tried different LUNS from different storage pools and are seeing the same result.

Here is the error message: Operating system error 665 (The requested operation could not be completed due to a file system limitation).

NTFS Compression is not enabled on the drives. This failure usually comes when the backup is about half completed. I believe when the backup file size is extended again. We are striping against eight files. The database is 60+ TB. I also saw some references to file fragmentation and I see that these drives show 0% percent.

I also saw several references to DBCC CHECKDB and sparse files. We are just doing a full backup and not any DBCC CHECKDBs. The goal would be to do this on a secondary server until any errors are recorded.

Answer 1

This error usually means that NTFS cannot add more fragments to the file being written to. Normally, this happens for CHECKDB sparse files but it could happen to any file. It is not sparse file specific.

I suspect your NTFS volume is already heavily fragmented. If there are many free space holes NTFS is prone to create many small fragments for new files. I have just seen a 16GB paging file with 120,000 fragments that was created on such a volume.

There is no way to tell NTFS to allocate more smartly. (Subjective note: NTFS is really stupid when it comes to allocating files. Many simple improvements come to mind but no investment is being made.)

Use a defrag tool that compacts the drive and removes holes. O&O Defrag can do that for example. Of course you can manually compact by recreating the partition. But I don't know how realistic that is at your storage sizes.

I also saw some references to file fragmentation and I see that these drives show 0% percent.

But the free space could be fragmented. That's where new data goes.

Answer 2

_{Community Wiki answer generated from a comment on the question by Sean Gallardy}

I've seen this when the backups are large, the cluster size is small and the file grows. You can try using the largest NTFS cluster size available to format the volume.

There are also hotfixes that may help:

A heavily fragmented file in an NTFS volume may not grow beyond a certain size

It may also help to use trace flag 3042.

To allow the backup file to grow only as needed to reach its final size, use trace flag 3042. Trace flag 3042 causes the backup operation to bypass the default backup compression pre-allocation algorithm. This trace flag is useful if you need to save on space by allocating only the actual size required for the compressed backup. However, using this trace flag might cause a slight performance penalty (a possible increase in the duration of the backup operation).