Sql FILESTREAM garbage collection issue

Question

Updated 3x

I'm using FILESTREAM, but am having an issue with garbage collection. The FILESTREAM files are not being cleaned up when I set the column to null.

• In the event log, there is an enormous number of this error: Internal FILESTREAM error: failed to access the garbage collection table.

• It is error number 5571 on this page: http://msdn.microsoft.com/en-us/library/cc645602.aspx

• Using this blog post I've confirmed that a 'tombstone' table exists, but I cannot query it (Invalid object name 'sys.filestream_tombstone_xxxxxxx'.)

• I'm not deleting rows, I'm setting the FILESTREAM column to null, as per this msdn article: "When a FILESTREAM field is set to NULL, the BLOB data associated with the field is deleted."

• The database is on a local drive.

• The database is using Simple Recovery.

• I have executed a number of CHECKPOINTs when activity was very low. This seems to have done nothing. The error message seems to indicate the FILESTREAM garbage collector is trying to garbage collect, but cannot access the table to find out what to do.

• SQL Server 2008 (10.0.2531.0, SP1, Enterprise Edition

• There are no other errors preceding the 5571 errors that I can see in the SQL Server Log or Event Log.

DBCC CHECKDB

There was an error detected. What do I do about this?

DBCC results for 'sys.filestream_tombstone_1819153526'.
Msg 8951, Level 16, State 1, Line 1
Table error: table 'sys.filestream_tombstone_1819153526' (ID 1819153526). Data row does not have a matching index row in the index 'FSTSNCIdx' (ID 2). Possible missing or invalid keys for the index row matching:
Msg 8955, Level 16, State 1, Line 1
Data row (4:4712151:3) identified by (oplsn_fseqno = 62856 and oplsn_bOffset = 82647 and oplsn_slotid = 2) with index values 'file_id = 65537 and rowset_guid = '3F972309-9B0B-4C4F-939A-5618897050B4' and column_guid = '4A143C0D-B877-494E-B1E6-B70B0A834BB6' and oplsn_fseqno = 62856 and oplsn_bOffset = 82647 and oplsn_slotid = 2'.
There are 209624 rows in 3382 pages for object "sys.filestream_tombstone_1819153526".

Questions:

• How should permissions be set up for FILESTREAM to work? Other people have access to this server, and this database may have been restored from a backup at some point.

• How do I check FILESTREAM (Windows) share/container account permissions and account under which SQL Server runs?

Thanks to @vgv8 for prompting for more info and ideas.

Answer 1

It seems like insufficient info to tell something definitely. For example, are you deleting through Transact-SQL or through Win32API, etc.

The FILESTREAM data is not deleted immediately from file system because SQL Server transaction logging under full and bulk recovery models permit the crash recovery.

Have you deleted with CHECKPOINT

delete from tablename CHECKPOINT

or tried to execute CHECKPOINT statement or use simple recovery model?

Also, IMO,:

• changing of FILESTREAM (re)configuration require restart of SQL Server
• FILESTREAM data is restricted to LOCAL only drives
• FILESTREAM operations depend on hardware, see Paul Randal's white paper
• "deleting or renaming any FILESTREAM files directly through the file system will result in database corruption"

Check Security and Reliability sections of Paul Randal's white paper

Related discussions containing many subreferences:

• SQL Server Filestream - Delete “speed”
• How to force the filestream garbage collector to complete its work with the highest priority?

Update:
Have you checked permissions of FILESTREAM (Windows) share/container vs. account under which SQL Server runs under. It should have local administrator permissions. It is recommended that no other account be granted permissions on the data container

Update2:
From Paul Randal's white paper:

• "The file system access open operations do not wait for any locks. Instead, the open operations fail immediately if they cannot access the data because of transaction isolation. The streaming API calls fail with ERROR_SHARING_VIOLATION if the open operation cannot continue because of isolation violation"
• "Antivirus software ... access to the BLOB data in the affected file will be prevented, and to SQL Server the file will appear to have been deleted."
• "Note that for a table to have one or more FILESTREAM columns, it must also have a column of the uniqueidentifier data type that has the ROWGUIDCOL attribute. This column must not allow null values and must have either a UNIQUE or PRIMARY KEY single-column constraint"
• "anything that can prevent transaction log truncation may also prevent a FILESTREAM file being physically deleted. Some examples are:"
• FILESTREAM data containers cannot be nested

Run
DBCC CHECKDB

Update3:
I cannot coach you online. This is Q&A board. I already had been banned many times for infringing the rules on StackExchange having discussions instead of posting question or answer

Run

DBCC CheckDB (QPS8, repair_rebuild)

DBCC CheckDB (QPS8) WITH No_INFOMSGS, ALL_ERRORMSGS

and put output here.

To check/set NTFS permissions: right-click on folder in Windows --> Properties --> Security

To check/set MSSQLServer: in command line type

SQLServerManager10.msc 

navigate to SQL Server Services and double-click corresponding instance of SQL Server

Update 3b:
Probably, you should drop and recreate index or you have further deeper issues and you should restore database from backup or perform database repair permitting loss of data. This is quite separate topic/question.
I do not want to take any responsibility to guide you on it.