Interesting DBCC CHECKDB scenario

Question

I woke up this morning to corruption in one of my databases. I have a job that runs daily DBCC CHECKDB's for all databases on a server. The output indicated corruption in only ONE table (fortunately).

Msg 8935, Level 16, State 1, Line 1
Table error: Object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data). The previous link (5:216001) on page (5:216002) does not match the previous page (5:2603841) that the parent (5:3902159), slot 23 expects for this page.
Msg 8936, Level 16, State 1, Line 1
Table error: Object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data). B-tree chain linkage mismatch. (5:628894)->next = (5:628895), but (5:628895)->Prev = (5:2602254).
Msg 8935, Level 16, State 1, Line 1
Table error: Object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data). The previous link (5:1165706) on page (5:1165707) does not match the previous page (5:2602253) that the parent (5:1529), slot 33 expects for this page.
Msg 8980, Level 16, State 1, Line 1
Table error: Object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data). Index node page (5:2239068), slot 33 refers to child page (5:2602254) and previous child (5:628894), but they were not encountered.
Msg 8980, Level 16, State 1, Line 1
Table error: Object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data). Index node page (5:3902159), slot 22 refers to child page (5:2603841) and previous child (5:216001), but they were not encountered.
Msg 8981, Level 16, State 1, Line 1
Table error: Object ID 1946163866, index ID 10, partition ID 72057601280507904, alloc unit ID 72057601393229824 (type In-row data). The next pointer of (5:1212172) refers to page (5:3489726). Neither (5:3489726) nor its parent were encountered. Possible bad chain linkage.
Msg 8980, Level 16, State 1, Line 1
Table error: Object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data). Index node page (5:1529), slot 32 refers to child page (5:2602253) and previous child (5:1165706), but they were not encountered.
Msg 8978, Level 16, State 1, Line 1
Table error: Object ID 1946163866, index ID 10, partition ID 72057601280507904, alloc unit ID 72057601393229824 (type In-row data). Page (5:3489727) is missing a reference from previous page (5:1212172). Possible chain linkage problem.
Msg 2533, Level 16, State 1, Line 1
Table error: page (5:2602253) allocated to object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data) was not seen. The page may be invalid or may have an incorrect alloc unit ID in its header.
Msg 2533, Level 16, State 1, Line 1
Table error: page (5:2602254) allocated to object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data) was not seen. The page may be invalid or may have an incorrect alloc unit ID in its header.
Msg 2533, Level 16, State 1, Line 1
Table error: page (5:2602255) allocated to object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data) was not seen. The page may be invalid or may have an incorrect alloc unit ID in its header.
Msg 2533, Level 16, State 1, Line 1
Table error: page (5:2603840) allocated to object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data) was not seen. The page may be invalid or may have an incorrect alloc unit ID in its header.
Msg 2533, Level 16, State 1, Line 1
Table error: page (5:2603841) allocated to object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data) was not seen. The page may be invalid or may have an incorrect alloc unit ID in its header.
Msg 2533, Level 16, State 1, Line 1
Table error: page (5:2603842) allocated to object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data) was not seen. The page may be invalid or may have an incorrect alloc unit ID in its header.
Msg 8935, Level 16, State 1, Line 1
Table error: Object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data). The previous link (5:2602255) on page (5:15918) does not match the previous page (5:15917) that the parent (5:3904979), slot 8 expects for this page.
Msg 8936, Level 16, State 1, Line 1
Table error: Object ID 1946163866, index ID 11, partition ID 72057601280442368, alloc unit ID 72057601393164288 (type In-row data). B-tree chain linkage mismatch. (5:15917)->next = (5:15918), but (5:15918)->Prev = (5:2602255).
CHECKDB found 0 allocation errors and 16 consistency errors in table 'AperioCompare.PolicyAmendmentCompare' (object ID 1946163866).
CHECKDB found 0 allocation errors and 16 consistency errors in database 'Aperio'.
repair_allow_data_loss is the minimum repair level for the errors found by DBCC CHECKDB (Aperio).

I'm always looking for the last line so I can figure out the minimum level of recovery involved. In this case, it says "repair_allow_data_loss is the minimum repair level for the errors found by DBCC CHECKDB".

I looked at the Object ID and verified OBJECT_NAME(1946163866) indeed pointed to my ONE table. The error report showed problems with index ID's 10 and 11. I then selected from sys.indexes for object_id=1946163866 and found the index rows for indexes 10 and 11 - both of these indexes are non-clustered. I dropped and recreated both non-clustered indexes and re-ran my DBCC CHECKDB - no errors were reported.

I'm puzzled by the minimum level of recovery reported by the DBCC CHECKDB as being repair_allow_data_loss. Shouldn't this be rebuild?

Am I missing something obvious here?

Answer 1

Yup - there are some errors where CHECKDB picks REPAIR_ALLOW_DATA_LOSS even though it's a nonclustered index. You should be able to manually rebuild these (begin tran, disable, rebuild, commit tran). My guess is a lost write, stale read, or possibly a new bug in 2016.

Answer 2

Well you are deleting data even though it is only the index. ;-) Deleting a non-clustered index however does not incur data loss.

There is an article over on stackoverflow that can give you some more information:
What (kind of) data is lost when using REPAIR_ALLOW_DATA_LOSS?

It references an article from Paul S. Randall's SQLSkills.com website where he writes in an article:

REPAIR_ALLOW_DATA_LOSS is the repair level that DBCC CHECKDB recommends when it finds corruptions. This is because fixing nearly anything that’s not a minor non-clustered index issue requires deleting something to repair it. So, REPAIR_ALLOW_DATA_LOSS will delete things. This means it will probably delete some of your data as well. If, for instance it finds a corrupt record on a data page, it may end up having to delete the entire data page, including all the other records on the page, to fix the corruption. That could be a lot of data. For this reason, the repair level name was carefully chosen. You can’t type in REPAIR_ALLOW_DATA_LOSS without realizing that you’re probably going to lose some data as part of the operation.

Short:

This is because fixing nearly anything that’s not a minor non-clustered index issue requires deleting something to repair it.

So there you have it. If it were something other than a non-clusterd index you would be losing data.

Fixing the index issues manually and then re-running the DBCC checkdb command is probably the best option as you stay in control of data loss. If after repairing the index you still have corrupt data, then you have to find out how important the data is and if you have to restore the database to a different server to recover the corrupt data.

BTW: Paul was on the Microsoft SQL Server development team for DBCC CHECKDB command back in version 2000 of SQL Server and optimized everything in the 2005 version Bio