Why filtered index on IS NULL value is not used?

Question

Assume we have a table definition like this:

CREATE TABLE MyTab (
    ID INT IDENTITY(1,1) CONSTRAINT PK_MyTab_ID PRIMARY KEY
    ,GroupByColumn NVARCHAR(10) NOT NULL
    ,WhereColumn DATETIME NULL
    )

And a filtered non-clustered index like this:

CREATE NONCLUSTERED INDEX IX_MyTab_GroupByColumn ON MyTab 
    (GroupByColumn)
WHERE (WhereColumn IS NULL) 

Why this index is not "covering" for this query:

SELECT 
    GroupByColumn
    ,COUNT(*)
FROM MyTab
WHERE WhereColumn IS NULL
GROUP BY GroupByColumn

I'm getting this execution plan:

The KeyLookup is for the WhereColumn IS NULL predicate.

Here is the plan: https://www.brentozar.com/pastetheplan/?id=SJcbLHxO7

Answer 1

Why this index is not "covering" for this query:

No good reason. That is a covering index for that query.

Please vote for the feeback item here: https://feedback.azure.com/d365community/idea/ffbf409a-5125-ec11-b6e6-000d3a4f0da0

And as a workaround include the WhereColumn in the filtered index:

CREATE NONCLUSTERED INDEX IX_MyTab_GroupByColumn 
ON MyTab (GroupByColumn) include (WhereColumn)
WHERE (WhereColumn IS NULL) 

Answer 2

I had the same issue I think when doing some testing weeks ago. I have a query with a primary predicate that requires that results returned have a NULL closedatetime and I thought about using a filtered index as 25K of 2M+ records are NULL and this figure will decrease very soon.

The filtered index didn't get used - I assumed due to 'non-uniqueness' or commonality - until I found a Microsoft support article that says:

To resolve this issue, include the column that is tested as NULL in the returned columns. Or, add this column as include columns in the index.

So adding the column to the Index (or Include) seems to be the official MS response.

Answer 3

I had the same situation. To me solution was to include the needed column (the one from the index filter) into the INCLUDE fields. I find it cheaper than putting it into directly searchable part of the index.

Actually, I've had the same situation even with the regular integer column, having a 1 in it.

Apparently, the SQL Server is not smart enough to deduce that the wanted value is already achieved.