8

Something very odd is happening here.

I have a query that looks like this.

SELECT CAST(FT.DOP AS SMALLINT) FROM TRACKING_DATA WHERE date > @mydate and identifier = 0000000000

When run as a raw query it returns data just fine.

When i place it in a stored procedure that alters the where clause it throws this error.

Msg 244, Level 16, State 2, Procedure myprocedure, Line 107 [Batch Start Line 2]
The conversion of the varchar value '58629' overflowed an INT2 column. Use a larger integer column.

So here is the oddity. I go through all the possible data for that where clause with a query like this.

SELECT DISTINCT DOP FROM TRACKING_DATA where identifier = 000000000000

and

SELECT DISTINCT CAST(DOP AS smallint) FROM TRACKING_DATA where identifier = 000000000000

And this is what i get back.

17
12
9
19
8
14
6
16
11
13
7
10
0
18
5
15
4

Nowhere does it have anything remotely too large for a SMALLINT. So i thought, ok maybe its a non-printable ASCII character. But i can't find any.

I'm a little perplexed at the moment. It runs find as a raw query, explodes as a procedure and all possible data based on the where is valid. My only suspicion is that the query plan is doing something odd with filtering or maybe runs with different validation when run as a procedure.

marked as duplicate by Paul White sql-server Apr 5 at 5:30

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

11

You're better off not depending on the indexing to avoid these errors, and instead writing the query to guard against this situation. Since you're on SQL Server 2012, one option would be to use TRY_CAST:

SELECT TRY_CAST(FT.DOP AS SMALLINT) 
FROM TRACKING_DATA 
WHERE date > @mydate and identifier = 0000000000;

This will result in NULL being selected for values that fail to convert from varchar to smallint. But as long as you know there aren't any results like that, or your application can handle the NULL results, you should be good.

11

So it was the damn query plan it kept deciding to use.

The value that it was exploding on was present in the table (which it shouldn't be but that's another problem) but it was associated with a completely different identifier.

The query in question was running a clustered seek on an index that covered the date but not the identifier this caused it to scan the ENTIRE table which is so, so, so wrong for a number of reasons.

Added a proper index for the where clause and the procedure is happily humming away now as it filters on the identifier before going after the date. I wish i could get before / after statistics but i'm pretty sure its running faster now too.

  • 1
    You beat me to your own answer. Sometimes stored procedures use different query plans than simple ad-hoc queries, and this can result in the system processing data in a different order. Usually, indexing is a good workaround, but sometimes you need to move the last operation outside of the data retrieval for things to work, i.e. SELECT CAST(FT.DOP AS SMALLINT) FROM (SELECT FT.DOP FROM TRACKING_DATA WHERE date > @mydate and identifier = 000000) As FT – Laughing Vergil Apr 3 at 16:15
  • 2
    @LaughingVergil - That is not guaranteed to work. The CAST can still get pushed before the filter, the most reliable way is to use TRY_CONVERT so then it doesn't matter if it runs against rows that would fail the cast but later be eliminated anyway – Martin Smith Apr 3 at 16:37
1

You are obviously getting different query plans and your cast is being tried against values which are filtered out by the where clause, I wouldn’t bother trying to figure out why — whatever the cause, it could obviously happen for a different reason later. Relying upon filtering is unreliable.

The thing to do is to write a query that won’t fail regardless of any index, hint, or statistics.

I think there are 3 ways to do this:

  1. Use a temp table/table variable.

    declare @result table (dop varchar);
    Insert into @result
    SELECT FT.DOP  FROM TRACKING_DATA 
    WHERE date > @mydate and identifier = 0000000000
    
    SELECT cast(dop as SMALLINT) dop from @result
    
  2. Use a case statement:

    SELECT CAST(case when FT.DOP like '%[^0-9]%' then null else ft.dop end AS SMALLINT) 
    FROM TRACKING_DATA 
    WHERE date > @mydate and identifier = 0000000000
    
  3. Use try_cast instead and of cast:

    SELECT try_CAST(FT.DOP AS SMALLINT) 
    FROM TRACKING_DATA 
    WHERE date > @mydate and identifier = 0000000000
    

Personally I would recommend 3 if possible.

Not the answer you're looking for? Browse other questions tagged or ask your own question.