I'm thinking that when we are working on full text search, we might want to view how sql server FT engine breaks the data and indexes it. For my case the column being indexed are simple nvarchar strings with spaces (no punctuation).

I'd want to be able to take the "tokens" out. Any idea if this is possible?

  • 2
    By "tokens" do you mean stoplist words, or something else? Do you just mean you want to pull all of the words out and count occurrences? Per row or for the whole table? What exactly do you mean by "viewing the population"? Commented Apr 16, 2013 at 19:23
  • Tokens = individual words in a sentence when we break it. For e.g take string deception point. Tokens you'get are deception and point
    – deostroll
    Commented Apr 17, 2013 at 3:03

1 Answer 1


I think you need to be having a look at the Full Text DMVs here: http://msdn.microsoft.com/en-us/library/ms174971%28v=sql.105%29.aspx

Specifiably I think you want something like:

SELECT * FROM sys.dm_fts_index_keywords(db_id('AdventureWorks2012'), object_id('HumanResources.JobCandidate'))


SELECT * FROM sys.dm_fts_index_keywords_by_document(db_id('AdventureWorks2012'), object_id('HumanResources.JobCandidate'))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.