Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm thinking that when we are working on full text search, we might want to view how sql server FT engine breaks the data and indexes it. For my case the column being indexed are simple nvarchar strings with spaces (no punctuation).

I'd want to be able to take the "tokens" out. Any idea if this is possible?

share|improve this question
By "tokens" do you mean stoplist words, or something else? Do you just mean you want to pull all of the words out and count occurrences? Per row or for the whole table? What exactly do you mean by "viewing the population"? – Aaron Bertrand Apr 16 '13 at 19:23
Tokens = individual words in a sentence when we break it. For e.g take string deception point. Tokens you'get are deception and point – deostroll Apr 17 '13 at 3:03
up vote 1 down vote accepted

I think you need to be having a look at the Full Text DMVs here:

Specifiably I think you want something like:

SELECT * FROM sys.dm_fts_index_keywords(db_id('AdventureWorks2012'), object_id('HumanResources.JobCandidate'))


SELECT * FROM sys.dm_fts_index_keywords_by_document(db_id('AdventureWorks2012'), object_id('HumanResources.JobCandidate'))
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.