Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I have a system that I'm building based on ElasticSearch. It works great, except I'm having trouble with one of my facets. I've got about 10 million documents, and each document has 1 to 200 interesting phrases that have been detected in the document. I have a facet that I use to calculate the "most common interesting phrases" and this times out.

The reason is that this has high cardinality. There are very few of these that happen more than 1 time. But, the ones that do, I need to count. And, I need to be able to query this based on interesting phrases in the last day, week, month, quarter. So, the counts need to be time-based.

I'm looking at a number of alternatives to solve this. I've considered using Graphite, but I'm not sure of the high cardinality. I've manually paged through all it of it in ES, and built it myself. But, that seems like overkill.

And, realistically, we need to do this type of counting for more than just the one series. This is the worst offender, but we do this with a lot of items.

Any thoughts on a better database to use? Or, would Graphite do this well?

Again, we're typically pulling the top 10/20/50 phrases over a timespan in high cardinality data.

EDIT

For those of you asking.. here's the ES query.

curl -X GET '1.2.3.4:9200/mentions/mention/_search?search_type=count&size=10&pretty' -d '{
  "query":{
    "match":{
      "keyword":{
        "query":[
          "hair accessories"
        ]
      }
    }
  },
  "facets":{
    "phrases":{
      "terms":{
        "field":"phrases",
        "size":10,
        "all_terms":false
      }
    }
  },
  "size":10}'
share|improve this question
    
can you show me example of time-out query? –  neo Jan 15 '13 at 13:52
    
sure.. here you go. –  Williamf Jan 15 '13 at 16:30
    
so, the thing with this query, is that even though the query only matches 14k documents, its still takes 55 seconds. So, its like when you add the facet, the query is ignored. –  Williamf Jan 15 '13 at 16:32
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.