I have a system that I'm building based on ElasticSearch. It works great, except I'm having trouble with one of my facets. I've got about 10 million documents, and each document has 1 to 200 interesting phrases that have been detected in the document. I have a facet that I use to calculate the "most common interesting phrases" and this times out.
The reason is that this has high cardinality. There are very few of these that happen more than 1 time. But, the ones that do, I need to count. And, I need to be able to query this based on interesting phrases in the last day, week, month, quarter. So, the counts need to be time-based.
I'm looking at a number of alternatives to solve this. I've considered using Graphite, but I'm not sure of the high cardinality. I've manually paged through all it of it in ES, and built it myself. But, that seems like overkill.
And, realistically, we need to do this type of counting for more than just the one series. This is the worst offender, but we do this with a lot of items.
Any thoughts on a better database to use? Or, would Graphite do this well?
Again, we're typically pulling the top 10/20/50 phrases over a timespan in high cardinality data.
EDIT
For those of you asking.. here's the ES query.
curl -X GET '1.2.3.4:9200/mentions/mention/_search?search_type=count&size=10&pretty' -d '{
"query":{
"match":{
"keyword":{
"query":[
"hair accessories"
]
}
}
},
"facets":{
"phrases":{
"terms":{
"field":"phrases",
"size":10,
"all_terms":false
}
}
},
"size":10}'