I'm using elastic search to index large amounts of sensor data for analytics purposes. The table has 4 million + rows and growing fast - expecting 40 million within the next year. This makes Elastic Search seem like a natural fit, especially with tools such as Kibana to easily display the data.

Elastic Search seems great, however there are are some more complex calculations that have to be performed as well. One such calculation is for our "average user time", where we take two data points (timestamp of item picked up and timestamp of item placed back), subtract them from each other and do an average of all these for one specific customer over a specific timeframe. The SQL query would look something like "select * from events where event_type = 'object picked up' or event_type = 'object placed back down'" then take all these events and get diffs on all their timestamps, add them all together then divide by count.

These types of calculations to my understanding are not the type of thing that Elastic Search is meant to do. I've had people recommend Hadoop but that could take a long time to get set up (we're a startup and have clients that need this info asap) and we can use a fast language like GO or Node/ Javascript to batch process things and add them to the DB periodically... but what is the RIGHT way to do this? Allowing for future scalability and working nicely with Elastic Search.

Our setup is: Rails, AngularJS, Elastic Search, Heroku, Postgres.


My experience is that some calculations you can perform using "aggregations" feature from elasticseach:

But at some point there is only the way to get some counters - results from queries that gives you not really the records, but numbers describing them - and preform calculations in separate own application using Elasticsearch API.

  • Please add the main points from those links. If they disappear, your answer will be much less useful. – dezso Sep 26 '16 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.