We are using elastic search hosted in AWS for apps wide search in our organization. We had initially dumped our existing data using logstash. However, we are struck at keeping the Elastic DB in sync with our databases. One option is running logstash in ec2 instance and installing Filebeat in our servers which will monitor the log files and send the updates to logstash and logstash will ingest the data into Elastic search. But we want to perform queries from different databases maintained on different servers and ingest the query resultant data into elastic search(Although this is in theory, I don't know if this approach works or not)
Another option is to manually add the trigger for each table and listen for updates in postgres and update the same in Elastic Search. This works, but I want to know if there is any better alternative.
Suggestion on any other alternatives would be appreciated. Thank you