0

I'm considering the use of InfluxDB to store thousands of measurements per second from routers in a network. Each measurement is lightweight (between 10-20 bytes). Now, I'm wondering what is the best strategy: InfluxDB + Zabbix? Or Telegraf? Or another alternative?

  1. InfluxDB + Zabbix: according to

https://www.zabbix.com/integrations/influxdb

there is a possible integration. However, it seems the integration involves taking data that was collected in a Zabbix database and moving it to an InfluxDB to take advantage of InfuxDB using much less storage. So they meant it to be for archiving large amounts of data.

However, we believe Zabbix itself is too slow, no? The integration above would not work for online data collection, right? We need fast storage, and we do not want Zabbix to be a bottleneck. Are there known benchmarks and recommendations about Zabbix + InfluxDB integration, for online collection + storage?

  1. InfluxDB + Telegraf: according to

    https://www.influxdata.com/blog/monitoring-openwrt-with-telegraf/

there is also the possibility of integrating InfluxDB + Telegraf for data collection. Are there best practices and/or benchmarks about data collection with Telegraf + storage with InfluxDB?

  1. Are there other alternatives and/or suggestions for large scale data collection and integration with InfluxDB?

1 Answer 1

0

You might want to look into using Spark, and if are able to introduce AWS into the architecture, Kinesis Data Firehouse.

Spark: https://medium.com/analytics-vidhya/processing-time-series-data-in-real-time-with-influxdb-and-structured-streaming-d1864154cf8b

Another article on Spark: https://www.influxdata.com/blog/zeppelin-spark-and-influxdb-for-big-data-time-series-scenarios/

Kinesis Data Firehose: https://aws.amazon.com/kinesis/data-firehose/?kinesis-blogs.sort-by=item.additionalFields.createdDate&kinesis-blogs.sort-order=desc

Spark is fairly widely used these days and integrates with InfluxDB, and Kinesis Data Firehose is used for heterogeneously ingesting large amounts of data in real time.

3
  • 1
    Thanks! Dear J.D., the first link is broken. Please, what is the correct reference?
    – Daniel S.
    Nov 3, 2020 at 0:40
  • @Daniel S. Sorry about that, they updated their link. I fixed the link and added an additional relevant article for InfluxDB using Spark.
    – J.D.
    Nov 3, 2020 at 1:14
  • Not sure why someone randomly downvoted me, I fixed the link and added additional useful information.
    – J.D.
    Nov 3, 2020 at 1:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.