I'm trying to find a solution for dealing with a data set of logs of about 500Gb. Ideally what I search for is an index over log message keywords that I parse from the log messages. Like:
Log message
2019-05-06 12:43:32.133 ERROR [ 23][Repository] Updating record 356/23:2,4
Keywords: repository, updating, record, "356/23:2,4"
That search over the keywords like select from logs where keywords contains '356/23:2,4' (not an SQL).
So far I've tried DBMS proof-of-concept with keywords table, logs table and keywords-logs relation table. It works but to put it mildly, it is kind of not ideal. I've also tried influxdb, but it does not support multiple values per tag (otherwise it would be exactly what I looking for).
It seems to be quite basic stuff, so I hope there is just a right solution for that, ideally installed locally (not in cloud) and opensource. Any recommendation?
ideally installed locally (not in cloud) and opensource.OMG - the Evil Empire has gone Open Source? First, it was SQL Server on Linux (the cancer) and now the source has gone wild! Yaay... :-) But seriously, there is F/LOSS stuff out there for logs - the Elastic Stack, Solr or Sphinx (et al.). p.s welcome to the forum! :-)