We need to store simple log data in a database, but we also need to get and filter data in realtime, based on 'user', simple 'tag' (ie. redis, memcache, php, mysql, etc.) and timestamp. We need to scale horizontally and real fast data access on billions rows.
In a SQL approach, table can be like this:
ID | timestamp | tag | user | log text
1 | 19543532 | 1 | root | { text log }
2 | 19543532 | 3 | redis-user | { text log }
where tag 1 and 3 are different and related to another table (ie. tag_id | tag_name). I think this is a relational approach and we can create three index (timestamp, tag and user) in order to speed up data access.
What is a good practice to reproduce this in a NoSQL database like DynamoDB (AWS) where we can create only HASH or RANGE index? Does a SQL database fit better than a DynamoDB?
My first attempt is:
First table: ID hash index
ID | log text
1 | { text log }
2 | { text log }
Second table: USER, TIMESTAMP range index
user | timestamp | id
root | 123145122 | 1
redis| 123491241 | 2
Third table: TAG index
tag | id
debug | 1
production | 2
Thank you in advice!