We need to store simple log data in a database, but we also need to get and filter data in realtime, based on 'user', simple 'tag' (ie. redis, memcache, php, mysql, etc.) and timestamp. We need to scale horizontally and real fast data access on billions rows.

In a SQL approach, table can be like this:

ID | timestamp | tag | user       | log text
1  | 19543532  | 1   | root       | { text log }
2  | 19543532  | 3   | redis-user | { text log }

where tag 1 and 3 are different and related to another table (ie. tag_id | tag_name). I think this is a relational approach and we can create three index (timestamp, tag and user) in order to speed up data access.

What is a good practice to reproduce this in a NoSQL database like DynamoDB (AWS) where we can create only HASH or RANGE index? Does a SQL database fit better than a DynamoDB?

My first attempt is:

First table: ID hash index

ID | log text
1  | { text log }
2  | { text log }

Second table: USER, TIMESTAMP range index

user | timestamp | id
root | 123145122 | 1
redis| 123491241 | 2

Third table: TAG index

tag        | id
debug      | 1
production | 2

Thank you in advice!

1 Answer 1


w/o any secondary index I can think of PIVOT. (Data in Key-Value pair way)

enter image description here

Depending on the usage and query you are expecting you need to decided if key "LogText" needs to go in to another table or ok to put in Table-2.

You can have key on both columns and that will give you option of defining local secondary index to improve performance for complex queries like

ALL DATA where (TimeStamps is between x and y) and User = "ABC"

this can be done in two steps,

select ALL data where KeyIDs are IN 
( get unique KeyIDs for that satisfy this WHERE conditions.)
  • EAV (entity-attribute-value) models are famous for their poor performance; you can't honestly offer this as a solution that must "scale horizontally" and provide "real fast data access on billions rows".
    – mustaccio
    May 21, 2015 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.