I have a table of stuff and some of it needs to be processed with cron daily or every minute. On SQL table I would add column last_time_processed and save timestamp in it. The query would take rows like
SELECT * FROM table WHERE last_time_processed < 123456
and would add an index on this column.
How to do the same thing on DynamoDB? I have to select primary key to make a query and primary keys are different for different rows and I need to select all rows where last_time_processed < 1234.
The scan would make a job but it would read the entire table, which will eat a lot of RCU.
Thinking about another way to add an extra column which would have fake-primary-key-x and x would be anything between 1..10 so, in theory, it would spread GSI to 10 nodes and bypass 10GB limit/node. And then query on each primary key + sort by last_time_processed but it would require several queries to make full select.
Any ideas?
