How to design a database for storing a sorted list?

Question

I am looking to store a sorted list inside a database. I want to perform the following operations efficiently.

1. Insert(x) - Insert record x into the table
2. Delete(x) - Delete record x from the table
3. Before(x,n) - Return the 'n' records preceding the record x in the sorted list.
4. After(x,n) - Return the 'n' records succeeding the record x in the sorted list.
5. First(n) - Return the first 'n' records from the sorted list.
6. Last(n) - Return the last 'n' records from the sorted list.
7. Compare(x,y) - Given two records x and y from the table, find if x > y.

The simple method I could think of is to store some kind of a 'rank' attribute in the table and query by sorting on that attribute. But in this method inserting/modifying a record with a rank becomes a costly operation. Is there a better method?

Specifically, I am looking to implement the table using Amazon's SimpleDB. But a general answer for a relational database should also be helpful.

Update on load profile:

Since I am planning this for a web application, it depends on the number of users that use the app.

If there are 100k active users (super optimism :P), then my very approximate estimate per day would be

500k selects, 100k inserts and deletes, 500k updates

I would expect the table to grow up to 500k in total.

I am looking to optimize on the updates, insert and the Compare operations. The rank of the items will be constantly changing and I need to keep the table updated.

Answer 1

If rank isn't completely arbitrary but is instead derivable from some other property (e.g. name, player score, etc.) then take a good look at Joel's answer.

If it is an arbitrary property of your data, then that should be stored as a column in your table of records. Assuming Amazon's SimpleDB is similar to the typical RDBMS, you can then index this column and quickly satisfy all your above queries with the appropriate indexing strategy. This is normal for an RDBMS.

Given that you expect high insert and update activity, but also relatively high read activity, I recommend doing the following:

• Cluster the table on the rank, especially if the vast majority of your queries are against rank. If not, or if choosing a clustering key is not available in SimpleDB, then just create an index with rank as the leading column. This would satisfy queries 3-6.
• An index on the record first and then rank (or, in the SQL Server world, just record and INCLUDE-ing rank, or just record if you've clustered on rank) would satisfy query 7.
• Operations 1 and 2 can be optimized by spacing out your data appropriately (i.e. setting the FILLFACTOR in SQL Server). This is especially important if you cluster on rank.
• As you insert or update ranks, maintain as much of a gap between rank numbers as possible to minimize that possibility that you will need to re-rank an existing record to accommodate a rank insert or update. For example, if you rank your records in steps of 1000 you leave enough room for about half that many changes and inserts with minimal chance you'll need to re-rank a record not directly involved in those changes.
• Every night re-rank all records to reset the rank gaps between them.
• You can tune the frequency of the mass re-rankings as well as the rank gap size to accommodate your expected number of inserts or updates relative to the number of existing records. So if you have 100K records and expect your inserts and updates to be 10% of that, leave enough room for 10K new ranks and re-rank nightly.
• Re-ranking 500K records is an expensive operation, but done once a day or week off-hours should be fine for a database like that. This off-hours mass re-ranking to maintain the rank gaps is what saves you having to re-rank many records for each rank update or insert during your normal and peak hours.

If you expect 100K+ reads on a 100K+ sized table I do not recommend using the linked list approach. It will not scale well to those sizes.

Answer 2

I generally use the "rank" method you describe. Rather than mess around with updating rows when items needed to be reordered I've often been able to get away with deleting all the records in the list and re-inserting new items in the proper order. This method is clearly optimized for retrieval.

An alternative approach would be to model the records as a linked list by using a "predecessor" reflexive foreign key column on the table:

ID   setID   item       predecessor
---  ------  ------     ------------
1    1       Apple      null
2    1       Orange     1
3    2       Cucumber   null
4    1       Pear       2
5    1       Grape      4
6    2       Carrot     3

You can easily retrieve a list and add and remove items with little overhead, but getting the records out in the proper order will be tricky. Perhaps there's a clever way to do it in a single query, probably with lots of aliased table joins.

I use this latter approach often when I'm modeling a tree-style relationship (categories, folders, sets and subsets). I've generally had a recursive function of some sort to reconstruct the full tree in my application.

Answer 3

I would think the thing to do is to store the property or properties that are used to calculate the rank and then build an index over them. Rather than trying to force the database to physically store the data in ranked order or using a manually managed linked list, why not let the database engine do what it was designed to do?

Answer 4

These are the limitations of a non-RDBMS like simpleDB. The features you require cannot be implemented on the DB side in simpleDB, they have to be implemented from the programming side/application.

For a RDBMS like SQL server, the features you require are rudimentary to the clustered index.

• Insert(x) - Insert record x into the table > Simple insert.
• Delete(x) - Delete record x from the table > Simple delete.

• Before(x,n) - Return the 'n' records preceding the record x in the sorted list. > Select top n results where x less than value and order by clause.

• After(x,n) - Return the 'n' records succeeding the record x in the sorted list. > Select top n results where x greater than value and order by clause.

• First(n) - Return the first 'n' records from the sorted list. > Select top n results.

• Last(n) - Return the last 'n' records from the sorted list. > Select top n results after order by desc.

• Compare(x,y) - Given two records x and y from the table, find if x > y. > TSQL IF statement.

Answer 5

Here's what I used to re-rank my Postgres table after every insert:

CREATE OR REPLACE FUNCTION re_rank_list() RETURNS trigger AS $re_rank_list$
DECLARE
    temprow record;
    row_idx integer := 1;    
BEGIN
    FOR temprow IN
    SELECT * FROM your_schema.your_list WHERE list_id = NEW.list_id ORDER BY rank ASC
    LOOP
        UPDATE your_schema.your_list SET rank = row_idx * 100 WHERE id = temprow.id;
        row_idx := row_idx + 1;
    END LOOP;
    RETURN NEW;
END;
$re_rank_list$ LANGUAGE plpgsql;


CREATE TRIGGER re_rank_list AFTER UPDATE ON your_schema.your_list_value
    FOR EACH ROW 
    WHEN (pg_trigger_depth() = 0)
    EXECUTE PROCEDURE re_rank_list();

For my use-case, performance is not a concern, but confidence that it'll never break or act oddly is important.