I'm working with a PostgreSQL table containing terabytes of data, and it grows by millions of rows weekly. Each row is identified by a KSUID, and my primary read patterns are:
- Retrieve a row by its KSUID.
- List rows by
user_id
in descending order, pagination acceptable.
Currently, the table is unpartitioned and read performance is sluggish. I'm contemplating partitioning the table by month using the KSUID column, leveraging its embedded uint32 timestamp, something like this:
CREATE TABLE table_y2023m09 PARTITION OF ksuid
FOR VALUES FROM ('[current_month_ts][128 zeros]') TO ('[next_month_ts][128 zeros]')
This allows each 'Get row by KSUID' query to be isolated to a single partition.
For listing rows by user_id
, I'm considering keyset pagination:
SELECT *
FROM table_name
WHERE user_id = ?
AND ksuid > last_seen_ksuid
ORDER BY ksuid
LIMIT 10;
However, this method still would need to search through multiple partitions depending on last_seen_ksuid
, but I guess that with an index by user_id
might be enough.
Questions:
- Is using KSUID as a partitioning key viable, especially given that the column can be represented as text or bytes?
- Is there a more efficient way to implement listing by
user_id
other than keyset pagination? - Are there any pitfalls or performance issues I should be aware of with this approach?
- Would it be better to just partition based on
created_at
and extract the timestamp from the ksuid on application layer and add it explicitly to the query?
select * from table where user_id = ? and ksuid = ?
and for the keyset pagination something like what I've written in the question itself