2

I'm working with a PostgreSQL table containing terabytes of data, and it grows by millions of rows weekly. Each row is identified by a KSUID, and my primary read patterns are:

  1. Retrieve a row by its KSUID.
  2. List rows by user_id in descending order, pagination acceptable.

Currently, the table is unpartitioned and read performance is sluggish. I'm contemplating partitioning the table by month using the KSUID column, leveraging its embedded uint32 timestamp, something like this:

CREATE TABLE table_y2023m09 PARTITION OF ksuid
FOR VALUES FROM ('[current_month_ts][128 zeros]') TO ('[next_month_ts][128 zeros]')

This allows each 'Get row by KSUID' query to be isolated to a single partition.

For listing rows by user_id, I'm considering keyset pagination:

SELECT *
FROM table_name
WHERE user_id = ?
  AND ksuid > last_seen_ksuid
ORDER BY ksuid
LIMIT 10;

However, this method still would need to search through multiple partitions depending on last_seen_ksuid, but I guess that with an index by user_id might be enough.

Questions:

  1. Is using KSUID as a partitioning key viable, especially given that the column can be represented as text or bytes?
  2. Is there a more efficient way to implement listing by user_id other than keyset pagination?
  3. Are there any pitfalls or performance issues I should be aware of with this approach?
  4. Would it be better to just partition based on created_at and extract the timestamp from the ksuid on application layer and add it explicitly to the query?
7
  • What do you want to achieve through partitioning? The answer will depend on that. Commented Sep 4, 2023 at 11:54
  • speed up the two read patterns that I mention
    – wheels
    Commented Sep 4, 2023 at 11:55
  • Thank you @vérace @laurenz-albe ! We can think of very simple schema (in reality it has like 20 columns) but the queries are very simple for example to get the row by id something like: select * from table where user_id = ? and ksuid = ? and for the keyset pagination something like what I've written in the question itself
    – wheels
    Commented Sep 4, 2023 at 12:25
  • for me the value is that they are k-sort (more or less sorted) so when you have an index by ksuid your writes are not so random, your btree doesn't need to rebalance constantly like happens with pure randomness with uuidv4 @vérace
    – wheels
    Commented Sep 4, 2023 at 12:48
  • 2
    If your search conditions are indexed, queries usually become slower with partitioning. A more promising pursuit is to analyze and tune your slow queries. Partitioning can still be useful to keep tables from growing too large, and particularly to get rid of old rows. Commented Sep 4, 2023 at 14:00

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.