2

I'm looking for an alternative to executing the CLUSTER command on large tables to keep data efficiently organized on disk, since it requires an ACCESS EXCLUSIVE lock on tables being clustered, making them unavailable for a considerable amount of time.

After reading about covering indexes I got the idea of creating a btree multi-column index specifying all of my table columns, and not just those relevant for the query condition.

The benefits, I suppose, would be twofold:

  1. The query plan would be optimized due to the condition relevant columns being specified first on the index, according to the query's expected data fetch pattern.
  2. All remaining columns would also be included in the index, meaning this index would be a copy of the table and already organized efficiently on disk due to the btree structure.

I believe queries over this index would be very efficient and since it is covering the whole table wouldn't need to access table data scattered on disk.

Would like to ask if this a reasonable solution, and what are the potential downsides to this approach? (besides the additional storage space required for the covering index and expected - small? - decrease in insert/update/delete performance)

Improve this question
5
  • 1
    If you have entirely static data, you might get away with indexing the world, but if your data changes, you are going to pay the costs of resource usage to keep all the attributes indexed. As far as not using CLUSTER, have you considered pg_repack?
    – bma
    Dec 10, 2017 at 22:50
  • It is mostly static data, less than 0.1% is altered after insertion. I'm considering pg_repack as well, but was trying to find a solution in which inserted data would immediately be dealt with, not requiring to run a long "all or nothing" repack operation periodically.
    – Thomas C. G. de Vilhena
    Dec 11, 2017 at 0:31
  • Did you verify that clustering your table in fact improves the performance of all your queries? A clustered index typically only improves the performance for a small fraction of queries, not all of them.
    – a_horse_with_no_name
    Dec 11, 2017 at 7:14
  • @a_horse_with_no_name Yes I did. The query I want to optimize fetches a few thousand rows from the table based on a two column key. From my tests the query will execute aprox. 9x faster if rows are efficiently organized on disk.
    – Thomas C. G. de Vilhena
    Dec 11, 2017 at 16:29
  • Other queries run less often and don't fetch too much data, so a minor decrease in their performance is acceptable.
    – Thomas C. G. de Vilhena
    Dec 11, 2017 at 16:42

1 Answer 1

Reset to default
3

PostgreSQL does not have covering indexes. There has been some work on them, but it has not yet been accepted into the code tree.

So there are limitations. If you have unique constraint on some prefix of the indexed columns, you will have to have a separate index to support that, it can't piggy-back on your larger index. Also, some data types do not support Btree operators which means they cannot be included into a btree index. For example, XML or many geometric types.

Also, an indexed value cannot exceed 1/3 of a page, or about 2700 bytes, so inclusion of some wide columns into the index could fail on this grounds (I don't think covering indexes, if they existed, would solve this problem)

Finally, you only avoid consulting the table for those table pages marked as "all visible". If the table is mostly static, then is probably not a problem. But it the table is very dynamic, it will take very aggressive vacuuming to maintain the "all visible" count.

Partitioning the table may be another option. It is not nearly as fine-grained as clustering it, but can impose less of an ongoing burden. Particularly if all the active updates go against 1 or 2 partitions, while the rest are effectively read-only.

Improve this answer
1
  • After evaluating the possible solutions we decided to use a staging table due to our data insertion pattern. Once a day we will move data from the stage table to the main table ordering the insertions by our "clustering" index. Partitioning would not be a good alternative due to the high number of partitioning keys. We expect to drastically recude the numberof disk pages loaded to complete our main query execution.
    – Thomas C. G. de Vilhena
    Dec 12, 2017 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.