33

I'm new to PostgreSQL and somewhat new to databases in general. Is there an established way of how we should index UUID values in Postgres? I'm split between using hashing and using a trie, unless there's already something built-in that it uses automatically. Whatever I use is going to be handling huge amounts of data.

The SP-GiST operator family "text_ops" indexes using a trie. Because UUIDs are quite long and very dissimilar, these sound appealing even though I would only ever do full match searches.

There's also a hash option. Hashing is O(1), and I won't need to do any comparisons besides equality of course, but because UUIDs are quite long, I'm afraid that generating hashes from them would waste a lot of time.

Or is this something that depends too much on system and use specifics?

I'd rather use bigserial in most cases, but I've been told to use uuid for this. We need uuid because we might have multiple servers using different databases, so there isn't a guarantee that we'll have unique bigints. We could use a different sequence (and seed) for each server, but it's still not as flexible as UUIDs. For example, we wouldn't be able to migrate database entries from one server to another without converting the IDs and their references everywhere.

  • 3
    I believe "federated database" is the buzzword for your situation. And, yes, UUIDs are the solution for that. That was the very reason UUIDs were invented decades ago: for sharing data amongst distributed systems without centralized coordination. – Basil Bourque Aug 18 '15 at 22:45
  • Months later: Indeed, the "federated database" Basil Bourque brought up is what we're going for. Not only do we have multiple servers, but we have clients (which can be thought of as more parts of the federated DB) creating IDs while offline, too. That's why we use UUIDs. – sudo Aug 19 '15 at 20:43
40

Use PostgreSQL's built-in uuid data type, and create a regular b-tree index on it.

There is no need to do anything special. This will result in an optimal index, and will also store the uuid field in as compact a form as is currently practical.

(Hash indexes in PostgreSQL prior to version 10 were not crash-safe and were really a historical relic that tended to perform no better than a b-tree anyway. Avoid them. On PostgreSQL 10 they've been made crash-safe and had some performance improvements made so you may wish to consider them.)

If for some reason you could not use the uuid type, you would generally create a b-tree on the text representation or, preferably, a bytea representation of the uuid.

| improve this answer | |
  • 2
    While the statement regarding hash indexes versus b-tree is a commonly held belief, I think it would be helpful to cite sources for such a claim. – Volte Apr 24 '17 at 20:30
  • 1
    As of PostgreSQL 10, hash indexes are now crash-safe. That said, hash indexes can only be used with =, so if you need any other operators, b-tree is still preferable. – rintaun Oct 31 '17 at 14:54
  • 1
    A couple of years later, in my experience, hash hasn't been much faster than b-tree, even in Postgres 10. But since hash indexes take so much less disk space than b-tree, it might be faster in a setup where big indexes become a problem, which I feel hasn't been the case for me. Well I'll keep an eye out now that I can actually use them safely in v10. – sudo Jan 30 '18 at 0:25
  • 1
    There are some good write ups on hash index perf improvements in v10 and v11: rhaas.blogspot.com/2017/09/… - amitkapila16.blogspot.com/2017/03/… – Glenn Morton Jun 30 '18 at 6:00
4

BRIN-index? If you use time-based (version 1) UUIDs then they are generated so that their value increase. In which case BRIN is suitable.

https://www.postgresql.org/docs/9.5/brin-intro.html :

BRIN stands for Block Range Index. BRIN is designed for handling very large tables in which certain columns have some natural correlation with their physical location within the table. A block range is a group of pages that are physically adjacent in the table; for each block range, some summary info is stored by the index. For example, a table storing a store's sale orders might have a date column on which each order was placed, and most of the time the entries for earlier orders will appear earlier in the table as well; a table storing a ZIP code column might have all codes for a city grouped together naturally.

BRIN indexes can satisfy queries via regular bitmap index scans, and will return all tuples in all pages within each range if the summary info stored by the index is consistent with the query conditions. The query executor is in charge of rechecking these tuples and discarding those that do not match the query conditions — in other words, these indexes are lossy. Because a BRIN index is very small, scanning the index adds little overhead compared to a sequential scan, but may avoid scanning large parts of the table that are known not to contain matching tuples.

The specific data that a BRIN index will store, as well as the specific queries that the index will be able to satisfy, depend on the operator class selected for each column of the index. Data types having a linear sort order can have operator classes that store the minimum and maximum value within each block range, for instance; geometrical types might store the bounding box for all the objects in the block range.

The size of the block range is determined at index creation time by the pages_per_range storage parameter. The number of index entries will be equal to the size of the relation in pages divided by the selected value for pages_per_range. Therefore, the smaller the number, the larger the index becomes (because of the need to store more index entries), but at the same time the summary data stored can be more precise and more data blocks can be skipped during an index scan.

Perfect for huge and "mostly" ordered data.

See this post for some benchmarks:

https://www.percona.com/blog/2019/07/16/brin-index-for-postgresql-dont-forget-the-benefits/

They genereated a 1.3 GB table of naturally ordered data (timestamps incemented). Then they generated a BRIN index (with pages_per_range = 32) and a B-Tree index on this database. Then they compared the SELECT execution time and the size of the indices. What they got:

B-Tree:

Planning Time: 22.225 ms Execution Time: 2.657 ms

public | testtab_date_idx | index | postgres | testtab | 171 MB

BRIN:

Planning Time: 0.272 ms Execution Time: 87.703 ms

public | testtab_date_brin_idx | index | postgres | testtab | 64 kB

meanwhile with no-index it would be:

Planning Time: 0.296 ms Execution Time: 1766.454 ms

Just to give a sense of orders.

What is important to discuss furthermore is the complexity of index update after INSERT of the two. While for BRIN it is O(1), since you write sequentially on the next free space on the memory and accordingly create new BRIN entries, however for B-Tree as we well know it is O(logN) (B-Trees) (higher the tree longer it takes).

| improve this answer | |
  • I think this is a very good answer. I don't understand all this hate... – Marcin Krupowicz Apr 14 at 9:49
  • @Najib, your edits look good and earned my upvote. My initial comment was meant to be constructive as the answer in its initial form didn't do much to explain the links. I hope you see the benefits you've now provided with a better explanation rather than just pasting in links without proper context. Everyone benefits from better answers, which is why I left the initial feedback. Sorry that your initial interaction with the community wasn't different, but I hope you stick around and make the community the place you'd like it to become. – John Eisbrener Apr 14 at 19:36
  • Also, I didn't downvote your answer at any point (I only downvote when answers are blatantly wrong). Like you, my initial interaction with the community wasn't exactly what I was expecting, but I've found this to be a great place for information and feedback. Don't let a few downvotes get the best of you, and I hope to see you stick around. – John Eisbrener Apr 14 at 19:42
  • 1
    @JohnEisbrener Thanks! Yes, I completely understood you, your comment was right, I shouldn't have written a short, shallow answer and I realized that soon and changed it for the better. However I think the way the pointing system works in SO could be improved, or maybe I took it too seriosuly. Anyways, thanks and hope I could contribute. – Najib Apr 14 at 22:09
  • Unfortunately, none of the RFC 4122 compliant UUID versions/variants are lexicographically sortable. Even the time-based ones - you can see this if you go to any online generator and generate a few. The first hex segment varies quickly, while the more stable hex segments come after. This is the opposite of what you want for a lexicographically sortable value. This is why SQL Server has the noncompliant NEWSEQUENTIALID. If one is willing to go non-RFC for PostGres, there's ULIDs which are made to be sortable. – DharmaTurtle Aug 22 at 16:45
3

Hash indexes are missing in action in PostgreSQL. PostgreSQL knows it needs hash indexes, and that it's code for hash indexes is old and moldy, but they don't remove it because they are waiting for someone to come along and overhaul hash indexing. See this thread:

http://www.postgresql.org/message-id/4407.1115698257@sss.pgh.pa.us

| improve this answer | |
  • Hash indexes work well in PostgreSQL under some circumstances, but I recently found they caused my queries to return no results when I tried optimizing with hash indexes on built-in UUID data type primary & foreign keys. There truly are benefits to hash indexes, if only they worked for all data types, and PostgreSQL devs know this, they're just too lazy to fix it themselves, and they keep their code situated as if they are praying to/for their eventual savior. – derekm Apr 10 '16 at 15:58
  • 2
    Someone has rescued hash indexes, I'm guessing because they play a critical role in data partitioning, which Pg10 has been focusing on: wiki.postgresql.org/wiki/… But they still don't give you everything I've seen theoretically useful in college database class ;) – sudo Jan 30 '18 at 0:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.