The problem
Here is a very similar case discussed on pgsql.general. It's about the limitation in a b-tree index, but it's all the same because a GIN index uses a b-tree index for keys internally and therefore runs into the same limitation for key size (instead of item size in a plain b-tree index).
I quote the manual about GIN index implementation:
Internally, a GIN index contains a B-tree index constructed over keys,
where each key is an element of one or more indexed items
Either way, at least one array element in your column data
is too big to be indexed. If this is just a singular freak value or some kind of accident you may be able to truncate the value and be done with it.
For the purpose of the following demo I'll assume otherwise: lots of long text values in the array.
Simple solution
You could replace elements in your array data
with according hash values. And send look-up values through the same hash function. Of course, you probably want to store your originals in addition somewhere. With that, we almost arrive at my second variant ...
Advanced solution
You could create a look-up table for array elements with a serial
column as surrogate primary key (effectively a radical kind of hash value) - which is all the more interesting if involved element values are not unique:
CREATE TABLE elem (
elem_id serial NOT NULL PRIMARY KEY
, elem text UNIQUE NOT NULL
);
Since we want to look up elem
, we add an index - but an index on an expression this time, with only the first 10 characters of the long text. That should be enough in most cases to narrow a search down to one or a few hits. Adapt the size to your data distribution. Or use a more sophisticated hash function.
CREATE INDEX elem_elem_left10_idx ON elem(left(elem,10));
Your column data
would then be of type int[]
. I renamed the table to data
and got rid of the ominous varchar(50)
in your example:
CREATE TABLE data(
data_id serial PRIMARY KEY
, data int[]
);
Each array element in data
refers to a elem.elem_id
. At this point, you may consider to replace the array column with an n:m table, thereby normalizing your schema and allowing Postgres to enforce referential integrity. Indexing and general handling becomes easier ...
However, for performance reasons, the int[]
column in combination with a GIN index may be superior. Storage size is a lot smaller. In this case we need the GIN index:
CREATE INDEX data_data_gin_idx ON data USING GIN (data);
Now, each key of the GIN index (= array element) is an integer
instead of a longish text
. The index will be smaller by several orders of magnitude, searches will consequently be much faster.
The downside: before you can actually perform a search you have to look up the elem_id
from the table elem
. Using my newly introduced functional index elem_elem_left10_idx
, this, too, will be much faster.
You can do it all in one simple query:
SELECT d.*, e.*
FROM elem e
JOIN data d ON ARRAY[e.elem_id] <@ d.data
WHERE left(e.elem, 10) = left('word1234word', 10) -- match index condition
AND e.elem = 'word1234word'; -- need to recheck, functional index is lossy
You may be interested in the extension intarray
, that supplies additional operators and operator classes.
db<>fiddle here
Old sqlfiddle
data
contains a list of tags like demonstrated in this related blog post by Scott Snyder? If that is the case, I might have a better solution for you.