Here is a very similar case discussed on pgsql.general. It's about the limitation in a b-tree index, but it's all the same because a GIN index uses a b-tree index for keys internally and therefore runs into the same limitation for key size (instead of item size in a plain b-tree index).
I quote the manual about GIN index implementation:
Internally, a GIN index contains a B-tree index constructed over keys,
where each key is an element of one or more indexed items
Either way, at least one array element in your column
data is too big to be indexed. If this is just a singular freak value or some kind of accident you may be able to truncate the value and be done with it.
For the purpose of the following demo I'll assume otherwise: lots of long text values in the array.
You could replace elements in your array
data with according hash values. And send look-up values through the same hash function. Of course, you probably want to store your originals in addition somewhere. With that, we almost arrive at my second variant ...
You could create a look-up table for array elements with a
serial column as surrogate primary key (effectively a radical kind of hash value) - which is all the more interesting if involved element values are not unique:
CREATE TABLE elem (
elem_id serial NOT NULL PRIMARY KEY
, elem text UNIQUE NOT NULL
Since we want to look up
elem, we add an index - but an index on an expression this time, with only the first 10 characters of the long text. That's should be enough in most cases to narrow a search down to one or a few hits. Adapt the size to your data distribution. Or use a more sophisticated hash function.
CREATE INDEX elem_elem_left10_idx ON elem(left(elem,10));
data would then be of type
int. I renamed the table to
data and got rid of the ominous
varchar(50) you had in your example:
CREATE TEMP TABLE data(
data_id serial PRIMARY KEY
, data int
Each array element in
data refers to a
elem.elem_id. At this point, you may consider to replace the array column with an n:m table, thereby normalizing your schema and allowing Postgres to enforce referential integrity. Indexing and general handling becomes easier ...
However, for performance reasons, the
int column in combination with a GIN index may be superior. Storage size is a lot smaller. In this case we need the GIN index:
CREATE INDEX data_data_gin_idx ON data USING GIN (data);
Now, each key of the GIN index (= array element) is an
integer instead of a longish
text. The index will be smaller by several orders of magnitude, searches will consequently be much faster.
The downside: before you can actually perform a search you have to look up the
elem_id from the table
elem. Using my newly introduced functional index
elem_elem_left10_idx, this, too, will be much faster.
You can do it all in one simple query:
SELECT d.*, e.*
FROM elem e
JOIN data d ON ARRAY[e.elem_id] <@ d.data
WHERE left(e.elem, 10) = left('word1234word', 10) -- match index condition
AND e.elem = 'word1234word'; -- need to recheck, functional index is lossy
You may be interested in the extension
intarray, that supplies additional operators and operator classes.