I have a message
table in Postgres 9.4 that contains a words
field of array
type, with random words of a message. Currently I have millions of messages:
\d messages
Table "public.messages"
Column | Type | Modifiers
------------------------------+-----------------------------+-----------
id_str | character varying(255) | not null
feed_id | integer |
message | character varying(255) |
posted_at | timestamp without time zone |
words | character varying(255)[] |
Indexes:
"messages_pkey" PRIMARY KEY, btree (id_str)
"index_messages_on_feed_id" btree (feed_id)
"index_messages_on_feed_id_posted_at" btree (feed_id, posted_at DESC NULLS LAST)
"index_messages_on_words" gin (words)
Now I have an array of words [A, B, C]
.
I need to find the number of messages that the words
contains each of the combinations of the list, i.e. [[A,B], [A,C], [B,C]]
.
Also I have this combination array created outside of the Postgres query already.
I can do this by creating 3 queries:
select count(*) from messages where words @> ARRAY['A','B']::varchar[];
select count(*) from messages where words @> ARRAY['A','C']::varchar[];
select count(*) from messages where words @> ARRAY['B','C']::varchar[];
Is there a way to do this in one query? And output the result in the following format?
A, B, count or [A, B], count
A, C, count
B, C, count
Edit:
Here's the query I run currently:
http://sqlfiddle.com/#!15/7c907/2/0
http://explain.depesz.com/s/Dot
And this one is using Erwin Brandstetter 's method:
http://sqlfiddle.com/#!15/7c907/1/0
http://explain.depesz.com/s/pr2
Now assume the messages table has more than 1 Million records, the CTE scan or seq scan in each of the query plan will become very slow.
GROUP BY
is your friend.\d tbl
in psql) is superior to describing the table definition with free text.words
? How many elements on average inwords
? How many unique elements in your search? You mentioned 1000 combinations? That would be ~ 45 distinct words in one search? You can probably make this (substantially) faster, but it will take some time to find 1000 combinations in a million arrays. the fiddles don't seem to work (the site may have problems), please include the query to work with in your question.messages.words
are around 15-20. And I have 50 distinct words, which is 1225 combinations, and this 1225 combinations are pre calculated. so I have this 1225 combination array ready. The problem right now is the performance of themessages.words
@> '{A, B}' even with thegin
index. it still does a seq scan on it. not sure how to improve this.