0

I want to perform KNN on 512-dimensional vector using the solution suggested here.

  1. I couldn't create table with 512-d cube array. Solved by changing the CUBE_MAX_DIM in the source code.
  2. Couldn't create index on table with dimensions > 200. Set block_size=32. Now works with dimensionality<=510.
  3. Got new error after changing block_size and trying to create index on table with 512-d vectors ERROR: index row requires 8208 bytes, maximum size is 8191

Is it possible to increase this limit?

0

That would require significant changes, and I doubt it can be done easily.

See these excerpts from src/include/access/itup.h:

/*
 * Index tuple header structure
 *
 * All index tuples start with IndexTupleData.  If the HasNulls bit is set,
 * this is followed by an IndexAttributeBitMapData.  The index attribute
 * values follow, beginning at a MAXALIGN boundary.
 *
 * Note that the space allocated for the bitmap does not vary with the number
 * of attributes; that is because we don't have room to store the number of
 * attributes in the header.  Given the MAXALIGN constraint there's no space
 * savings to be had anyway, for usual values of INDEX_MAX_KEYS.
 */

typedef struct IndexTupleData
{
    ItemPointerData t_tid;      /* reference TID to heap tuple */

    /* ---------------
     * t_info is laid out in the following fashion:
     *
     * 15th (high) bit: has nulls
     * 14th bit: has var-width attributes
     * 13th bit: AM-defined meaning
     * 12-0 bit: size of tuple
     * ---------------
     */

    unsigned short t_info;      /* various info about tuple */

} IndexTupleData;               /* MORE DATA FOLLOWS AT END OF STRUCT */
[...]

/*
 * t_info manipulation macros
 */
#define INDEX_SIZE_MASK 0x1FFF
#define INDEX_AM_RESERVED_BIT 0x2000    /* reserved for index-AM specific
                                         * usage */
#define INDEX_VAR_MASK  0x4000
#define INDEX_NULL_MASK 0x8000

The limit you are hitting is INDEX_SIZE_MASK, and to increase it, you'd have to change the tuple header so that t_info has more than two bytes.

Perhaps it is as simple as that, but it might have repercussions in other parts of the code.

  • I am using Postgres v10.5. 13th bit of t_info is unused in this version. I just changed INDEX_SIZE_MASK to 0x3FFF so 13th bit can be used for the size and it worked. Thank you! – Alibi Yeslambek Aug 6 at 7:48
  • Fine, but that way you are barring your way to a PostgreSQL upgrade. Are you sure? – Laurenz Albe Aug 6 at 8:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.