I'm trying to create a database in DynamoDB that has many points with high dimensions. For the most part, I have designed the database to the regular access patterns. However, for a search endpoint, I quickly realized that my points need to be close by similarity.

Initially I thought it would be ok for me to use an index that represented a vector. For instance if there is a cat and no dog and a frog in picture 1 the vector would be 101

But in order to get a list of pictures with cats I would have to query for 1XX, where X is a wildcard. In this case it seems fine as there are only 3 dimensions so it would be O(1)

but with a high dimension it quickly becomes a limiting factor as we would have to query m number of dimensions O(m). I get that each of these points are on a different partition so each m dimensions can be queried at the same time. However, this solution is still not acceptable given the future usage.

Is there a way to define the distance function when using the sort index in DynamoDB or a more clever way of defining the feature vector in such a way that the points would be sorted by distance?

  • 1
    If what you're looking for is something akin to a keyword search, DynamoDB probably isn't the right platform
    – bbaird
    Jan 19 '21 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.