I'm trying to create a database in DynamoDB that has many points with high dimensions. For the most part, I have designed the database to the regular access patterns. However, for a search endpoint, I quickly realized that my points need to be close by similarity.
Initially I thought it would be ok for me to use an index that represented a vector. For instance if there is a cat and no dog and a frog in picture 1 the vector would be 101
But in order to get a list of pictures with cats I would have to query for 1XX, where X is a wildcard. In this case it seems fine as there are only 3 dimensions so it would be O(1)
but with a high dimension it quickly becomes a limiting factor as we would have to query m number of dimensions O(m). I get that each of these points are on a different partition so each m dimensions can be queried at the same time. However, this solution is still not acceptable given the future usage.
Is there a way to define the distance function when using the sort index in DynamoDB or a more clever way of defining the feature vector in such a way that the points would be sorted by distance?