I'm building an application that will fetch messages with JSON every once in a while and store these in a postgres db. Now I want to prevent duplicates, so I calculate the sha-512 of the message and set it as the primary key. Since I want to prevent duplicates I can do two things:
- Fetch all hashes from the database and see when inserting new messages if the hash already exists (checking in code)
- Use an insert ignore (in postgres an update probably, since postgres doesn't support insert ignore) and let the database handle these exceptions
Considering I will probably be inserting a few messages at a time, I was wondering if method two will look up the primary key and see if exists for every insert as opposed to just storing a list in memory with method 1. What would be the most efficient?