I'm trying to load data from one database to another. I am using the lookup transform that looks at the target table to see if the record already exists.
I am getting duplicate data when the package succeeds.
Why doesn't the lookup transform refresh the table it's looking at every time it is being run?
Any ideas on how to create the lookup transformation so that it will refresh its connection to the table?
I want to keep the records I already have, and add any new records but the package should not load any duplicates.
How do I make sure not to get duplicated data by making sure the lookup transformation sees the newly added data immediately?