I have the following scenario:
I have 2 source systems, one running Postgres, the other running MySQL. An ETL process (using Azure DataFactory and Azure Integration runtimes) runs the Extract for a number of Tables in each system, pulling them into an Azure SQL Database (where the Transformation and eventual load happens).
Most of the tables do not have a TimeStamp for last modified (or similar) field.
The current process involves truncating the destination tables and then bulk loading the entire table into the destination DB.
This process takes about 15-20 minutes all up, however we are looking at expanding the tables and columns we import to run some additional reporting, to the point where it might adversely affect the time the process takes.
Is there a quick or elegant way that this process can be changed to only select the new and updated rows for importation?