My company is in the process of migrating a customer's data from their third party system into our own system. We currently use a java application that we have developed in house that selects data from the source (third party database), transforms it into our schema, and inserts into the target (our application database). Unfortunately there are multiple issues with this method:
- Inefficient. Our current customer has a huge amount of data and the total time for migrating the data is nearly 6 days of processing time.
- Customer Downtime. The data is only is current up to the time we begin the migration. This means the customer must cease use of their current system during this time, leaving us a limited amount of time to perform the migration, with little room for error.
- Error prone. The customer's data is full of inconsistencies that need handling. We also mostly rewrite this application to fit each customer, as there is little overlap in the systems they use. This requires large amounts of developer and QA time.
This must be a fairly common problem, as companies often move between applications and need to migrate their current data. Some possible solutions I am looking into are:
- Oracle Data Integrator.
- SQL Server Integration Services
Our own web application is designed to use either Oracle or SQL Server, so I am open to using products used with both Oracle and SQL Server databases.
Is there a solution that would allow continuous replication of a customer's source data into our target database while transforming the data into our schema?