We have a SQL Server legacy database, db1, to put data of Project1. With change of certain tables(insert/update) in db1, we need to transfer the changes in another postgres database, db2 (used by Project2).
Presently the system is adhoc. With any change of table1 in db1, we keep the record (table1, id, actiontype) in a 'Queue' table, upon which a cron job runs every 5 minutes, which transfers data to db2 using the information in db1.Queue.
Problem is, following this way, we can't see real-time reflection in db2, i.e. data from db1 wouldn't immediately propagate to db2. We have to wait for the cron job to run.
Any suggestion to improve the solution highly appreciated.
Is distributed async task queue (e.g. Celery) a good option here?