From the documentation it's mentioned that "DMS collects changes to the database logs using the database engine's native API" and replicates to target. But, I didn't see anywhere that, at what rate it replicates from source to target in terms of number of records and is there any way we can control this settings or can I know each time how many rows it replicates?
Add a comment
|
2 Answers
The reason you didn't find any specific information in AWS's documentation on the replication rate is because it mostly depends on the source database. For example, in Microsoft SQL Server, there's a feature called Change Data Capture (CDC) which logs data changes in real-time. This means DMS will gather all logs from the CDC tables in the source Microsoft SQL Server database since the last time DMS ran.
Other database systems might log changes at slower rates, so there isn't once answer from AWS's perspective. Additionally, I believe you can schedule DMS Tasks to run at their own intervals as well.
But in general, the goal of AWS's DMS Tool is to replicate all changes since it last synced.
-
Thanks J.D., I want to run my DMS task continuously. In my case source is RDS SQL Server and the target is also another RDS SQL Server. Here, I am looking for capturing event as soon as record got inserted into target table based on log or something else with which I can trigger SNS/Lambda (not at task level). Is there any option we can do this?– Pand005Commented Jan 6, 2021 at 3:03
-
1You'll want to enable Change Data Capture in your source database then (follow the link in my answer for more info). That will log changes in real-time. On the AWS side I'm not sure (and unfortunately from my experience of trying to work with it, a lot of people are apparently unsure) the best way to create a real-time process to read from those CDC logs. I think it's possible with DMS. Your best course of action is to talk to an AWS Solutions Architect for advisement on the best course of action with their services. You can request a meeting through your account rep.– J.D.Commented Jan 6, 2021 at 12:24
When I used DMS, it was work with SQL Server Replication. But the general rule is the same. DMS does not read your CDC tables, it reads your data which is changed from the log file or log backup. But, it wants to take advantage of CDC or Transaction Replication for it does.
More info Part - 1
For Microsoft SQL Server, AWS DMS uses MS-Replication or MS-CDC to write information to the SQL Server transaction log. It then uses the fn_dblog() or fn_dump_dblog() function in SQL Server to read the changes in the transaction log based on the log sequence number (LSN).
More info Part - 2
During CDC, AWS DMS needs to look up SQL Server transaction log backups to read changes.
There are a few options but only work with full data load.
Also, you can track data changes with Task Monitoring - CDCIncomingChanges
CDCIncomingChanges The total number of change events at a point-in-time that are waiting to be applied to the target. Note that this is not the same as a measure of the transaction change rate of the source endpoint. A large number for this metric usually indicates AWS DMS is unable to apply captured changes in a timely manner, thus causing high target latency.
answered Jan 6, 2021 at 12:44
-
Thanks Yunus, does transaction logs contains info about how many rows for table or the row info it is trying to replicate to target? This is where I am trying to fire an aws event as soon as record gets inserted/replicated into target table.– Pand005Commented Jan 6, 2021 at 15:52
-
You can use those tables for tracking on SQL Server docs.aws.amazon.com/dms/latest/userguide/… Commented Jan 6, 2021 at 16:05
-
Thanks again Yunus, its true with dmslogs.awsdms_history we can see total number of records count for all the tables it is replicated to target. Can we fire aws SNS/Lambda on each replication?– Pand005Commented Jan 6, 2021 at 16:50
-