Skip to main content

Questions tagged [change-data-capture]

Questions under this tag are expected to relate to Change Data Capture technologies available in many RDBMS server products. Change data capture is designed to capture insert, update, and delete activity applied to Server tables, and to make the details of the changes available in an easily consumed relational format.

Filter by
Sorted by
Tagged with
0 votes
0 answers
43 views

SQL Server: Querying Change Data Tables directly?

I am messing around with Change Data Capture on SQL Server. Documentation for table containing the change data (cdc.<capture_instance>_CT) says: We recommend that you do not query the system ...
Yano_of_Queenscastle's user avatar
0 votes
1 answer
22 views

Do old inserts get treated as new CDC events during node bootstrap?

When a new Cassandra node bootstraps and receives data from existing tables for which Change Data Capture (CDC) is enabled, do old inserts get treated as new CDC events? I don't see any reference to ...
tdhso's user avatar
  • 5
1 vote
1 answer
72 views

CDC Capture job fails with 'Cannot insert duplicate key row in object cdc.Table_CT' with unique index 'CT Clustered Index'

When the CDC Capture job runs, it fails with this error: Cannot insert duplicate key row in object cdc.Table_CT' with unique index 'CT Clustered Index. This index is a SQL Server CDC created index ...
Pam Moodley's user avatar
6 votes
1 answer
128 views

Running sp_repldone on a secondary node of an Availability Group with @reset = 1 not working

I have a two-node SQL Server 2019 Availability Group with a database that has CDC enabled for several tables. The CDC is used to keep track of transactions that have not been captured for population ...
Hannah Vernon's user avatar
  • 70.2k
0 votes
0 answers
61 views

invalidating slot "a" because its restart_lsn 6093/6BB6F1B0 exceeds max_slot_wal_keep_size

we use google cloud sql (version is PostgreSQL 15.5), the config as follows: config, and max_slot_wal_keep_size = -1, but slot always become lost wal status (restart LSN become null) when running for ...
user avatar
1 vote
0 answers
71 views

Anyone using CDC and having performance issues extracting net_changes? We have 8 tables tracked with CDC "@support_net_changes=1"

So 8 tables are being tracked with CDC "@support_net_changes=1". 7 of them are ok in regards to performance. As some of these tables are only 10-40 million rows and data doesn't change ...
sherveyj's user avatar
  • 111
0 votes
0 answers
29 views

Modify CDC Schema In SQL Server

Can we add a column in CDC captured tables? Basically we are going to process CDC rows and we want to tracked which rows are already processed. So we are thinking about adding a bit field to the CDC ...
Muhammad Faisal's user avatar
0 votes
0 answers
32 views

CLEANUP and CAPTURE jobs not being created when CDC is enabled in an AOAG setup

We are in the process of upgrading to ENTERPRISE SQL SERVER 2022. The DB is configured with AOAG. When we enable CDC at the database/table level, for some reason, the CLEANUP and CAPTURE jobs are not ...
S T's user avatar
  • 1
0 votes
1 answer
31 views

Clarifications on Maintenance jobs & CDC on upgrade to ENTERPRISE EDITION

Raising these on behalf of our DBA team. We are looking at moving to ENTERPRISE SQL SERVER 2022 soon and would need help clarifying a few things: We are looking at a 3-NODE setup: (1) NODE-1: Primary (...
S T's user avatar
  • 1
0 votes
1 answer
42 views

How i can get changes in DB postgresql without triggers

I have database, and in this database ~10 users changes rows, how i can get this changes and user who changed, without triggers. I tried read WAL, and there no info about who what change.
smdsa1337's user avatar
0 votes
1 answer
72 views

CDC cleanup doesn't work even when using sp_cdc_cleanup_change_table which completes successfully

I'm running the query below to find CDC tables with data beyond the configured retention depth: SELECT DB_ID() AS database_id, ct.capture_instance, object_id, source_object_id , rcounts....
Eitan Blumin's user avatar
1 vote
0 answers
160 views

SQL Server - CDC - Log-Scan Process failed to construct a replicated command from log sequence number (LSN)

CDC is enabled on the database and multiple tables are included for cdc. We had been receiving the below alert intermittently, monthly once which is causing the changes not written on the CT tables. ...
Dharkatukey's user avatar
1 vote
0 answers
29 views

On demand snapshot and CDC from Postgresql multitenant DB

I have a Postgresql multitenant DB where each table contains tenant level metadata. e.g. Table "connector" Column | Type |...
deGee's user avatar
  • 111
0 votes
0 answers
13 views

Incremental CDC from Postgresql for select rows

I have a postgresql DB which has tables containing multi-tenant data. Example: Table Customer CustId Cust Name First CustA Second CustB Table CustomerShoppingDet Item Name CustId. Buying Date ...
deGee's user avatar
  • 111
0 votes
1 answer
83 views

How do you correlate SQL Database Audit spec's log column transaction_id with a LSN? from trans log or CDC capture table?

How do you correlate SQL Database Audit log column "transaction_id" with a LSN either from the trans log or a CDC capture table? I am trying to do this because cdc only captures the data ...
Joe Papp's user avatar
1 vote
0 answers
73 views

Cleanup job for CDC failed

The clean up job for CDC failed with this error: Msg 596, Level 21, State 1, Line 0 Cannot continue the execution because the session is in the kill state. Msg 0, Level 20, State 0, Line 0 A severe ...
Meyssam Toluie's user avatar
1 vote
1 answer
382 views

CDC is not working after restoring the databse

I have just restored my database using the below script, USE [master] GO RESTORE DATABASE [MyDB] FROM DISK = N'C:\Program Files\Microsoft SQL Server\MSSQL14.SQLSTANDARD\MSSQL\Backup\mydb.bak' WITH ...
Imran Qadir Baksh - Baloch's user avatar
0 votes
1 answer
464 views

What is the meaning of lsn_time_mapping.start_lsn

I have enable Change Data Capture on a table. I'm trying to understand cdc.lsn_time_mapping. Unfortunately the documentation is vague and missing a column from the results. What's the difference ...
Michael Green's user avatar
0 votes
0 answers
29 views

Using CDC on Read only Secondary Managed instance

We are planning to migrate our on-premises database to Azure Managed instance. Is it possible to use Read Only secondary of the managed instance for CDC.
Anoop Agarwal's user avatar
0 votes
1 answer
61 views

Correlating SQL Audit transaction_id to LSN

I have a relatively simple question that is "Google resistant". I have a SQL Database Audit spec that logs a transaction_id. I also have a Change Data Capture (CDC) spec that logs __$seqval (...
Vocoder's user avatar
  • 117
5 votes
2 answers
339 views

Can I change the owner of the cdc schema?

I've enabled Change Data Capture (CDC) on a database using exec sys.sp_cdc_enable_db This creates a new cdc schema which is owned by the cdc user select s.* , dp.name from sys.schemas as s ...
Zikato's user avatar
  • 5,102
1 vote
2 answers
118 views

Commit logs under cdc_raw folder getting corrupted

I am using cassandra 3.11.12 version. I have recently enabled cdc_raw in cassandra configuration and created new key space and table with WITH cdc=true; and after inserting values ran the alter ...
rahuls_'s user avatar
  • 19
1 vote
1 answer
327 views

CDC Error - NULL Source Schema and Source Table - Error msg on both enable and disable table

Today I re-enabled CDC on the primary node of an availability group. When the jobs were created the cleanup job failed with the following error: Could not delete change table entries made obsolete ...
JoeDBA18's user avatar
1 vote
0 answers
57 views

DR - Missing binlogs in Maxwell

We have a standard db.r6g.2xlarge RDS MySQL instance running in production with 2 replicas. Maxwell CDC is setup to publish binlogs to a Kafka topic. From a RPO point of view: in the worst case (say ...
Amit Dalal's user avatar
1 vote
1 answer
707 views

Oracle query taking a very long time to execute

I am trying to run a couple of reconciliation queries to identify missed deletes from our change data capture tool, but I need it to run relatively fast... My source and target tables are identical ...
Lauren_G's user avatar
2 votes
1 answer
645 views

How can I force a system function to use an index in SQL Server?

Good morning team, I’m really struggling to improve the performance of this function: sys.fn_cdc_map_time_to_lsn. It is a system function which is used by CDC, you can read a full description of what ...
Zoe's user avatar
  • 41
1 vote
1 answer
2k views

SQL Server DB Log keeps growing / CDC enabled database

I've enabled change data capture on a SQL Server database. The database is in full recovery mode. The LOG backups of the database are being done hourly. The capture jobs was running without any issues ...
saso's user avatar
  • 243
2 votes
1 answer
793 views

Merging data from identical tables from multiple databases into one larger table. N-1 table replication

There are N databases with identical table structures in a multitenant, multi-database system. The desire is to replicate a table(s) from these databases into a single larger table in a OLAP database, ...
Ross Bush's user avatar
  • 567
0 votes
1 answer
214 views

Can we keep change-tracking on a replicated table going after changing the distribution database?

We have an MSSQL replicated table which has change-tracking set up on the subscribed table. The change tracking is used to send changes into Snowflake. However - now we need to reconfigure replication ...
Rawheiser's user avatar
  • 133
0 votes
0 answers
67 views

Resuming incremental SSIS CDC sync with big CDC "backlog"

We're using an SSIS package which syncs changes from one (main) database to another (reporting) using CDC. The package runs periodically syncing the data in small increments. After having an issue ...
axk's user avatar
  • 119
-1 votes
1 answer
694 views

sp_rename column and CDC

I have a table in a SQL Server 2019 instance which is being CDC tracked. However when I want to rename a column it failes with the error: Cannot alter column 'Columnn2' because it is 'REPLICATED. I ...
Peter's user avatar
  • 2,372
0 votes
2 answers
164 views

postgres 11.3 checksum validation post migration

I am in the process of migrating our AWS RDS postgresql from one account to another with zero downtime. I am going with pg_dump/pg_restore + AWS CDC for ongoing replication. I am looking for a way to ...
Falcon's user avatar
  • 101
2 votes
0 answers
347 views

Sensor data from Postgres to Redis

I have a Postgres database with a table that is updated with new data every 5 minutes from a weather station. Now, as soon as the table is updated with a new record from the sensor, I would like this ...
user17326436's user avatar
1 vote
1 answer
611 views

CDC - I cannot disable that on the second instance

I have change data capture enabled for most of the tables in one of my databases. Database is hosted on SQL Server instance with version 2016. I have some testing to be done, so I wanted to restore ...
Radek Gąska's user avatar
-1 votes
1 answer
1k views

how to read from cdc tables?

I wanted to try out the change data capture feature of sql server, but I cannot get it to work What I did -- enable cdc EXEC sys.sp_cdc_enable_db -- enable it on a table, for testing EXEC sys....
GuidoG's user avatar
  • 129
0 votes
1 answer
120 views

SQL Server modification timestamps

I'm implementing ingestion job from SQL Server 2016. For performance reasons we want to implement incremental loads. Few facts: Customer has read replica as a part of always on availability group ...
czajek's user avatar
  • 101
1 vote
1 answer
270 views

Create Audit Trail from SQL CDC

I have a few tables on which I have enabled CDC. As each row in CDC table contains all the columns regardless if they were changed or not, so it becomes difficult to figure out only the changed data. ...
user17146433's user avatar
0 votes
1 answer
300 views

Do we need additional license to dump goldengate trail logs to kafka?

We have a Oracle (12c) GoldenGate license only (we do not have an oracle goldengate for big data). We need to transfer the gg trail logs to kafka then to hdfs. Do we need to buy the oracle goldengate ...
jimmiing's user avatar
0 votes
2 answers
910 views

Auditing via change data capture (CDC) for MSSQL

I want to ask the DBA community about best practices for auditing the MSSQL database. I set up CDC MSSQL and experience some difficulties with it. It's working all fine until I change the structure of ...
Valentinas's user avatar
1 vote
1 answer
910 views

insert and update triggers do not fire on replicated table

I have created a publication and a subscription on two servers with wal_level = logical in my postgresql.conf. Replication is functioning normally. My problem is to track changes in the replicated ...
Boris Kalinin's user avatar
0 votes
0 answers
76 views

SQL Server 2014 any way to suppress KB3030352?

I'm upgrading from SQL Server 2008 R2 to SQL Server 2014 (SP3-CU4-GDR) (KB4583462) - 12.0.6433.1. Compatibility mode is set to 120. Trace Flag 9481 is enabled. The problem still is present in lower ...
user17114970's user avatar
0 votes
1 answer
177 views

Is the WAL data text format documented anywhere?

Looking around the Web, I can find plenty of information explaining how to set up WAL replication, and how to get the changes with pg_logical_slot_get_changes or pg_logical_slot_peek_changes. This ...
Mason Wheeler's user avatar
0 votes
0 answers
352 views

MSSQL CDC for views and stored procedures

is it possible to enable CDC for Views or Stored procedures? BR, Val
Valentinas's user avatar
1 vote
2 answers
741 views

Identify and delete specific data from Usertable_CT captured from a CDC enabled table

Scenario We recently performed a heavy update operation updating two columns on a 2 tb user table. The User table is CDC enabled, which caused Usertable_CT table to bulk up. The User table is a heavy ...
user234005's user avatar
2 votes
1 answer
1k views

SQL Server DML auditing - Triggers vs SQL Server Audit Feature (vs temporal tables vs change data capture)

I am looking to implement an auditing process on an instance I have on SQL Server 2017. However, I am not finding a complete conclusion on which auditing feature would be best to use for a DB with a ...
user1930901's user avatar
0 votes
3 answers
647 views

Replicating contained SQL databases to customer in near real-time

We host a number of customer SQL Server databases that we need to replicate to external customer sites in near real-time. The main constraint we have is that, for security reasons, our databases are ...
Simon Brangwin's user avatar
0 votes
2 answers
42 views

T SQL, Get new customer added or deleted from historical table

I have a Table containing historical data. Every day the same staff list will be added with the new dates (between 1200-12500 staff each day however only a few staff might be newly added todaysDate or ...
Nemo's user avatar
  • 3
1 vote
2 answers
737 views

CDC on Sql Server 2017 with HA cluster stopped working [closed]

I am currently using Sql Server 2017 (14.0.3294.2) running on Windows Server 2016 as an HA cluster, one primary with two secondaries. A few days ago, the primary was rebooted and I think the person ...
boggy's user avatar
  • 427
1 vote
2 answers
2k views

How does AWS DMS on-going replication works internally?

From the documentation it's mentioned that "DMS collects changes to the database logs using the database engine's native API" and replicates to target. But, I didn't see anywhere that, at ...
Pand005's user avatar
  • 151
3 votes
3 answers
20k views

AWS DMS Task getting failed after completion with error as AlwaysOn BACKUP-ed data is not available

I have created DMS task for migrating data from one RDS MS SQL Server instance to another RDS MS SQL Server instance with "Full load + ongoing replication", and data copied to target ...
Pand005's user avatar
  • 151