Questions tagged [change-data-capture]

Questions under this tag are expected to relate to Change Data Capture technologies available in many RDBMS server products. Change data capture is designed to capture insert, update, and delete activity applied to Server tables, and to make the details of the changes available in an easily consumed relational format.

Filter by
Sorted by
Tagged with
0 votes
1 answer
21 views

Clarifications on Maintenance jobs & CDC on upgrade to ENTERPRISE EDITION

Raising these on behalf of our DBA team. We are looking at moving to ENTERPRISE SQL SERVER 2022 soon and would need help clarifying a few things: We are looking at a 3-NODE setup: (1) NODE-1: Primary (...
S T's user avatar
  • 1
0 votes
1 answer
33 views

How i can get changes in DB postgresql without triggers

I have database, and in this database ~10 users changes rows, how i can get this changes and user who changed, without triggers. I tried read WAL, and there no info about who what change.
smdsa1337's user avatar
0 votes
1 answer
30 views

CDC cleanup doesn't work even when using sp_cdc_cleanup_change_table which completes successfully

I'm running the query below to find CDC tables with data beyond the configured retention depth: SELECT DB_ID() AS database_id, ct.capture_instance, object_id, source_object_id , rcounts....
Eitan Blumin's user avatar
1 vote
0 answers
62 views

SQL Server - CDC - Log-Scan Process failed to construct a replicated command from log sequence number (LSN)

CDC is enabled on the database and multiple tables are included for cdc. We had been receiving the below alert intermittently, monthly once which is causing the changes not written on the CT tables. ...
Dharkatukey's user avatar
1 vote
0 answers
23 views

On demand snapshot and CDC from Postgresql multitenant DB

I have a Postgresql multitenant DB where each table contains tenant level metadata. e.g. Table "connector" Column | Type |...
deGee's user avatar
  • 111
0 votes
0 answers
9 views

Incremental CDC from Postgresql for select rows

I have a postgresql DB which has tables containing multi-tenant data. Example: Table Customer CustId Cust Name First CustA Second CustB Table CustomerShoppingDet Item Name CustId. Buying Date ...
deGee's user avatar
  • 111
0 votes
1 answer
48 views

How do you correlate SQL Database Audit spec's log column transaction_id with a LSN? from trans log or CDC capture table?

How do you correlate SQL Database Audit log column "transaction_id" with a LSN either from the trans log or a CDC capture table? I am trying to do this because cdc only captures the data ...
Joe Papp's user avatar
1 vote
0 answers
42 views

Cleanup job for CDC failed

The clean up job for CDC failed with this error: Msg 596, Level 21, State 1, Line 0 Cannot continue the execution because the session is in the kill state. Msg 0, Level 20, State 0, Line 0 A severe ...
Meyssam Toluie's user avatar
1 vote
1 answer
144 views

CDC is not working after restoring the databse

I have just restored my database using the below script, USE [master] GO RESTORE DATABASE [MyDB] FROM DISK = N'C:\Program Files\Microsoft SQL Server\MSSQL14.SQLSTANDARD\MSSQL\Backup\mydb.bak' WITH ...
Imran Qadir Baksh - Baloch's user avatar
0 votes
1 answer
182 views

What is the meaning of lsn_time_mapping.start_lsn

I have enable Change Data Capture on a table. I'm trying to understand cdc.lsn_time_mapping. Unfortunately the documentation is vague and missing a column from the results. What's the difference ...
Michael Green's user avatar
0 votes
0 answers
22 views

Using CDC on Read only Secondary Managed instance

We are planning to migrate our on-premises database to Azure Managed instance. Is it possible to use Read Only secondary of the managed instance for CDC.
Anoop Agarwal's user avatar
0 votes
1 answer
37 views

Correlating SQL Audit transaction_id to LSN

I have a relatively simple question that is "Google resistant". I have a SQL Database Audit spec that logs a transaction_id. I also have a Change Data Capture (CDC) spec that logs __$seqval (...
Vocoder's user avatar
  • 117
5 votes
2 answers
216 views

Can I change the owner of the cdc schema?

I've enabled Change Data Capture (CDC) on a database using exec sys.sp_cdc_enable_db This creates a new cdc schema which is owned by the cdc user select s.* , dp.name from sys.schemas as s ...
Zikato's user avatar
  • 4,418
1 vote
2 answers
99 views

Commit logs under cdc_raw folder getting corrupted

I am using cassandra 3.11.12 version. I have recently enabled cdc_raw in cassandra configuration and created new key space and table with WITH cdc=true; and after inserting values ran the alter ...
rahuls_'s user avatar
  • 19
1 vote
1 answer
168 views

CDC Error - NULL Source Schema and Source Table - Error msg on both enable and disable table

Today I re-enabled CDC on the primary node of an availability group. When the jobs were created the cleanup job failed with the following error: Could not delete change table entries made obsolete ...
JoeDBA18's user avatar
0 votes
0 answers
99 views

SQL Server 2019 sys.sp_cdc_cleanup_change_table procedure not working

Although CDC works and all transactions are logged, we cannot clear the respective CDC table using the builtin procedure sys.sp_cdc_cleanup_change_table which should remove all the transaction except ...
user avatar
0 votes
0 answers
45 views

CDC on postgreSQL to Redshift

I am trying to implement CDC on PostgreSQL to Redshift and looking for some approaches. I had a look at Debezium but it looks like it's not ideal for my requirement. I want to read all delta changes ...
Bab's user avatar
  • 1
1 vote
0 answers
43 views

DR - Missing binlogs in Maxwell

We have a standard db.r6g.2xlarge RDS MySQL instance running in production with 2 replicas. Maxwell CDC is setup to publish binlogs to a Kafka topic. From a RPO point of view: in the worst case (say ...
Amit Dalal's user avatar
1 vote
1 answer
282 views

Oracle query taking a very long time to execute

I am trying to run a couple of reconciliation queries to identify missed deletes from our change data capture tool, but I need it to run relatively fast... My source and target tables are identical ...
Lauren_G's user avatar
2 votes
1 answer
449 views

How can I force a system function to use an index in SQL Server?

Good morning team, I’m really struggling to improve the performance of this function: sys.fn_cdc_map_time_to_lsn. It is a system function which is used by CDC, you can read a full description of what ...
Zoe's user avatar
  • 41
1 vote
1 answer
1k views

SQL Server DB Log keeps growing / CDC enabled database

I've enabled change data capture on a SQL Server database. The database is in full recovery mode. The LOG backups of the database are being done hourly. The capture jobs was running without any issues ...
saso's user avatar
  • 243
2 votes
1 answer
550 views

Merging data from identical tables from multiple databases into one larger table. N-1 table replication

There are N databases with identical table structures in a multitenant, multi-database system. The desire is to replicate a table(s) from these databases into a single larger table in a OLAP database, ...
Ross Bush's user avatar
  • 505
0 votes
1 answer
154 views

Can we keep change-tracking on a replicated table going after changing the distribution database?

We have an MSSQL replicated table which has change-tracking set up on the subscribed table. The change tracking is used to send changes into Snowflake. However - now we need to reconfigure replication ...
Rawheiser's user avatar
  • 133
0 votes
0 answers
59 views

Resuming incremental SSIS CDC sync with big CDC "backlog"

We're using an SSIS package which syncs changes from one (main) database to another (reporting) using CDC. The package runs periodically syncing the data in small increments. After having an issue ...
axk's user avatar
  • 119
-1 votes
1 answer
549 views

sp_rename column and CDC

I have a table in a SQL Server 2019 instance which is being CDC tracked. However when I want to rename a column it failes with the error: Cannot alter column 'Columnn2' because it is 'REPLICATED. I ...
Peter's user avatar
  • 2,227
0 votes
2 answers
146 views

postgres 11.3 checksum validation post migration

I am in the process of migrating our AWS RDS postgresql from one account to another with zero downtime. I am going with pg_dump/pg_restore + AWS CDC for ongoing replication. I am looking for a way to ...
Falcon's user avatar
  • 91
2 votes
0 answers
305 views

Sensor data from Postgres to Redis

I have a Postgres database with a table that is updated with new data every 5 minutes from a weather station. Now, as soon as the table is updated with a new record from the sensor, I would like this ...
user17326436's user avatar
1 vote
1 answer
448 views

CDC - I cannot disable that on the second instance

I have change data capture enabled for most of the tables in one of my databases. Database is hosted on SQL Server instance with version 2016. I have some testing to be done, so I wanted to restore ...
Radek Gąska's user avatar
-1 votes
1 answer
492 views

how to read from cdc tables?

I wanted to try out the change data capture feature of sql server, but I cannot get it to work What I did -- enable cdc EXEC sys.sp_cdc_enable_db -- enable it on a table, for testing EXEC sys....
GuidoG's user avatar
  • 129
0 votes
1 answer
107 views

SQL Server modification timestamps

I'm implementing ingestion job from SQL Server 2016. For performance reasons we want to implement incremental loads. Few facts: Customer has read replica as a part of always on availability group ...
czajek's user avatar
  • 101
1 vote
1 answer
236 views

Create Audit Trail from SQL CDC

I have a few tables on which I have enabled CDC. As each row in CDC table contains all the columns regardless if they were changed or not, so it becomes difficult to figure out only the changed data. ...
user17146433's user avatar
0 votes
1 answer
283 views

Do we need additional license to dump goldengate trail logs to kafka?

We have a Oracle (12c) GoldenGate license only (we do not have an oracle goldengate for big data). We need to transfer the gg trail logs to kafka then to hdfs. Do we need to buy the oracle goldengate ...
jimmiing's user avatar
0 votes
2 answers
805 views

Auditing via change data capture (CDC) for MSSQL

I want to ask the DBA community about best practices for auditing the MSSQL database. I set up CDC MSSQL and experience some difficulties with it. It's working all fine until I change the structure of ...
Valentinas's user avatar
0 votes
1 answer
717 views

insert and update triggers do not fire on replicated table

I have created a publication and a subscription on two servers with wal_level = logical in my postgresql.conf. Replication is functioning normally. My problem is to track changes in the replicated ...
Boris Kalinin's user avatar
0 votes
0 answers
74 views

SQL Server 2014 any way to suppress KB3030352?

I'm upgrading from SQL Server 2008 R2 to SQL Server 2014 (SP3-CU4-GDR) (KB4583462) - 12.0.6433.1. Compatibility mode is set to 120. Trace Flag 9481 is enabled. The problem still is present in lower ...
user17114970's user avatar
0 votes
1 answer
164 views

Is the WAL data text format documented anywhere?

Looking around the Web, I can find plenty of information explaining how to set up WAL replication, and how to get the changes with pg_logical_slot_get_changes or pg_logical_slot_peek_changes. This ...
Mason Wheeler's user avatar
0 votes
0 answers
276 views

MSSQL CDC for views and stored procedures

is it possible to enable CDC for Views or Stored procedures? BR, Val
Valentinas's user avatar
1 vote
2 answers
673 views

Identify and delete specific data from Usertable_CT captured from a CDC enabled table

Scenario We recently performed a heavy update operation updating two columns on a 2 tb user table. The User table is CDC enabled, which caused Usertable_CT table to bulk up. The User table is a heavy ...
user234005's user avatar
2 votes
1 answer
893 views

SQL Server DML auditing - Triggers vs SQL Server Audit Feature (vs temporal tables vs change data capture)

I am looking to implement an auditing process on an instance I have on SQL Server 2017. However, I am not finding a complete conclusion on which auditing feature would be best to use for a DB with a ...
user1930901's user avatar
0 votes
3 answers
592 views

Replicating contained SQL databases to customer in near real-time

We host a number of customer SQL Server databases that we need to replicate to external customer sites in near real-time. The main constraint we have is that, for security reasons, our databases are ...
Simon Brangwin's user avatar
0 votes
2 answers
42 views

T SQL, Get new customer added or deleted from historical table

I have a Table containing historical data. Every day the same staff list will be added with the new dates (between 1200-12500 staff each day however only a few staff might be newly added todaysDate or ...
Nemo's user avatar
  • 3
1 vote
2 answers
656 views

CDC on Sql Server 2017 with HA cluster stopped working [closed]

I am currently using Sql Server 2017 (14.0.3294.2) running on Windows Server 2016 as an HA cluster, one primary with two secondaries. A few days ago, the primary was rebooted and I think the person ...
boggy's user avatar
  • 427
1 vote
2 answers
2k views

How does AWS DMS on-going replication works internally?

From the documentation it's mentioned that "DMS collects changes to the database logs using the database engine's native API" and replicates to target. But, I didn't see anywhere that, at ...
Pand005's user avatar
  • 151
3 votes
3 answers
18k views

AWS DMS Task getting failed after completion with error as AlwaysOn BACKUP-ed data is not available

I have created DMS task for migrating data from one RDS MS SQL Server instance to another RDS MS SQL Server instance with "Full load + ongoing replication", and data copied to target ...
Pand005's user avatar
  • 151
2 votes
2 answers
896 views

identifier 'DBMS_CDC_PUBLISH.CREATE_CHANGE_SET' must be declared [closed]

I am using oracle 11g express edition and I am getting the below error when trying to create a change set. Error report: ORA-06550: line 2, column 3: PLS-00201: identifier 'DBMS_CDC_PUBLISH....
lse23's user avatar
  • 131
0 votes
2 answers
323 views

Why am I getting error 35353 when enabling CDC on a table with a CCI?

I'm trying to enable Change Data Capture on a table with a clustered columnstore index, a unique non-clustered index, and a non-unique non-clustered index in SQL Server Standard 2019. The command ...
Avyncentia's user avatar
0 votes
0 answers
217 views

Missing records in cdc capture instance - MSSQL Server

We have a Source table (MSSQL Server database) and cdc is enabled on it. Most of the time cdc captures all changes that happened in Source table correctly, but time to time changed records are missing ...
Lana's user avatar
  • 1
3 votes
0 answers
312 views

Is there a reliable way to determine the order in which Postgres transactions were committed without logical decoding?

Is there a reliable way to determine the serial order in which Postgres transactions were committed without logical decoding? From what I understand pg_xact_commit_timestamp can’t necessarily be ...
Jordan's user avatar
  • 141
1 vote
2 answers
1k views

SQL Server 2019 standard edition CDC with SSIS standard edition

The primary question is: Are there any methods for using SQL Server standard edition CDC for data synchronization to cloud based non SQL server databases that don't require use of SSIS Enterprise ...
phinman's user avatar
  • 13
1 vote
1 answer
579 views

Conditionally track CDC on a table in SQL Server 2016

I have CDC working on many tables in SQL Server 2016 without any issues. I now have a requirement to capture data changes (CDC) on a table when a specific criteria is met. Is it possible to capture ...
Harry's user avatar
  • 11