Questions tagged [amazon-dms]
AWS Database Migration Service (AWS DMS)
31
questions
1
vote
0
answers
42
views
How to troubleshoot logical replication on Postgresql, when it comes to high replication lag?
I have a use case in which we use AWS DMS to stream data to our data lakes (S3), from time to time the replication lag gets out of control, leading to over 500Gb in replication slot lag.
It was also ...
- 11
0
votes
0
answers
47
views
How does AWS Data Migration Service actually work with MSSQL?
I'm very confused as to how AWS Data Migration Service is actually achieving it's replication from an on prem MS SQL 2019 instance to RDS 2019.
I understand from the documentation that tables with a ...
- 31
0
votes
1
answer
38
views
aurora mysql - migrating large tables
We have write intensive application using aurora mysql 3.02 with 2xlarge instance size in prod. I want to migrate 10 tables (biggest one being 450GB) from 2nd aurora mysql into this prod db.
Tried ...
- 11
0
votes
0
answers
53
views
MySQL time field type conversion issue with AWS DMS
I have a DB MySQL v5.6 community edition as a source DB and AWS RDS MySQL v5.7 as target in AWS DMS and have an issue with time field type. DMS task is full load and replication. During full load all ...
- 1
0
votes
0
answers
110
views
Data validation for RDS PostgreSQL migration
I'm planning to replace a bunch of RDS PostgreSQL databases using this method defined by AWS.
The DMS task will run will be CDC-only. I came up with data validation requirements for the migration and ...
- 129
1
vote
1
answer
390
views
Time zone conversion in DMS from DB2 to Aurora Postgres
Challenge
We are using AWS DMS to convert data from IBM DB2 into AWS Auroa Postgres
IBM DB2 server is set to Eastern time (ET) zone. Timestamp related data inside DB2 is therefore in ET.
Findings
AWS ...
3
votes
1
answer
1k
views
Getting Error 1236 (Could not find first log file name in binary log index file) reading binlog in AWS DMS
I have created two DMS task (full load + CDC) with RDS MYSQL 'Read' replica as source endpoint and REDSHIFT as target endpoint. For both the task source and target endpoint is same. One task with only ...
- 83
1
vote
0
answers
151
views
aws documentation for DMS CDC is confusing
I am following this documentation https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html#CHAP_Source.PostgreSQL.Security to enable CDC using logical replication.
I have pglogical ...
- 101
0
votes
0
answers
454
views
Will SQLNET.EXPIRE_TIME=5 prevent AWS Gateway Load Balancer from timing out after 350 seconds?
We have Amazon Web Services (AWS) Database Migration Service (DMS) tasks erroring out against an on premises Oracle 11.2.0.3 database with ORA-03135 errors. Our network traffic between AWS and the ...
- 221
1
vote
1
answer
2k
views
What does MISSING_TARGET means on Amazon DMS (Database Migration Service) validation?
I'm running Amazon DMS from RDS to Aurora, the error when full restarting (never been successful before) from awsdms_validation_failures_v1 tables are:
[
{
"TASK_NAME": "...
- 1,373
0
votes
0
answers
1k
views
AWS DMS Task failed with error: Error executing source loop; Stream component failed at subtask 0
I want to migrate my PostgresDB hosted in Citus cloud service to AWS RDS Aurora Postgres. I am using AWS DMS service. Have created task but getting following errors:
Last failure message Last Error ...
- 101
0
votes
1
answer
778
views
How to set PostgreSQL's permission when migrate other database to it on AWS?
We want to migrate an Oracle DB to a PostgreSQL DB in RDS with DMS.
There is an official guide for preparision in the PostgreSQL's DB:
https://docs.aws.amazon.com/dms/latest/sbs/chap-oracle2postgresql....
- 177
0
votes
0
answers
393
views
Continous data replication between two databases(same RDS instance) in AWS RDS
We have multiple databases in a single RDS instance. Forex A, B, C
A and B DB contains the plans table. But some of the fields will be missing under the plans table in B DB.
What is the smart way to ...
- 101
0
votes
0
answers
1k
views
AWS DMS migration not migrated Foreign keys, sequences and indexes got renamed
I have used DMS for AURORA PostgreSQL to PostgreSQL SCHEMA Migration. Migrate Data only type is used for DMS activity.
After a Full load of Data, all tables were created on the target endpoint with ...
1
vote
2
answers
2k
views
How does AWS DMS on-going replication works internally?
From the documentation it's mentioned that "DMS collects changes to the database logs using the database engine's native API" and replicates to target. But, I didn't see anywhere that, at ...
- 151
1
vote
0
answers
280
views
How to modify the block size of my S3 parquet files that are being queried in Athena?
I read that adjusting the block size of the parquet files being queried with Athena can affect and possibly improve the performance of the queries. My parquet files for my database are currently ...
- 38.6k
0
votes
1
answer
58
views
Migrate Oracle 12c to 18c in AWS RDS
Currently we have an Amazon RDS Oracle 12c instance and we need to upgrade it to Oracle 18c in a new RDS instance.
What is the correct approach to do this?
Can we set up an 18c instance staright ...
- 147
0
votes
0
answers
1k
views
AWS DMS Log growth problem - SQL Server as Source
I have a problem using the AWS Database Migration Service for implementing a transactional replication from SQL Server as a source database engine, a help is highly appreciated.
The 'safeguardPolicy' ...
- 31
3
votes
0
answers
2k
views
Postgresql v10.1 - How to fix ERROR: duplicate key value violates unique constraint
I'm not sure how this happened to this table but have an issue with ERROR: duplicate key value violates unique constraint trying to use AWS's DMS service to move a postgres v10.1 DB to RDS v10
The ...
- 175
0
votes
1
answer
1k
views
AWS DMS: 2 million CDCIncomingChanges resolved without target throughput
We're attempting to use AWS DMS to replicate MS-SQL to a customer's on-prem DB, and running into a confusing situation. At 13:00 UTC there are over 2 million CDCIncomingChanges, with CDCTargetLatency ...
- 111
3
votes
1
answer
5k
views
Do not replicate Delete operations AWS DMS
I am using AWS DMS to replicate ongoing changes with SQL Server as a source and target endpoint. The tasks are running and replicating data with low latency. However, I need the tasks configured not ...
- 31
0
votes
1
answer
63
views
Redshift is missing in target database list of AWS SCT Schema Convertion Tool
I want to convert schema of SQL Server database to Amazon Redshift using AWS SCT tool
But as target database, Redshift is not listed
When I use PostgreSQL compatible with IP address of the cluster, ...
- 182
1
vote
1
answer
9k
views
AWS DMS Fails with "Command failed to load data with exit error code 1"
I am trying to Migrate my Postgresql RDS to Aurora Postgresql using DMS.
All my tasks are failing, and I am unable to understand what can be done to help the tasks progress.
The error from cloud ...
- 201
2
votes
1
answer
676
views
AWS DMS - How to select different databases for source and target points for a task
Does aws DMS(Data Migration service) allow to select different databases for source and target points for a task in a replication instance? My source engine is MySQL.
example: I want to move table A ...
- 49
2
votes
1
answer
663
views
Replication from Amazon's RDS to Postgres via DMS errors out
I have a transactional replication from SQL Server 2008 to RDS SQL Server 2016 in 2014 compatibility mode. This works okay. I now want to replicate this RDS database via DMS to Postgresql. This errors ...
- 66
3
votes
2
answers
6k
views
Failure with ongoing replication from SQL Server RDS using AWS DMS
I'm trying to replicate from an SQL Server RDS database using DMS, and running into problems.
I am able to load all data from one table into the destination (~1M rows) as a single 'Migrate existing ...
- 151
3
votes
1
answer
7k
views
AWS DMS endpoint test connection error
I am trying to migrate Microsoft SQL Server databases from on-premise to an RDS instance. I have created a SQL Server 2014 Express Edition instance in RDS and I have SQL Server 2014 Express Edition on ...
- 31
5
votes
1
answer
7k
views
Spike in AWS RDS Oldest Replication Slot lag
Does anyone know about this "Oldest Replication Slot Lag" in AWS RDS? What does it represent?
I have a PostgreSQL instance on RDS (with Logical Replication switched on) for transferring data from ...
- 309
4
votes
1
answer
2k
views
Create event trigger as non-superuser in Postgres
Is it possible to give a role or a set of attributes to a non-superuser in postgres that would allow the user to create an EVENT TRIGGER? Simple example of what (obviously) doesn't work in psql:
=&...
- 141
1
vote
1
answer
427
views
Why did the data type of my DB change?
I had PostgreSQL installation in an EC2 instance. I wanted to migrate to RDS and thus used the DMS service. However, I later came to realize that the data type of the columns have been changed. The ...
- 86
7
votes
2
answers
8k
views
AWS DMS creates tables with no auto_increment
I am migrating my databases to Amazon's RDS using their DMS tool. Pretty much everything looks great except that columns that were marked as auto_increment on the source database did not get marked ...
- 103