I'm planning to replace a bunch of RDS PostgreSQL databases using this method defined by AWS.
The DMS task will run will be CDC-only. I came up with data validation requirements for the migration and am open to feedback.
Requirements
- If a table has a PK, validate it using AWS DMS.
- If a table has a column that would be considered LOB, such as JSON, validate the column by hashing specific column on the source and target, then diffing the sums (method A).
- If a table doesn’t have a primary key but has a column that is effectively unique, but not declared as
UNIQUE
in the schema, validate the table using WbDataDiff (method B). The effectively unique column will be specified as the alternate key (--alternateKey
). - If a table doesn’t have a primary key and has no unique columns, validate the table by hashing a set of columns with highly variable data on the source and target, then diffing the sums (method A).
Validation Methods
A. Hash a single column or set of columns with highly variable data and diff.
- This method can validate LOB migration, e.g. JSON columns.
- This method can validate tables that don't have a primary key or any effectively
UNIQUE
column. - SQL below came from this source.
SELECT
-- Prepends an x the extracted first 8 characters of the hash
-- (generates a hash string; the purpose of the x is so that postgres interprets them as hex strings when casting to a number)
-- then converts the string to a 32 bit int
-- Finally, all of the ints are summed
sum(('x' || SUBSTRING(hash, 1, 8)) :: BIT(32) :: BIGINT) as sum1,
-- Does the same as the column above, but with the next 8 characters
sum(('x' || SUBSTRING(hash, 9, 8)) :: BIT(32) :: BIGINT) as sum2,
sum(('x' || SUBSTRING(hash, 17, 8)) :: BIT(32) :: BIGINT) as sum3,
sum(('x' || SUBSTRING(hash, 25, 8)) :: BIT(32) :: BIGINT) as sum4
FROM
(
SELECT
md5 (
-- When the column value is null, use a space
COALESCE(md5('address' :: TEXT), ' ') ||
COALESCE(md5('date'::TEXT), ' ')
) AS hash
FROM
example_table
) AS t;
B. Use SQLWorkbench/J’s WbDataDiff data comparison tool to validate tables that don't have a PK or UNIQUE
key. This method can show exact differences in the data between the source and target.
- The tool accepts an alternative key. The alternative key does not need to be declared as
UNIQUE
in the schema.
WbDataDiff -referenceProfile="source"
-targetProfile="target"
-referenceTables=lerg1
-file=lerg1_diff.sql
-includeDelete=true
-alternateKey='lerg1=ocn_no'
-excludeRealPK=true
-showProgress=true
md5('column'::text)
is wrong, the blog post meantmd5(column::text)