The Duplication tag covers the topic of duplicated data and the strategies involved in removing the duplicates, often referred to as deduplication or simply dedupe.

learn more… | top users | synonyms (1)

1
vote
1answer
289 views

Check if query will cause duplicate row

I have a database with a primary key on the first column (words) I want to trim the trailing and leading ends of my data, but I get an error that this causes duplicate data EX: ' David ' will become '...
2
votes
3answers
4k views

Find duplicate records across one-to-many relationship tables

Let's say I have two tables that have a one-to-many relationship: People and Attributes. I'm trying to find duplicates based on the first_name, last_name, and all attributes must match exactly. ...
1
vote
2answers
64 views

RMAN backup set, nothing else, how can I duplicate?

I have what appears to be an RMAN backup set, it has been multiplexed in 32 files, each containing 2-3 dbf's inside. The backupset refers to an SPFILE file, but it was not included in the set, the ...
4
votes
2answers
680 views

Duplicate database leaves orphan archivelog files

I am cloning the production database to another one, lets say CLONEDB, in a regular basis. I am using rman duplicate database script like: run { duplicate target database to CLONEDB FROM ACTIVE ...
0
votes
2answers
56 views

Duplicate databases error

I have two Oracle 11gR2 databases on separate servers. I want to duplicate these databases. For this, I do these steps : In auxiliary db : create password file with password that is same if ...
1
vote
1answer
3k views

Remove all records with duplicates in db2. (Not just the duplicate records)

How can I remove all the records with duplicates in db2. I have looked at various answers but they only remove the duplicates leaving one record from that set in the table. This is what I found ...
0
votes
1answer
634 views

Remove duplicate terms from column

This the tags column where all terms should be listed only once. Would like to remove all additional words from this column. For example the below contant: Lakshmi Bar & Restaurant, Dining &...
2
votes
1answer
2k views

Using EXCEPTION to ignore duplicates during bulk inserts

In pq 9.3.5, I'm importing records from an external source where duplicates are VERY rare, but they do happen. Given a readings table with a unique compound key on (real_time_device_id, recorded_at), ...
1
vote
1answer
228 views

Error while duplicating Oracle database after failover, using 11.2.0.4

I've recently updated oracle from version 11.2.0.1 to 11.2.0.4 and my redundant system is no longer able to recover from failover. I'm using scripts both on primary and standby node to get back to a ...
0
votes
2answers
330 views

SQL Server De-duplication Trigger

I have a table that contains columns of Country, City, Location, Description. Due to the size of the description field it's not possible to bind the four fields into a composite key to prevent ...
1
vote
2answers
568 views

Delete duplicates with specific columns as identifier

I'm required to delete all duplicate entries of a table. The duplicates are identified by two non-key values num (int) and com (varbinary(512)).. My amateur-ish SQLfoo lead me to this non-functioning ...
2
votes
1answer
77 views

Marking duplicate records

I have a list of things, some of which are duplicates. create table things ( id int primary key, title text not null, author text not null, duplicate_of int null references things(id) ); I'...
2
votes
1answer
107 views

How do I remove duplicate records in a table?

I have a table table1 with 1 millions rows of data. I want to remove all the duplicate records in the table1 I was looking at this link: How do I remove duplicate records in a join table in ...
0
votes
0answers
137 views

Looking for a recipe to merge multiple rows to single row

I have a aggregation table that is written by multiple processors (around 20). Each row has a composite key - location-id, campaign-id,... and aggregate stats about it. I need to merge the ...
0
votes
1answer
86 views

Does PostgreSQL have a variable character storage layer that optimizes storage space by automatic dedupe?

My application has a Product and a Purchased table. The Purchased table includes a full copy of the Product row at time of purchase, so that, even if a product name or description was changed, the ...
1
vote
2answers
86 views

Preventing duplicate data when usual normalization only makes sense for some of the records

Background I have two tables, one called issues and the other called status_checks, which both contain (among others) the field comment (VARCHAR). An issue is raised by a user when something needs ...
6
votes
2answers
1k views

Partial db dump/restore

A part of my database (PostgreSQL 9.3) relies on extra tables (ex. County, City, Town, ... ). I don't manage these tables, they are updated regularly by a third party. Each time I get a new full dump, ...
1
vote
2answers
992 views

How do I find duplicate values in multiple columns over different rows?

I have a database with a table of games (some columns omitted): gameid hometeam awayteam `date` Sometimes games are added with the same home and away teams, but on a different date that is just 1 ...
0
votes
1answer
3k views

Oracle Duplicate Database with no archived logs

I am trying to perform a duplicate operation from one server to another (testing disaster recovery) on oracle 11g. I am very much still learning about Oracle. The system is not in production, it is ...
0
votes
1answer
59 views

Working out Duplicated Data in a Customer Data Base

I have a database in SQL2008 of customers, There is an ID number for the customer. What im trying to do is to create a NEWID field. We have duplicated customers that have different ID fields, I am ...
0
votes
1answer
202 views

duplicate entry issue with large mysql database table

I have a very large my sql database table. The other day, for no apparent reason, I now keep getting :- Duplicate entry '148245' for key 'PRIMARY' If I remove the entry at with primary of 148245, ...
3
votes
1answer
141 views

Consolidating 2 Tables with same Values

I am working on a project trying to consolidate two databases, DB1 and DB2, for my organization into a single database, DB3. Initially our organization started with DB1. A few years back they made a ...
10
votes
4answers
18k views

Checking whether two tables have identical content in PostgreSQL

This has already been asked on Stack Overflow, but only for MySQL. I'm using PostgreSQL. Unfortunately (and surprisingly) PostgreSQL does not seem to have something like CHECKSUM table. A PostgreSQL ...
0
votes
1answer
54 views

Move Commonly-Repeated Fields into a new Table

I'm still at the design stage for a new system, and I've noticed there are several places where different types of object share similar fields. In this case, we have customers, offices, warehouses ...
5
votes
1answer
609 views

Percona MySQL 5.5 Unique key is duplicated

I'm totally out from ideas, so maybe somebody else could answer my question. We have a MySQL 5.5 server from Percona with high traffic. The application is in PHP and writes always to master. We have ...
0
votes
2answers
1k views

Msg 2627 Violation of PRIMARY KEY constraint

trying to make a database for teams in a tv show here. but when I try and insert data into tblShowteam the following error made its appearance. Msg 2627, Level 14, State 1, Line 2 Violation of ...
1
vote
1answer
2k views

Master-Master Replication Duplicate Key

I have set two servers master-master replication mysql , i manage to set successfully but the problem is the duplicate key i got MASTER 1 Configuration /etc/my.cnf [mysqld] server-id=1 bind-address ...
0
votes
0answers
77 views

Finding duplicate data

I have noticed dup’s generated in my mail system. some service was causing duplicate calls to my insert proc and generated multiple duplicate messages with about a 4 mili-second difference in a date ...
4
votes
1answer
607 views

MySQL - GROUP_CONCAT returns duplicate data, can't use DISTINCT

I have a normalized database and I'm trying to return data from multiple tables using JOINs and GROUP_CONCAT. Problem: Rows are being duplicated with GROUP_CONCAT. I can't use DISTINCT because some ...
-1
votes
1answer
305 views

Why Do I Still Get a Duplicate Entry Error When There is No Duplicate Entry?

I am trying to insert this data into my database: https://docs.google.com/spreadsheet/ccc?key=0AjQflI82acM3dFI1Mk04dWlYSFJtM0hsZ0g4ZlExMWc&usp=sharing I receive the following error: 1062 - ...
0
votes
2answers
369 views

Make Oracle backup files available for duplicate

I have a Red Hat Enterprise Linux server with Oracle 10.2.0.5 and a database on it (SID is TNATEST). I have a remote CentOS server with Oracle 10.2.0.5 on it that I want to duplicate TNATEST to. The ...
1
vote
1answer
893 views

How do I validate all my constraints in postgresql?

It seems like I have been bitten by the first bug described in the release notes of postgresql 9.3.4: http://www.postgresql.org/docs/devel/static/release-9-3-4.html I now have e.g. duplicate primary ...
4
votes
2answers
1k views

How do I delete unique rows from a table with duplicate keys?

I have the following data: road | ID | other column | other column | ------------------+------+---------------+--------------+ MAIN ST | 1 | ... | ...
0
votes
1answer
60 views

What is meant by the RMAN Duplicate “noncurrent point in time”?

When doing a RMAN DUPLICATE from backup connected to the target, how current does the duplication make the database when no until time is set? The closest thing I can find in the documentation is the ...
2
votes
2answers
4k views

Insert into … (…) (Select…) on duplicate key update

Can you help me to solve a problem. The task is to insert a record in MySQL database. terms: If the record exists - to update the record. If the record does not exist - to create a new one. ...
2
votes
2answers
5k views

Master - Master replication Duplicate entry

I've got a Master - Master replication setup and it worked fine for a couple of days but I keep getting: Duplicate entry errors after a few days... My setup is as follow: One server in The ...
0
votes
1answer
10k views

INSERT / UPDATE to merge data, when empty data is an empty string

Consider a table of some family members: mysql> SELECT * FROM some_table; +--------+--------+--------+ | f_name | l_name | work | +--------+--------+--------+ | Meirav | Cohen | school | | ...
9
votes
3answers
66k views

select rows where column contains same data in more than one record

I have a table that has a column called article_title. Let's say the table name is articles. I need to find out the records where the article_title data is the same on more than one record. Here's ...
1
vote
3answers
196 views

Duplicates while doing multiple joins

I have 4 MySQL tables 1. companies - {id*, company} 2. employees - {id*, fname, lname, id_company*} 3. technologies - {id*, technology} 4. company_technologies - {id_technology*, id_company*} Field ...
3
votes
1answer
119 views

Storing statistic results

I'm doing a small database for statistics, it's a really simple statistic like calculating the cost of an event and divide it by the amount of people that went to get the cost per attendee, but it can ...
3
votes
2answers
225 views

How do I deal with items that I need to repeat monthly?

I have a table where users will enter transactions on a weekly basis, these transactions will either be single items or will be recurring i.e. rent would be recurring and should be the same for every ...
1
vote
1answer
143 views

Should I duplicate the data or have linking table?

We have the following data: Collections of items Individual items These will go into the following tables: ItemCollectionTable (Id, Name, OtherFields (desc, date, etc..)) ItemTable (Id, ...
1
vote
1answer
3k views

Duplicate database for standby returns error ORA-19625: error identifying file /home/oracle/stdbyctrl.ctl

I'm working on creating duplication scripts for standby purposes. and it's constantly failing even though the backups are located in the exact same folder. DB structure is identical and spfile is ...
1
vote
2answers
3k views

Distinct rows from a join

I have two MySQL tables: ╔═══════════════════╗ ║ wpss_question_set ║ ╠═══════════════════╣ ║ id ║ ║ quesset ║ ╚═══════════════════╝ ╔════════════════╗ ║ wpss_questions ║ ╠═══...
0
votes
0answers
29 views

Merge rows with common field value in MySQL [duplicate]

I have a table which was imported from MS-SQL, it has about 1.9 million rows. After importing I realized that a lot of the rows are duplicates and there are no primary key constraints. The rules to ...
1
vote
0answers
672 views

Duplicate GROUP_CONCAT values

Writing a plugin for Wordpress using a custom post type and an additional alias table to hold repetition of the post events. The problem comes with duplicate post meta values. wp_post_meta table: ...
0
votes
0answers
110 views

(My)SQL: management and duplication of multiple sub-databases [closed]

I have a pretty simple question, but it seems that there are controversies as to how to solve it in the best possible way. Say I have a web app that makes online surveys. My challenge is that each of ...
0
votes
1answer
109 views

How to group by when querying duplicate records?

I have a table with duplicates and I want to present it to the stakeholders so they can help me determine the source of duplication. Currently I can print 1 row for each duplicate entry, but I want ...
0
votes
0answers
137 views

Synchronization between master and backup server

I am searching for an option to synchronize data between production server and backup server automatically. Does somebody know a method, how I can do this: if I delete, insert or update a record it ...
1
vote
1answer
167 views

How can I achieve a unique constraint with two fields? [duplicate]

I have a table with e.g. Name and IsDeleted fields. I want to add a row constraint so that only one Name value can have IsDeleted as 'false'. There can be many duplicate Name values, but they must all ...