The Duplication tag covers the topic of duplicated data and the strategies involved in removing the duplicates, often referred to as deduplication or simply dedupe.

learn more… | top users | synonyms (1)

0
votes
1answer
33 views

Grouping operator in DBMS (γ) is duplicate-impervious. Why?

I'm reading Database Systems by Ullman where its mentioned that γ is duplicate-impervious. I don't understand why a simple group operation should cause elimination of duplicates. Aliter, ...
1
vote
1answer
39 views

Exclude All Matching Rows

I have a table that has a unique ID, name, email, and some other fields. I want to match this table against another table on email, but sometimes emails are shared. The unique ID is not shared among ...
1
vote
1answer
31 views

Fast DELETE with double subquery on big table for dupe remover

I have a table with 9million rows. At some point in time i changed unique key from one column to two columns(for partitioning). After this i created duplicates of my first column which is unwanted, ...
1
vote
2answers
39 views

Foreign keys update automation

I'm on SQL Server 2014 and I need to clean a table from duplicates. Problem is: if a duplicate has references in other tables, that references should be updated to reference the chosen non-duplicate ...
1
vote
1answer
56 views

Finding list of Duplicate records from 2 tables in DB2

I have a Person table and Address Table. Their relationship is Person.AddrId = Address.Id Person table has First_name, Last_name, and AddrId. The Address table has Address1, Zipcode, and Id. I ...
0
votes
1answer
39 views

Flagging duplicates

I have a huge table with a couple of duplicates: I want to add a new column that shows some value when duplicates occur, something like: case when fist.firstname = last.firstname then 'yes' else ...
2
votes
1answer
24 views

De-duplication method for fields that are identical except for Unique ID

I'm looking for a decent de-duplication method. Basically I have a table which has five fields plus a Unique ID (key) field. I've run a duplicate which detection query which finds any records where ...
2
votes
1answer
24 views

Duplicating rows but incrementing the ID each time

I have the strenuous task of merging multiple Magento databases together and I am importing hundreds of blocks into the main database but subsequently have to go through every one and change the store ...
1
vote
2answers
44 views

How can I merge duplicate rows in a single table?

I have a table with duplicate barcodes (barcode) and carts (cart) and (Quantity_counted_IN) and (Quantity_Counted_out). I want to merge all duplicate records that have the same barcode and cart ...
1
vote
1answer
31 views

Show only duplicate values

How to select rows with duplicate values (non-distinct rows) and summarize with a count of the number of duplicate values in order of the most frequent duplicates? Given a table of: person_ name_ ...
3
votes
3answers
72 views

Remove duplicates in sequence based on keys

I am looking to remove duplicates on a row by row comparison based on keys Member and Group and data sorted ASC on Member, Group and StartDate. Just return the records in the select query and not ...
1
vote
2answers
73 views

Updating duplicate records with different data

I have a migration specific issue. Data from a large table has been migrated into another table, but data for one destination column is missing due to certain issues (that are not specific to this ...
0
votes
1answer
101 views

Delete duplicate with same ID PHP/PDO/MYSQL

I'm new to databases and I need your help I'm working on a website and I have few problems. I'm using PDO. There is a user table, with unique ID, name and password. User enters his username and ...
1
vote
0answers
47 views

Finding Duplicate URLs

The following URL varchar(2000) row holds URL values in a 3rd party database that I'm accessing directly, i.e. going around the HTTP API. MySQL [database]> desc TABLE; ...
3
votes
3answers
210 views

Selecting only one duplicate

I have a database of purchase orders and stockroom checkouts, and it seems that the stockroom checkouts tends to get duplicates inserted (we source this data from an older system nightly). What would ...
3
votes
2answers
42 views

How to prevent duplicates while storing result from different application

For my case multiple application uses data from db based on date, this data is date specific so if for a date data exist application should just read it otherwise it should fetch information from ...
0
votes
1answer
35 views

Removing duplicate rows and changing related foreign keys in PostgreSQL

I have found all duplicate rows and now want to delete them after I update related foreign keys in other tables so they point to the correct row. How do I get a list of foreign keys related to my ...
2
votes
2answers
90 views

Merge two huge tables keeping only the unique rows

I have two tables: table1 with 61 million rows and table2 with 59 millions rows. The columns in both are the same (name and type). Both are imported from backup files. I want to merge these two ...
1
vote
1answer
281 views

SQLite3: Remove duplicates

Given the following table CREATE TABLE `sms` ( `_id` INTEGER, `thread_id` INTEGER, `address` TEXT, `person` INTEGER, `date` INTEGER, `protocol` INTEGER, `read` ...
1
vote
2answers
199 views

what is the quickest way to delete duplicate rows?

I need to delete duplicate rows from a large table. what is the best way to achieve that? currently I use this algorithm: declare @t table ([key] int ) insert into @t select 1 insert into @t ...
2
votes
2answers
91 views

Nested table hierarchy: foreign key duplication vs. join

I have a nested table hierarchy, similar to the following one (this is an example but demonstrates the schema well enough): +---------+ +---------+ +---------+ +-------------+ | users | | ...
1
vote
1answer
44 views

PostgreSQL, how to extract all the possible couples of a primary key in a table?

I would like to find out a PostgreSQL command that reads the content of a table, and returns all the possible couplings of the primary key of the table. Something like this: input_table (ID is the ...
1
vote
1answer
52 views

Sum currencies excluding duplicatied records

I have a CRMOffer table containing customerID and other CRMOffer data. And a Sell table containing customerID and sellValue and sellPK. Sell table has a explicit PK, CRMOffer sadly doesn't have one ...
3
votes
2answers
3k views

INSERT… ON DUPLICATE KEY UPDATE not working as I expect

I have a table called "Example" CREATE TABLE IF NOT EXISTS `example` ( `id` int(11) NOT NULL AUTO_INCREMENT, `a` int(11) NOT NULL, `b` int(11) NOT NULL, `c` int(11) NOT NULL, PRIMARY KEY ...
3
votes
1answer
49 views

What is the correct way to map relationships between structurally similar comments tables and commentable entities?

I'm working on the usual comments table in postgres that has a one-to-many relationship with a commentable entity. In other words, one comment belongs to a post, and a post can have many comments. ...
2
votes
0answers
38 views

Join duplicates on T-sql [closed]

Background: This is largely a learning exercise. I am working on a task scheduling program. Part of the program requires making a report to print out. I am working with the original version of the ...
2
votes
0answers
28 views

MS Access database returns multiple entries for same person with more than one child

I know this is a very basic question, but I've watched about six Access database videos and still can't find the answer. I'm a volunteer creating a membership database for a church in MS Access. The ...
2
votes
1answer
37 views

Should I create many ID for a same Person/entity in my DB?

In my model, I must store the name of the teacher of some kids and the classroom in which the teachers work, let’s say. All of the hundreds of teachers in the schools have 1 and only 1 classroom, ...
1
vote
1answer
181 views

Optimizing Queries HAVING COUNT and INSERT…ON DUPLICATE KEY UPDATE

There are two main queries I'm tying to improve, one question is much more conceptual and learning about databases and another is much more straightforward. Both cases deal with the same table It has ...
1
vote
1answer
393 views

MySQL RDS Error 1062

Normally we can skip 1062 duplicate error but in rds can we skip multiple 1062 errors in one go. For example we define slave skip error =1062 in my.cnf and restart slave. but in RDS MYSQL how can we ...
0
votes
1answer
83 views

Mysql: Create a view with multiple self joins without duplicates in result

Just to clarify i can't change the tables structure, so please leave out the "you should change your tables to this and that" answers, thank you. So i have a table entities_attributes_values where an ...
1
vote
1answer
185 views

Merge duplicate rows into one row

I have a table like below in MySQL: +-----+------+-------+-----------+----------+---------+ | #id | NAME | empid | c_lan | Java_lan | Dot_lan | ...
2
votes
1answer
468 views

Remove duplicate records from table, leaving one copy

I have inherited a MySQL table from one of my colleagues, and am in need of a little advice. The table has the following structure: | PHONE_NO | ACC_NO | GET_MAIL | DATETIME_CRE | INT_ACC ...
1
vote
1answer
74 views

How does RANK() work in this query? [closed]

I want to delete duplicate rows from a table. I have searched for the query online, and found the following sample: DELETE FROM dbo.LeadsContacts WHERE Sr_No IN ( SELECT Sr_No FROM ( ...
1
vote
1answer
1k views

SQLite3 - remove duplicates by multiple columns

I'm looking for the most efficient way to remove duplicates from a table where id's are unique but there are equal rows when you check other columns. Example: id name age x 1 peter 25 ...
1
vote
1answer
408 views

How to remove duplicates and keep the one with the most additional information

I have a postgres table (postgres version 9.3.4) of addresses with duplicate addresses: Column | Type | Modifiers ...
-1
votes
1answer
98 views

SELECT rows with duplicate values

I have a table of names in mysql InnoDB as Firstname Lastname Steve Ross John Doe J. Doe Mike Ross Michael Ross Roger ...
0
votes
1answer
51 views

Uniq join 3 tables with relationship table not uniq

I am using Postgres. I have 3 tables: table_1 id table_1_2 table_1_id, table_2_id table_2 id The problem is the relationship table table_1_2 is not uniq. I.e., table_1_2 will have the ...
1
vote
1answer
174 views

Delete Duplicates Leaving One of Two Identical Records [closed]

Delete Duplicates Leaving One of Two Identical Records One of my MySQL database tables contains four BLObs in each row. When it was first set up the table had no keys. However when I now try to add a ...
0
votes
1answer
69 views

How to avoid duplications using a function

I am using this function in a bulk insert to avoid duplication of url paths. CREATE OR REPLACE FUNCTION "univ"."gc_landing"(IN _name text, OUT landing_id int4) RETURNS "int4" AS $BODY$ DECLARE ...
0
votes
1answer
971 views

Oracle: duplicate schema to another database

I want to copy an Oracle schema from a production database to a validation database. Both databases are Oracle 11g R1 standard edition and need less interrupting time as I can (but I can do if it ...
1
vote
1answer
284 views

Check if query will cause duplicate row

I have a database with a primary key on the first column (words) I want to trim the trailing and leading ends of my data, but I get an error that this causes duplicate data EX: ' David ' will become ...
2
votes
3answers
4k views

Find duplicate records across one-to-many relationship tables

Let's say I have two tables that have a one-to-many relationship: People and Attributes. I'm trying to find duplicates based on the first_name, last_name, and all attributes must match exactly. ...
1
vote
2answers
60 views

RMAN backup set, nothing else, how can I duplicate?

I have what appears to be an RMAN backup set, it has been multiplexed in 32 files, each containing 2-3 dbf's inside. The backupset refers to an SPFILE file, but it was not included in the set, the ...
4
votes
2answers
630 views

Duplicate database leaves orphan archivelog files

I am cloning the production database to another one, lets say CLONEDB, in a regular basis. I am using rman duplicate database script like: run { duplicate target database to CLONEDB FROM ACTIVE ...
0
votes
2answers
55 views

Duplicate databases error

I have two Oracle 11gR2 databases on separate servers. I want to duplicate these databases. For this, I do these steps : In auxiliary db : create password file with password that is same if ...
1
vote
1answer
2k views

Remove all records with duplicates in db2. (Not just the duplicate records)

How can I remove all the records with duplicates in db2. I have looked at various answers but they only remove the duplicates leaving one record from that set in the table. This is what I found ...
0
votes
1answer
568 views

Remove duplicate terms from column

This the tags column where all terms should be listed only once. Would like to remove all additional words from this column. For example the below contant: Lakshmi Bar & Restaurant, Dining ...
2
votes
1answer
2k views

Using EXCEPTION to ignore duplicates during bulk inserts

In pq 9.3.5, I'm importing records from an external source where duplicates are VERY rare, but they do happen. Given a readings table with a unique compound key on (real_time_device_id, recorded_at), ...
1
vote
1answer
206 views

Error while duplicating Oracle database after failover, using 11.2.0.4

I've recently updated oracle from version 11.2.0.1 to 11.2.0.4 and my redundant system is no longer able to recover from failover. I'm using scripts both on primary and standby node to get back to a ...