Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [duplication]

The Duplication tag covers the topic of duplicated data and the strategies involved in removing the duplicates, often referred to as deduplication or simply dedupe.

0
votes
1answer
53 views

RMAN Duplication - Oracle Data Guard - Syntax error(s) RMAN-01009 / RMAN-01008 / RMAN-01007

New to DB. Interested in Oracle Data Guard. New working with RMAN. All of this is for testing only and does not involve any kind of production env. Making use of ORACLE documentation: Step by Step ...
-1
votes
2answers
24 views

SQL to return preferred duplicate rows

Have a data set that some rows have a duplicate. Looking to return the row with IND = Y if it is present, otherwise to return the N. +------------+---------------+------------+----------+ | ...
2
votes
2answers
48 views

Sum up counts in subquery

I am writing a query to check if I have got duplicates in a certain column. SELECT COUNT(thecol) FROM thetable WHERE thecol IS NOT NULL GROUP BY thecol HAVING COUNT(*) > 1 That will give me an ...
1
vote
2answers
27 views

Comparing duplicates and choosing the duplicate with most recent date in a join

Assume i have table A: id: name: timestamp: 1 Bob 2018-05-31 04:46:37 2 Alice 2019-01-31 07:13:56 1 Bob 2019-01-15 18:23:74 And i want to join with another table B: ...
1
vote
2answers
53 views

How to remove partial duplicates from a table

I have a database that contains some erroneous results that are essentially but not technically duplicates. The structure is like this: id_page (PK), id_site, label, create_date There should only be ...
0
votes
1answer
25 views

Removing duplicate-ish paths generated from recursive CTE

I have some multi-parented data, that I run a recursive CTE against to find common ancestors: item_id |item_name |path_name |id_path_array | --------|----------|-------------|--------------| 1 ...
0
votes
1answer
35 views

How Compare two columns of Same SQL table to find duplicate records

I have one table with two columns: StartDateTime EndDateTime What will be best query to find out which rows has same (StartDateTime, EndDateTime)?
0
votes
1answer
21 views

Looking for a query that eliminates near-repeats

I have a database with four rows: county_a, county_b, flow_a_to_b, and flow_b_to_a. Essentially, the data is repeating in other rows, but the values are switched around. Here's an example: Entry 1: ...
1
vote
0answers
58 views

MS SQL, fastest way to delete 99% of rows from 100+ million rows table to deduplicate data

Due to an error in our application, one of the tables in our MS SQL has been flooded with duplicate rows differing only in PK identity column and timestamp +--------+---------------------+------------...
0
votes
1answer
36 views

Removing duplicates values with join

I have a query that looks like this: SELECT B.Company, COUNT(*) AS QTY FROM Table1 A INNER JOIN Table2 B ON A.CompanySk = B.EDWCompanySK WHERE A.StartedDate >= '20171128' GROUP BY B.Company ORDER ...
0
votes
2answers
215 views

Remove Duplicates (or Select Unique) from multiple rows, with all random data

I have a table that gets results from three different sources. Each column represents a source, and each row a result of an outcome. There are over 50k rows for a total of 150k results. I need to ...
2
votes
2answers
70 views

Ignore duplicates while using join

I'm trying to ignore duplicates between two providers in my tables while using join. I'm using this query for now: SELECT * FROM A AS a1, A AS a2 WHERE a1.provider <> a2.provider AND a1.etype =...
2
votes
2answers
69 views

Duplicate index - Same key, one with INCLUDE columns, which one to drop?

I have 2 indexes that have the same key, but one also has INCLUDE columns. Which one can I drop? They are the same, except IndexInclude has 6 INCLUDE columns. Both have some reads (IndexNoInclude ...
-1
votes
1answer
32 views

eliminating duplicates

I want to pull users that only have ONE deviceID in a given month. Let's say I have the following table: User Device ID Greg 1 Greg 2 Mike 3 Laura 4 Laura 5 Lisa 6 George 7 Peter ...
1
vote
1answer
46 views

Delete duplicates and reroute referencing rows to new master

Given three tables: A users table with id's: ╔════╗ ║ id ║ ╠════╣ ║ 1 ║ ╟────╢ ║ 2 ║ ╟────╢ ║ 3 ║ ╚════╝ A purposes table with user_id's and name's: ╔════╤═════════╤══════════╗ ║ id │ user_id │ ...
0
votes
2answers
48 views

How to delete records without a primary key in a stardard way (i.e. not using ctid)

Case of use We have a log table without a PK with two columns (date, event); You inserted a row. Then you want to undo the inserted record. Is there a way to delete that row (without deleting ...
1
vote
2answers
653 views

Insert into Table ignoring duplicate values

I am having a brain fart on figuring this out. I have the following two Tables: Table: parts part_id INT IDENTITY(1,1) NOT NULL, part_number VARCHAR(50) UNIQUE NOT NULL, ...
0
votes
1answer
378 views

oracle sql: How to not select select duplicate records from table

Can any help me to fix the below sql query SELECT DISTINCT ORDER_NUMBER, FLAG FROM TABLE Below was the result Order_number FLAG LP-13288 false LP-13288 true LP-13292 false LP-13290 ...
-1
votes
0answers
37 views

Weird MySQL behavior - duplicate entry which doesn't exist

I'm working on a piece of code to duplicate a database row but with a different foreign key and language id. So, lets say I got 2 tables: TABLE A: - id int PK - image varchar - order_num int - ...
2
votes
3answers
85 views

Exclude Duplicates With Certain Value

I'm been beating my head up against a wall with this all day. I have a SQL Server table called Accounts CREATE TABLE [dbo].[Accounts]( [IDNum] [INT] NULL, [FirstName] [VARCHAR](50) NULL, ...
2
votes
1answer
181 views

How to remove duplicates from any array and preserve ordering in PostgreSQL?

I'm looking for a way to eliminate duplicates from a PostgreSQL array while preserving the ordering of its elements. What I currently have are the following functions: create function array_unique( ...
2
votes
1answer
25 views

Looking for help reporting duplicates and null marks

Someone over at StackOverflow suggested this might be the place to ask this. I hope this will question is proper for this forum. I am looking for suggestions / criticisms of alternative methods of ...
0
votes
0answers
47 views

Duplicate Active Database - Dataguard Oracle 12C

Folks, Good evening. I am performing the configuration of a DataGuard in Oracle 12c. At the time of performing duplicate active database, the process has been aborted, reporting the error below. I ...
5
votes
1answer
402 views

Transfer row from 'parent' to 'child' without creating duplicate and without knowing all 'parent' columns

I want to set up a database with different schemas that correspond to different data-entry systems. There needs to be a “master” (administration) schema with a “parties” table, and multiple other ...
1
vote
1answer
33 views

MySQL - How to convert an existing int column with duplicates to auto-increment?

After adding some thousand records in a table, I noticed that the "Primary" and "Auto-Increment" properties of the main column are removed. Now instead of auto-incremented numbers, most of them (but ...
1
vote
0answers
52 views

How to remove duplicate records from result set of owner

I'm working with one project like rental agreement system. There are two roles types of users (User Role) Signer = 0 Observer = 1 Anyone can register and create a document(which will be referred as ...
5
votes
1answer
957 views

Multiple on conflict targets

I have two unique indexes on columns a and b. I need something like this: insert into my_table (a, b) values (1, 2), (1, 2) on conflict (a) do update set c = 'a_violation' on conflict (b) do update ...
0
votes
0answers
17 views

Is it possible to remove duplicate information derived from relationships while preserving constraint?

I am designing an application on MySQL database with the following structure: Competition 1-to-many Course Course 1-to-many Class Course 1-to-many Question Class 1-to-many Participant Question 1-to-...
1
vote
1answer
24 views

Oracle 11gR2, duplicating and refreshing cloned database

Assume I've cloned PROD database to TEST with RMAN "duplicate target database to CLONE from active database;". After several days I need to refresh CLONE. Will RMAN transfer all the datafiles from ...
0
votes
1answer
397 views

How to find index details on Temporary tables in SYBASE

I am a layman in SYBASE world. In order to improve performance, I am creating indexes after data load of temporary tables in many stored procedures. Say, one proc A has an insert into temporary ...
1
vote
2answers
38 views

How can I do a query that shows any row that have more than twice in the same table without using GROUP BY or COUNT

I want to select all same item that occurs in the same table twice without GROUP BY or COUNT sentence. How to write the query? For example, if the table is: City --------- New York Los Angeles ...
2
votes
1answer
151 views

Deduplicate Data Using Having with Subquery

I've inherited a DB that was lacking a unique constraint so data was/is duplicated. I'm now trying to remove the duplicate records, then add a constraint to block this going forward. I have this ...
1
vote
4answers
4k views

Select unique values for one column

There is a table with different sets a(1,2), b(2,3,4) and etc. -- table1 -- set_name | value a | 1 a | 2 b | 2 b | 3 b | 4 c | 3 c | 5 ... | ... ...
2
votes
1answer
208 views

Prevent the collision data because duplicate id from imported database MariaDB

I am currently having a trouble with my Web Application. I made a Web application for a certain company. I made the app using CodeIgniter 3. I built the database using Maria DB. For the id in each ...
1
vote
1answer
59 views

Data Scrub - Duplicate Records

I wanted to delete duplicate records in my database, but it's a little more in depth than just one field. Status ID P 150013511822029 P 150022555544009 P 150022551512029 X 150014520920007 R ...
1
vote
3answers
181 views

How to prevent duplicate or similar data from getting processed?

I have a stored procedure which calculates the distance between 2 coordinate points: CREATE PROC loc.CalcDistanceByPostalCode ( @From VARCHAR(10), @To VARCHAR(10) ) AS BEGIN DECLARE @data table ...
3
votes
1answer
8k views

How do I insert record only if the record doesn't exist?

I have a PaymentInformation table ID NAME Start End 1 Tina 1/7/2014 1/17/2014 2 Alice 3/7/2014 3/17/2014 3 John 5/7/2014 5/17/2014 4 Michelle 7/...
8
votes
1answer
503 views

Duplicate records returned from table with no duplicates

I have a stored procedure that queries a busy queue table that is used to distribute work in our system. The table in question has a primary key on WorkID and no duplicates. A simplified version of ...
3
votes
1answer
773 views

Why does MySQL flag a duplicate key where none exists?

I'm working on adding an auto-increment field to a large table. The field is already populated (roughly 1-500M) and new inserts are manually incrementing correctly. To verify, I ran a query to show ...
1
vote
1answer
35 views

Is this data duplication when two exact pieces of data have different lifespans?

In my system, user has many bank accounts stored in bank_accounts table. Each bank account has a currency column. Now, users can receive payments to those bank accounts, stored in payments table. So, ...
0
votes
1answer
39 views

How PostgresSQL stores duplicated VARCHAR values in a column

I have a VARCHAR(255) column in a table with about 50 million rows. In that column there is only 10 unique values, which act as labels. I'm wondering whether Postgres does something smart and stores ...
1
vote
1answer
662 views

Find rows that have the same value on a column in MySQL

In an employee table, some rows have the same value for the reportsTo column. employeeNumber | reportsTo ---------------|----------------- 1056 | 1002 1076 | 1002 1088 | ...
3
votes
3answers
594 views

Contacts table with multiple entries?

Here is a part of our Database: Table: Users ID INT First Name VARCHAR Last Name VARCHAR Email Address VARCHAR ... Table: Contacts ID INT User_id INT (Foreign Key)...
0
votes
1answer
800 views

Removing duplicate results via subqueries in reference to join conditions for multiple tables

With the code below I was originally attempting to not use sub queries to eliminate "duplicate data" (its technically not duplicate data since each order has several lines of data where prices and ...
3
votes
1answer
52 views

XML domains(lookups): Prevent duplicates between domains

Background: I have some GIS domains (a.k.a. lookup tables) that are stored in an XML column in a single system table. The domains are not directly accessible via SQL, so I use a view to extract the ...
4
votes
1answer
748 views

MySQL 5.5 vs 5.6,5.7 above same database same query but different output, limit have duplicated ID

I am using Running same source code and same database for WordPress, only the mysql version is different. It make my pagination post have duplication. Here is what I found out. Any idea how to solve ...
0
votes
1answer
55 views

Manipulate duplicated records

In a table ( in Microsoft Access), I have a large number of records that have duplicated field, for example field-1. I have to change the table and set field-1 as primary key. But as you know, it is ...
1
vote
2answers
959 views

How to select a specific row based on another row from same table [duplicate]

I have a table with duplicated rows, I want to select only one row out of many that has a field populated in it. Example: +--------+------+---------------+ | id | col1 | col2 | +--...
3
votes
1answer
899 views

does INSERT INTO … ON DUPLICATE KEY UPDATE executes even if the row has not changed?

Let me explain : Let us assume we have a database "user" : ID Name Surname =================== 1 John Doe 2 Foo Bar 3 Foo Baz Now I wanna change all these three rows. I will ...
3
votes
2answers
647 views

Remove Duplicate Entries from DB w/ Unique ID that is NOT Primary Key - Keep one of course!

I have a CDR Table with a "unique id" column which is not a Primary Key. During an upgrade, there were a few days where instead of one entry for a caller ringing a queue, there was multiple entries ...