Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [duplication]

The Duplication tag covers the topic of duplicated data and the strategies involved in removing the duplicates, often referred to as deduplication or simply dedupe.

42 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2 votes
0 answers
204 views

My relational schema and a few questions about relations and duplicate values in columns

I'm trying to create database in which I could store thousands of job postings. I put here 5 sample records of these job postings so that you could better understand how everything looks like: Now, ...
beginsql's user avatar
2 votes
0 answers
854 views

Postgresql 14 Primary Key not working, allows duplicates

I discovered that my 3-column (integers, not null) primary key is allowing duplicates! Then I created new unique index and replace the current one, and the problem still exists, how is that even ...
sh4rkyy's user avatar
  • 139
2 votes
1 answer
95 views

How to copy part of a database with inconsistent privileges?

I have a Postgres database and I would like to copy certain tables from a single schema into a new empty database. So far I have copied a schema-only dump from from the old DB and applied it to a ...
jase81's user avatar
  • 541
2 votes
0 answers
112 views

MS Access database returns multiple entries for same person with more than one child

I know this is a very basic question, but I've watched about six Access database videos and still can't find the answer. I'm a volunteer creating a membership database for a church in MS Access. The ...
Amanda Lietman's user avatar
2 votes
1 answer
574 views

DISTINCT gives duplicate values

Can it be considered a bug? This query gives duplicate values despite DISTINCT: select distinct '1' from ( select * from dual connect by level <= 10 ) order by dbms_random.value fetch first 10 ...
Andy DB Analyst's user avatar
2 votes
1 answer
392 views

Does PostgreSQL have a variable character storage layer that optimizes storage space by automatic dedupe?

My application has a Product and a Purchased table. The Purchased table includes a full copy of the Product row at time of purchase, so that, even if a product name or description was changed, the ...
Chris Betti's user avatar
1 vote
0 answers
217 views

Preventing three SQL Left Joins from creating duplicate rows

I have 4 tables [tA] is the main data point. A customer could have more than one ID Code, or the same ID Code which has been renewed, and ID Codes can be transferred between customers, but a customer ...
sql-newbie's user avatar
1 vote
0 answers
102 views

Composed primary keys : "should" duplicates and null values be accepted, and according to which norm?

I just passed some SQL test including a question about the possibility to have duplicates in a primary key column. The expected answer was NO. Mine is, "Yes, if the column is part of a composed ...
Balmipour's user avatar
  • 131
1 vote
0 answers
28 views

Selecting Duplicates on All Fields

I have an MS Access (no laughing at the back) database I've used to import a bunch of IIS logs into. Having looked at the Excel files I pulled these in from, I'm worried going by the dates that some ...
user788561's user avatar
1 vote
1 answer
940 views

Duplicate rows with primary key constraint in PostgreSQL 10.6

I have a table that's had CONSTRAINT table_pkey PRIMARY KEY (id) since setup. However, I recently found that I have around 100 duplicate rows (I ran select count(*) from (select count(*) from table ...
mckennab's user avatar
  • 111
1 vote
0 answers
454 views

MS SQL, fastest way to delete 99% of rows from 100+ million rows table to deduplicate data

Due to an error in our application, one of the tables in our MS SQL has been flooded with duplicate rows differing only in PK identity column and timestamp +--------+---------------------+------------...
Ivan Koshelev's user avatar
1 vote
0 answers
64 views

How to remove duplicate records from result set of owner

I'm working with one project like rental agreement system. There are two roles types of users (User Role) Signer = 0 Observer = 1 Anyone can register and create a document(which will be referred as ...
Div's user avatar
  • 111
1 vote
0 answers
363 views

Definition and declaration of a PRIMARY KEY for a temporal table

My teacher gave us an example bitemporal table holding rows related to medical records to test our homework on. The table has the columns that follow: first name last name parameter value unit valid ...
Gil-Mor's user avatar
  • 111
1 vote
1 answer
2k views

duplicate key error index MongoDB

I have rocket.chat server connected to LDAP. When one of users try logging to their account he get logged to someone other account, it's probably because indexes. My logs look like this: [...
Simonell's user avatar
1 vote
0 answers
59 views

SQL server schema to accomodate duplicate PK/FK values from autonomous Access data bases

I have several test stations with Access databases that have identical structures (12 related tables each). They are independent and all internal PK & FK values (Autonumber PK & Long Integer ...
LLK's user avatar
  • 11
1 vote
0 answers
58 views

How do I deduplicate Drupal taxonomies on a PostgreSQL database?

Background I've been tasked with deduplicating a Drupal 7 install. There are about 15 thousand articles published, which have been imported directly into the database using some custom in-house ...
That Brazilian Guy's user avatar
1 vote
0 answers
741 views

Finding Duplicate URLs

The following URL varchar(2000) row holds URL values in a 3rd party database that I'm accessing directly, i.e. going around the HTTP API. MySQL [database]> desc TABLE; +--------------------+-------...
Kevin Meredith's user avatar
1 vote
2 answers
2k views

Mysql: Create a view with multiple self joins without duplicates in result

Just to clarify i can't change the tables structure, so please leave out the "you should change your tables to this and that" answers, thank you. So i have a table entities_attributes_values where an ...
Mr. Sam's user avatar
  • 111
1 vote
0 answers
235 views

Looking for a recipe to merge multiple rows to single row

I have a aggregation table that is written by multiple processors (around 20). Each row has a composite key - location-id, campaign-id,... and aggregate stats about it. I need to merge the ...
Avihai Marchiano's user avatar
1 vote
0 answers
6k views

Duplicate database for standby returns error ORA-19625: error identifying file /home/oracle/stdbyctrl.ctl

I'm working on creating duplication scripts for standby purposes. and it's constantly failing even though the backups are located in the exact same folder. DB structure is identical and spfile is ...
Nicolas de Fontenay's user avatar
1 vote
0 answers
1k views

Duplicate GROUP_CONCAT values

Writing a plugin for Wordpress using a custom post type and an additional alias table to hold repetition of the post events. The problem comes with duplicate post meta values. wp_post_meta table: ...
user25554's user avatar
0 votes
1 answer
58 views

History Logging Trigger Causing Duplication

I'm fairly certain I'm probably just overlooking something simple but here's the situation: I'm trying to create a "reusable" historical logging trigger for a production database. The intent ...
G_Hosa_Phat's user avatar
0 votes
1 answer
19 views

Flag identical group of rows

I would need to find a way to identify lines that are the same. The comparison must be made on the first 4 columns (A-B-C-D) If these are identical I would like to identify them by inserting 1 in ...
Olivier's user avatar
0 votes
1 answer
75 views

What is the best option to remove duplicates in big heap?

Our Server is SQL Server 2016 Database with Full Recovery model. We have table "A" - heap which has 280 millions of records and size 90 GB with 50 columns. This table contains around 5 ...
adam.g's user avatar
  • 355
0 votes
0 answers
24 views

Representing state of duplicate records across databases

I'm needing help formalizing this problem so that I can better get a handle on it. Or so I can at least better do google searches for it. Suppose an external system and your system maintain the same ...
Orion Yeung's user avatar
0 votes
0 answers
99 views

How to prevent insertion of row which differs only in primary key from existing row?

If my table contains the following +----+-----+------------+-----------+ | id | uid | date | operation | +----+-----+------------+-----------+ | 1 | abc | 2022-09-21 | create | | 2 | xyz | ...
loris's user avatar
  • 103
0 votes
0 answers
440 views

Why is restoring a fresh database from a backup failing due to a duplicate key error?

I'm trying to build a replica from an active database with no downtime for replication purposes. I run the following statement to get a backup, avoid locking the tables, and flush the logs so that the ...
Timothy Fisher's user avatar
0 votes
1 answer
175 views

merging nearly- duplicate rows from different databases in the same table

I am sure , that many people before me faced a need to merge nearly- duplicate rows from several databases. However, I have not been able to find a simple yet general solution. I provided a data ...
YW T's user avatar
  • 11
0 votes
0 answers
22 views

how to Manipulate duplicated records using access

Using Microsoft Access, I have a number of records that have duplicated field-1. I need to form a query such that the duplicate records get unique ; BUT the next filed-2 data must be seen in columns ...
user4549384's user avatar
0 votes
0 answers
916 views

Bulk INSERT ... ON DUPLICATE KEY UPDATE inserts only a small fraction of rows in MariaDB

I have a table like the following: CREATE TABLE `demo_upsert` ( `ID` BIGINT(20) NOT NULL AUTO_INCREMENT, `Url` TEXT NOT NULL COLLATE 'utf8mb4_unicode_ci', `LastListID` BIGINT(20) NULL ...
Serge Rogatch's user avatar
0 votes
1 answer
29 views

Need to Bring Back most Recent Departure Date from Sales table for Customers in Recipient table

Looking at 3 Tables - Recipient, HouseholdScores, Sales. I need to bring back the most Recent tsDepart (departure date) from sales for the iCustomerId in the Recipient Table. There is no Depart date ...
Sam Tresidder's user avatar
0 votes
1 answer
44 views

Remove "almost" duplicate rows (may or may not start with plus sign)

To remove duplicated rows from Oracle, I tend to use the following: DELETE FROM your_table WHERE rowid not in (SELECT MIN(rowid) FROM your_table GROUP BY column1, column2, column3); (taken from SO) ...
menteith's user avatar
  • 101
0 votes
1 answer
162 views

How to remove these duplicate records in an Oracle Database 12c?

I have these two tables: messages(id primary key, message_date, created_at, ...) user_messages(user_id, message_id references messages(id)) I have some duplicate rows in messages: select user_id, ...
orly's user avatar
  • 21
0 votes
0 answers
73 views

How to properly find duplicate rows with spatial and temporal criteria

I have a table with historical rows indicating the positions in which ocean samples were taken. From this table, I would like to find duplicates based on spatial and temporal criteria. Thus, the ...
Amaru's user avatar
  • 1
0 votes
2 answers
6k views

How to delete records without a primary key in a stardard way (i.e. not using ctid)

Case of use We have a log table without a PK with two columns (date, event); You inserted a row. Then you want to undo the inserted record. Is there a way to delete that row (without deleting ...
Emilio Platzer's user avatar
0 votes
1 answer
73 views

Working out Duplicated Data in a Customer Data Base

I have a database in SQL2008 of customers, There is an ID number for the customer. What im trying to do is to create a NEWID field. We have duplicated customers that have different ID fields, I am ...
Aron.Ridgway's user avatar
0 votes
0 answers
188 views

Synchronization between master and backup server

I am searching for an option to synchronize data between production server and backup server automatically. Does somebody know a method, how I can do this: if I delete, insert or update a record it ...
user avatar
0 votes
1 answer
469 views

Manipulate duplicated records

In a table ( in Microsoft Access), I have a large number of records that have duplicated field, for example field-1. I have to change the table and set field-1 as primary key. But as you know, it is ...
hasanghaforian's user avatar
0 votes
1 answer
68 views

Prevent creation of duplicate rows in database dynamically

I am new to databases so please forgive my mistakes. Here's the scenario. I use the web for my data source. Sometimes, like, for a keyword, what my crawler do is crawl same keyword repeatedly (not the ...
Aman Rawat's user avatar
-1 votes
2 answers
190 views

SQL to return preferred duplicate rows

Have a data set that some rows have a duplicate. Looking to return the row with IND = Y if it is present, otherwise to return the N. +------------+---------------+------------+----------+ | ...
Ken's user avatar
  • 1
-1 votes
2 answers
565 views

Duplicate sequence of rows

I have a history table (EnrollmentHistory) used to track changes to another table (Enrollment) in my database. The EnrollmentHistory table is populated by a trigger that fires whenever a change is ...
Tim's user avatar
  • 19
-1 votes
1 answer
734 views

Why Do I Still Get a Duplicate Entry Error When There is No Duplicate Entry?

I am trying to insert this data into my database: https://docs.google.com/spreadsheet/ccc?key=0AjQflI82acM3dFI1Mk04dWlYSFJtM0hsZ0g4ZlExMWc&usp=sharing I receive the following error: 1062 - ...
tacudtap's user avatar