The Duplication tag covers the topic of duplicated data and the strategies involved in removing the duplicates, often referred to as deduplication or simply dedupe.

learn more… | top users | synonyms (1)

0
votes
2answers
294 views

SQL Server De-duplication Trigger

I have a table that contains columns of Country, City, Location, Description. Due to the size of the description field it's not possible to bind the four fields into a composite key to prevent ...
1
vote
2answers
505 views

Delete duplicates with specific columns as identifier

I'm required to delete all duplicate entries of a table. The duplicates are identified by two non-key values num (int) and com (varbinary(512)).. My amateur-ish SQLfoo lead me to this non-functioning ...
2
votes
1answer
72 views

Marking duplicate records

I have a list of things, some of which are duplicates. create table things ( id int primary key, title text not null, author text not null, duplicate_of int null references things(id) ); ...
2
votes
1answer
104 views

How do I remove duplicate records in a table?

I have a table table1 with 1 millions rows of data. I want to remove all the duplicate records in the table1 I was looking at this link: How do I remove duplicate records in a join table in ...
0
votes
0answers
130 views

Looking for a recipe to merge multiple rows to single row

I have a aggregation table that is written by multiple processors (around 20). Each row has a composite key - location-id, campaign-id,... and aggregate stats about it. I need to merge the ...
0
votes
1answer
78 views

Does PostgreSQL have a variable character storage layer that optimizes storage space by automatic dedupe?

My application has a Product and a Purchased table. The Purchased table includes a full copy of the Product row at time of purchase, so that, even if a product name or description was changed, the ...
1
vote
2answers
82 views

Preventing duplicate data when usual normalization only makes sense for some of the records

Background I have two tables, one called issues and the other called status_checks, which both contain (among others) the field comment (VARCHAR). An issue is raised by a user when something needs ...
5
votes
2answers
1k views

Partial db dump/restore

A part of my database (PostgreSQL 9.3) relies on extra tables (ex. County, City, Town, ... ). I don't manage these tables, they are updated regularly by a third party. Each time I get a new full dump, ...
1
vote
2answers
923 views

How do I find duplicate values in multiple columns over different rows?

I have a database with a table of games (some columns omitted): gameid hometeam awayteam `date` Sometimes games are added with the same home and away teams, but on a different date that is just 1 ...
0
votes
1answer
3k views

Oracle Duplicate Database with no archived logs

I am trying to perform a duplicate operation from one server to another (testing disaster recovery) on oracle 11g. I am very much still learning about Oracle. The system is not in production, it is ...
0
votes
1answer
57 views

Working out Duplicated Data in a Customer Data Base

I have a database in SQL2008 of customers, There is an ID number for the customer. What im trying to do is to create a NEWID field. We have duplicated customers that have different ID fields, I am ...
0
votes
1answer
185 views

duplicate entry issue with large mysql database table

I have a very large my sql database table. The other day, for no apparent reason, I now keep getting :- Duplicate entry '148245' for key 'PRIMARY' If I remove the entry at with primary of 148245, ...
3
votes
1answer
136 views

Consolidating 2 Tables with same Values

I am working on a project trying to consolidate two databases, DB1 and DB2, for my organization into a single database, DB3. Initially our organization started with DB1. A few years back they made a ...
10
votes
3answers
16k views

Checking whether two tables have identical content in PostgreSQL

This has already been asked on Stack Overflow, but only for MySQL. I'm using PostgreSQL. Unfortunately (and surprisingly) PostgreSQL does not seem to have something like CHECKSUM table. A PostgreSQL ...
0
votes
1answer
53 views

Move Commonly-Repeated Fields into a new Table

I'm still at the design stage for a new system, and I've noticed there are several places where different types of object share similar fields. In this case, we have customers, offices, warehouses ...
5
votes
1answer
583 views

Percona MySQL 5.5 Unique key is duplicated

I'm totally out from ideas, so maybe somebody else could answer my question. We have a MySQL 5.5 server from Percona with high traffic. The application is in PHP and writes always to master. We have ...
0
votes
2answers
1k views

Msg 2627 Violation of PRIMARY KEY constraint

trying to make a database for teams in a tv show here. but when I try and insert data into tblShowteam the following error made its appearance. Msg 2627, Level 14, State 1, Line 2 Violation of ...
1
vote
1answer
2k views

Master-Master Replication Duplicate Key

I have set two servers master-master replication mysql , i manage to set successfully but the problem is the duplicate key i got MASTER 1 Configuration /etc/my.cnf [mysqld] server-id=1 bind-address ...
0
votes
0answers
77 views

Finding duplicate data

I have noticed dup’s generated in my mail system. some service was causing duplicate calls to my insert proc and generated multiple duplicate messages with about a 4 mili-second difference in a date ...
4
votes
1answer
579 views

MySQL - GROUP_CONCAT returns duplicate data, can't use DISTINCT

I have a normalized database and I'm trying to return data from multiple tables using JOINs and GROUP_CONCAT. Problem: Rows are being duplicated with GROUP_CONCAT. I can't use DISTINCT because some ...
-1
votes
1answer
291 views

Why Do I Still Get a Duplicate Entry Error When There is No Duplicate Entry?

I am trying to insert this data into my database: https://docs.google.com/spreadsheet/ccc?key=0AjQflI82acM3dFI1Mk04dWlYSFJtM0hsZ0g4ZlExMWc&usp=sharing I receive the following error: 1062 - ...
0
votes
2answers
351 views

Make Oracle backup files available for duplicate

I have a Red Hat Enterprise Linux server with Oracle 10.2.0.5 and a database on it (SID is TNATEST). I have a remote CentOS server with Oracle 10.2.0.5 on it that I want to duplicate TNATEST to. The ...
1
vote
1answer
829 views

How do I validate all my constraints in postgresql?

It seems like I have been bitten by the first bug described in the release notes of postgresql 9.3.4: http://www.postgresql.org/docs/devel/static/release-9-3-4.html I now have e.g. duplicate primary ...
4
votes
2answers
1k views

How do I delete unique rows from a table with duplicate keys?

I have the following data: road | ID | other column | other column | ------------------+------+---------------+--------------+ MAIN ST | 1 | ... | ...
0
votes
1answer
60 views

What is meant by the RMAN Duplicate “noncurrent point in time”?

When doing a RMAN DUPLICATE from backup connected to the target, how current does the duplication make the database when no until time is set? The closest thing I can find in the documentation is the ...
2
votes
2answers
4k views

Insert into … (…) (Select…) on duplicate key update

Can you help me to solve a problem. The task is to insert a record in MySQL database. terms: If the record exists - to update the record. If the record does not exist - to create a new one. ...
2
votes
2answers
5k views

Master - Master replication Duplicate entry

I've got a Master - Master replication setup and it worked fine for a couple of days but I keep getting: Duplicate entry errors after a few days... My setup is as follow: One server in The ...
0
votes
1answer
9k views

INSERT / UPDATE to merge data, when empty data is an empty string

Consider a table of some family members: mysql> SELECT * FROM some_table; +--------+--------+--------+ | f_name | l_name | work | +--------+--------+--------+ | Meirav | Cohen | school | | ...
8
votes
3answers
61k views

select rows where column contains same data in more than one record

I have a table that has a column called article_title. Let's say the table name is articles. I need to find out the records where the article_title data is the same on more than one record. Here's ...
1
vote
3answers
170 views

Duplicates while doing multiple joins

I have 4 MySQL tables 1. companies - {id*, company} 2. employees - {id*, fname, lname, id_company*} 3. technologies - {id*, technology} 4. company_technologies - {id_technology*, id_company*} Field ...
3
votes
1answer
117 views

Storing statistic results

I'm doing a small database for statistics, it's a really simple statistic like calculating the cost of an event and divide it by the amount of people that went to get the cost per attendee, but it can ...
3
votes
2answers
214 views

How do I deal with items that I need to repeat monthly?

I have a table where users will enter transactions on a weekly basis, these transactions will either be single items or will be recurring i.e. rent would be recurring and should be the same for every ...
1
vote
1answer
136 views

Should I duplicate the data or have linking table?

We have the following data: Collections of items Individual items These will go into the following tables: ItemCollectionTable (Id, Name, OtherFields (desc, date, etc..)) ItemTable (Id, ...
1
vote
1answer
3k views

Duplicate database for standby returns error ORA-19625: error identifying file /home/oracle/stdbyctrl.ctl

I'm working on creating duplication scripts for standby purposes. and it's constantly failing even though the backups are located in the exact same folder. DB structure is identical and spfile is ...
1
vote
2answers
2k views

Distinct rows from a join

I have two MySQL tables: ╔═══════════════════╗ ║ wpss_question_set ║ ╠═══════════════════╣ ║ id ║ ║ quesset ║ ╚═══════════════════╝ ╔════════════════╗ ║ wpss_questions ║ ...
0
votes
0answers
29 views

Merge rows with common field value in MySQL [duplicate]

I have a table which was imported from MS-SQL, it has about 1.9 million rows. After importing I realized that a lot of the rows are duplicates and there are no primary key constraints. The rules to ...
1
vote
0answers
640 views

Duplicate GROUP_CONCAT values

Writing a plugin for Wordpress using a custom post type and an additional alias table to hold repetition of the post events. The problem comes with duplicate post meta values. wp_post_meta table: ...
0
votes
0answers
110 views

(My)SQL: management and duplication of multiple sub-databases [closed]

I have a pretty simple question, but it seems that there are controversies as to how to solve it in the best possible way. Say I have a web app that makes online surveys. My challenge is that each of ...
0
votes
1answer
107 views

How to group by when querying duplicate records?

I have a table with duplicates and I want to present it to the stakeholders so they can help me determine the source of duplication. Currently I can print 1 row for each duplicate entry, but I want ...
0
votes
0answers
136 views

Synchronization between master and backup server

I am searching for an option to synchronize data between production server and backup server automatically. Does somebody know a method, how I can do this: if I delete, insert or update a record it ...
1
vote
1answer
166 views

How can I achieve a unique constraint with two fields? [duplicate]

I have a table with e.g. Name and IsDeleted fields. I want to add a row constraint so that only one Name value can have IsDeleted as 'false'. There can be many duplicate Name values, but they must all ...
0
votes
0answers
489 views

Duplication Oracle Database to remote host

I have two scenarios. 1 ) Let's say , I have the same file system and same SID on both nodes. I've created any directories necessary for start the duplicate database. First I used command below. ...
2
votes
2answers
656 views

Duplicating a Database with RMAN

I would like to duplicate to the same server Oracle Database with RMAN. I've configured flash recovery area for Oracle Database. My database is using SPFILE as shown below. SQL> SELECT ...
1
vote
1answer
77 views

Summation query with complex grouping criteria

I have a database of coded actions with this structure. {User, Date, ActionCode} I need to generate reports based on dates and action codes, and the action codes are not clean because we have to ...
7
votes
1answer
19k views

On duplicate key do nothing

I am inserting into the following table using LuaSQL with PtokaX API. CREATE TABLE `requests` ( `id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT, `ctg` VARCHAR(15) NOT NULL, `msg` ...
4
votes
2answers
7k views

How do I remove duplicate records in a join table in PostgreSQL?

I have a table that has a schema like this: create_table "questions_tags", :id => false, :force => true do |t| t.integer "question_id" t.integer "tag_id" end ...
0
votes
1answer
134 views

Row locking or avoiding duplicate insertion on primarykey/otherfield combo

I have a table that stores additional data pertaining to files/images saved in files elsewhere. Each entry can be associated with another type in the system. For users there may be one or more ...
20
votes
1answer
37k views

Optimal way to ignore duplicate inserts? [duplicate]

Background This problem relates to ignoring duplicate inserts using PostgreSQL 9.2 or greater. The reason I ask is because of this code: -- Ignores duplicates. INSERT INTO db_table ...
5
votes
2answers
1k views

Resolving DUPLICATE for a column with UNIQUE

I have a few column with UNIQUE INDEX. In some queries, duplicates should be acceptable, in which a postfix should be added to the value. For example, if test title exits, we change it to test ...
5
votes
2answers
795 views

MySQL check duplicate with group by using wildcard?

+----+--------------+-----+-----------+----------+ | ID | NAME | AGE | ADDRESS | SALARY | +----+--------------+-----+-----------+----------+ | 1 | Ramesh Olive | 32 | ...