Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've been tasked with identifying a solution to help identify possible duplicates when inserting new user records into a MySQL database. Does anyone know of any cheap, preferably free, solutions?

The sort of duplicates I need to detect are common forms of shortened English names - such as Bill for William, Bob for Robert etc - along with misspellings as well as plain duplicates. I've read a bit about using Lucene but it seems to be more for full text searching and I'm unsure if it supports the dupliatcae name matching I'm after?

share|improve this question

closed as off topic by Nick Chammas, Derek Downey, Paul White, Martin Smith, Mark Storey-Smith Nov 6 '12 at 15:35

Questions on Database Administrators Stack Exchange are expected to relate to database administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

Hm. Right question in the wrong place? Any luck so far? Perhaps SO should help as @atxdba says. If you ask me, it would be easier to build a custom db for the normalizer, starting say, from "baby names" sites. – Kris Jan 11 '12 at 6:40
Here's my SO question…. – sipwiz Jan 11 '12 at 9:19
up vote 3 down vote accepted

This might be better placed on stack overflow. This isn't something you'll accomplish with MySQL. What you're talking about is referred to as 'stemming' in search. Similar to matching different conjugations of a regular word e.g. run => runs ,ran.

I don't know of any such applications for proper names off hand but when you find one that will sit alongside your primary application to "normalize" the name before inserting the record into your database. Mysql, sqlserver, mongo, whatever. The DB technology is irrelevant as your task is out side the scope of storing data/documents.

Lucene would be a better tool for your task. But I couldn't speak to it's prepackaged ability to stem names like you want.


After thinking about it I think I misspoke when I said Lucene would be a "better" approach in of itself for what you want. My understanding is stemmers exist outside of core lucene and then proxy a search for "bob" into ("bob" or "robert") to feed into the lucene engine.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.