Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I'm not very good with databases so please keep in mind even if the solution sounds like something everyone would have already tried, please post it because I likely have not tried it and it would save a ton of time if it works.

I have two tables:

  • items: ~40,000 Rows
  • inventory: ~1,000,000 Rows

I have a query that matches up the inventory with the items the way the old system did, and updates the itemid column in the inventory table, to the id of the matched row in the items table (to work with the new system).

The problem is that the queries are taking forever, and I have very similar queries that I have to run in order to prioritize similar matches so waiting this amount of time isn't very great as the data in the old database is not consistent at all, so I have to continuously tweak, rerun, tweak, rerun, etc. It was taking a little over 40 minutes per query when there was only 250,000 rows in the inventory table.

In a nutshell, I need to make this query faster:

UPDATE inventory r
JOIN items t ON (FIND_IN_SET(r.orderNumber,
                REPLACE(
                    REPLACE(
                        REPLACE(t.orderNumber,'/',','),
                    ' ', ''),
                '[]',',')
            )
            AND r.selected_number != '99999'
            AND r.guideid = t.guideid
            AND (r.itemid IS NULL OR r.itemid = '')
                 )
SET r.itemid = t.id

My indices are (comma delimited obviously being composite, please excuse the names they were auto-generated by SQLYog):

items table:

PRIMARY => `id`
guideid => `guideid`, `orderNumber`
guideid_2 => `guideid`
orderNumber => `orderNumber`

inventory table:

PRIMARY => `id`
guideid => `guideid`, `orderNumber`
temp_usNum => `temp_usNum`
orderNumber => `orderNumber`
selected_number => `selected_number`
guideid_2 => `guideid`
itemid => `itemid`
itemid_2 => `itemid`, `guideid`, `orderNumber`, `selected_number`

Here is the result of the explain query:

explain output

I have access to all server settings, as it is on my local machine.

share|improve this question

2 Answers 2

up vote 6 down vote accepted

With your current structure, your query may be doing >1,000,000,000 row scans. This is probably due to the FIND_IN_SET - meaning the column cannot be used in an index, resulting in the optimizer choosing other indexes with very bad selectivity (you have to read 500000 rows instead of 1 million, that is very bad selectivity). The fact that an index is being used does not mean good performance, if it is the wrong index. To understand why, you must know that function(column) = value cannot be indexed directly in MySQL.

I would start by changing the column items.ordernumber and putting that information on a separate table, with only one item per column, instead of the denormalized arrangement you have currently.

With that, and the proper indexes, you will be able to do the query in the items -> inventory direction and use ordernumber to filter adequately your rows.

As a secondary thing, you have many redundant indexes, like (guideid, orderNumber) and (guideid). You are in need of a good cleanup in terms of design.

To measure the number of row operations that you are doing for a particular query, do the following:

FLUSH STATUS;
-- Execute query
SHOW STATUS like 'Hand%';
share|improve this answer
    
@Paul White Thanks for the edit, sometimes I excite myself too much when answering O:-) –  jynus Aug 7 at 14:53

I think the quickest way is to do it outside the DB or do the same with procedures and temp tables in the DB.

Naive approach assuming the guideid is useful, but probably its value range is too narrow and creating too many join results:

SELECT xyz FROM r join t using (guideid) WHERE r.selected_number != '99999' AND (r.itemid IS NULL OR r.itemid = '')

Otherwise get the orderNumber from t into an own table with one row id and one row ordnerNumber, so you can use a nice join statement.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.