What I'm trying to do is running a job on more than 100 million domains which haven't processed before.

I have two tables, "domain" and "domain_setting", on every batch (10.000 domains per batch) I'm getting a list of domains from domain table by checking their status on "domain_setting". At first it was working fine but now approximately 10 million domain processed and it's getting slower.

I know that my server is crappy for this kind of database, it has only 8 GB of memory and it's also shared with Apache web server (150 concurrent connections), some other small databases and some lightweight PHP applications. My innodb_buffer_pool_size is only 1 GB.

Anyway, I would like to learn if there is a better way or thoughts to write this query more efficiently or any other approach for doing this job and speed up the process. Here is my SQL query:

SELECT * FROM domain
WHERE domain_ID NOT IN (SELECT domain_ID FROM domain_setting) OR domain_ID IN (SELECT domain_ID FROM domain_setting WHERE is_keyword_checked = 0)
LIMIT 0,10000;

I must get domains which are not exist in domain_setting table OR has "is_keyword_checked = 0" value in domain_setting table. How can I optimize this query?

Thanks in advance :)

EDIT: I can rewrite my query with LEFT JOIN as

FROM domain
LEFT JOIN domain_setting ON domain_setting.domain_ID = domain.domain_ID
WHERE domain_setting.is_keyword_checked = 0 OR domain_setting.is_keyword_checked IS NULL
ORDER BY domain.domain_ID ASC
LIMIT 0,10000;

but the main problem here seems full table scan. I can run it backwards by ORDER by domain.domain_ID DESC for recent added records and I can get them very fast. But still needs to scan full table if you didn't run it for a while and if recent records became more than millions of rows. So, is there any way to avoid full table scan for such work?

  • 1
    The answer depends, in part, on whether domain_ID is the primary key or is at least a unique key in domain_setting. I'm resisting the temptation to assume that it is. May 10, 2013 at 0:12
  • @Michael-sqlbot domain_ID is primary key for both tables. Also I have index for "is_keyword_checked" on domain_setting table. May 10, 2013 at 7:39
  • How many possible values can the is_keyword_checked column take? Just two (0,1) or more? May 10, 2013 at 9:24
  • @ypercube yes, only 1 and 0 May 10, 2013 at 9:34

2 Answers 2


Since is_keyword_checked can take only two possible values, 0 or 1 (and it's not nullable), you can try the rewriting. There is no OR and the index on (is_keyword_checked) will be used:

FROM domain
      ( SELECT domain_ID 
        FROM domain_setting 
        WHERE is_keyword_checked = 1
      ) ;

or with a NOT EXISTS subquery - which should have similar execution plan:

FROM domain AS d
      ( SELECT * 
        FROM domain_setting AS ds
        WHERE ds.is_keyword_checked = 1
          AND ds.domain_ID = d.domain_ID
      ) ;
  • Thank you very much, since the first query makes full table scans on both table (also because is_keyword_checked=1 data are about ten millions of rows), second query seems nearly 2x faster for now. Also it seems faster than my second query with LEFT JOIN. I can't predict what it would be when is_keyword_checked=1 data reaches 100 millions of rows but we will see :) Then probably I will only work on recent data by descend ordering domain_ID and it should be much faster. May 10, 2013 at 11:46

If I'm reading your query correctly your asking for anything NOT in domain_setting EXCEPT for those with is_keyword_checked = 0. You should be be able to just get rid of that first not in check.

Further subqueries are not always as efficient as you might think. You could rewrite the query to be

SELECT * FROM domain join domain_settings using(domain_id) where is_keyword_checked =0

You'll want indexes on domain_id on domain. for domain_settings a compound index across (domain_id, is_keyword_checked) would be beneficial.

  • Thank you very much but your query getting only domains on "domain" has already record in domain_setting table and is_keyword_checked=0. So unfortunately it doesn't get domains which doesn't exist in domain_setting table. In default, no domain has record on domain_setting table, after processing them I add record to domain_setting table. I need to get unprocessed ones and processed ones with is_keyword_checked=0 May 10, 2013 at 8:06
  • So my query needs anything NOT in domain_setting INCLUDING processed ones with domain_setting.is_keyword_checked = 0 May 10, 2013 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.