I pretty much addressed this in StackOverflow, but I would like to elaborate more on using the permanent temp (PermTemp) table. (permanent temp, isn't that an oxymoron ?)
In StackOverflow, I had the stored procedure test.CreateSampleTable and test.GetMissingIntegers make a sample table and then create a dynamic temp table to populate before doing the big JOIN to find differences.
This time, let's create the sample table along with the permanent table table.
Here is test.LoadSampleTables:
DELIMITER $$
DROP PROCEDURE IF EXISTS `LoadSampleTables` $$
CREATE DEFINER=`lwdba`@`127.0.0.1` PROCEDURE `LoadSampleTables`(maxinttoload INT)
BEGIN
DECLARE X,OKTOUSE,MAXLOOP INT;
DROP TABLE IF EXISTS test.id_key_table;
DROP TABLE IF EXISTS test.id_key_table_keys;
CREATE TABLE test.id_key_table (id_key INT(16)) ENGINE=MyISAM;
CREATE TABLE test.id_key_table_keys (id_key INT(16)) ENGINE=MyISAM;
SET X=1;
WHILE X <= maxinttoload DO
INSERT INTO test.id_key_table VALUES (X);
SET X = X + 1;
END WHILE;
ALTER TABLE test.id_key_table ADD PRIMARY KEY (id_key);
SET MAXLOOP = FLOOR(SQRT(maxinttoload));
SET X = 2;
WHILE X <= MAXLOOP DO
DELETE FROM test.id_key_table WHERE MOD(id_key,X) = 0 AND id_key > X;
SELECT MIN(id_key) INTO OKTOUSE FROM test.id_key_table WHERE id_key > X;
SET X = OKTOUSE;
END WHILE;
OPTIMIZE TABLE test.id_key_table;
INSERT INTO test.id_key_table_keys SELECT id_key FROM test.id_key_table;
ALTER TABLE test.id_key_table_keys ADD PRIMARY KEY (id_key);
OPTIMIZE TABLE test.id_key_table_keys;
END $$
DELIMITER ;
After running this, here are the tables and their contents:
mysql> call test.loadsampletables(25);
+-------------------+----------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+-------------------+----------+----------+----------+
| test.id_key_table | optimize | status | OK |
+-------------------+----------+----------+----------+
1 row in set (0.20 sec)
+------------------------+----------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+------------------------+----------+----------+----------+
| test.id_key_table_keys | optimize | status | OK |
+------------------------+----------+----------+----------+
1 row in set (0.28 sec)
Query OK, 0 rows affected (0.29 sec)
mysql> select * from test.id_key_table;
+--------+
| id_key |
+--------+
| 1 |
| 2 |
| 3 |
| 5 |
| 7 |
| 11 |
| 13 |
| 17 |
| 19 |
| 23 |
+--------+
10 rows in set (0.00 sec)
mysql> select * from test.id_key_table_keys;
+--------+
| id_key |
+--------+
| 1 |
| 2 |
| 3 |
| 5 |
| 7 |
| 11 |
| 13 |
| 17 |
| 19 |
| 23 |
+--------+
10 rows in set (0.00 sec)
Here are the Triggers for the PermTemp table
mysql> DELIMITER $$
mysql>
mysql> CREATE TRIGGER test.AddPermTempKey AFTER INSERT ON test.id_key_table
-> FOR EACH ROW
-> BEGIN
-> INSERT IGNORE INTO test.id_key_table_keys VALUES (NEW.id_key);
-> END $$
Query OK, 0 rows affected (0.09 sec)
mysql>
mysql> CREATE TRIGGER test.DeletePermTempKey AFTER DELETE ON test.id_key_table
-> FOR EACH ROW
-> BEGIN
-> DELETE FROM test.id_key_table_keys WHERE id_key = OLD.id_key;
-> END $$
Query OK, 0 rows affected (0.08 sec)
mysql>
mysql> DELIMITER ;
Now, lets import a new batch of records, table test.weekly_batch, some keys used before, other keys brand spanking new:
mysql> CREATE TABLE test.weekly_batch (id_key INT(16)) ENGINE=MyISAM;
Query OK, 0 rows affected (0.04 sec)
mysql> INSERT INTO test.weekly_batch VALUES (17),(19),(23),(29),(31),(37),(41);
Query OK, 7 rows affected (0.00 sec)
Records: 7 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE test.weekly_batch ADD PRIMARY KEY (id_key);
Query OK, 7 rows affected (0.08 sec)
Records: 7 Duplicates: 0 Warnings: 0
Let's take test.weekly_batch and safely merge it into test.id_key_table_keys and form the table test.new_keys_to_load:
DELIMITER $$
DROP PROCEDURE IF EXISTS `test`.`ImportWeeklyBatch` $$
CREATE PROCEDURE `test`.`ImportWeeklyBatch` ()
TheStoredProcedure:BEGIN
DECLARE RCOUNT INT;
SELECT COUNT(1) INTO RCOUNT FROM information_schema.tables
WHERE table_schema='test' AND table_name='weekly_batch';
IF RCOUNT = 0 THEN
LEAVE TheStoredProcedure;
END IF;
SELECT COUNT(1) INTO RCOUNT FROM test.weekly_batch;
IF RCOUNT = 0 THEN
LEAVE TheStoredProcedure;
END IF;
DROP TABLE IF EXISTS test.new_keys_to_load;
CREATE TABLE test.new_keys_to_load (id_key INT(16));
INSERT INTO test.new_keys_to_load (id_key)
SELECT id_key FROM test.weekly_batch A
LEFT JOIN test.id_key_table_keys B USING (id_key)
WHERE B.id_key IS NULL;
SELECT * FROM test.new_keys_to_load;
END $$
DELIMITER ;
Here is the result:
mysql> call test.importweeklybatch;
+--------+
| id_key |
+--------+
| 29 |
| 31 |
| 37 |
| 41 |
+--------+
4 rows in set (0.14 sec)
From this point, just use the new_keys_to_load table as the list of brand spanking new keys to import. Since new_keys_to_load is smaller than the PermTemp table, you should always use new_keys_to_load on the left side of the LEFT JOIN.
