Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I have a database with over 1B rows and two columns that are indexed (in addition to the PK). Is it better to have the index pre-defined in the table before the load infile or better to index after the data has been loaded?

A couple of notes regarding data size and system:

  • System is Linux w/ 8 cores and 32GB memory (currently maxed out unless I move to new HW)
  • DB is 1B rows that in raw data size is 150GB data.
  • Database is MyISAM and is mainly read-only after it's loaded.
share|improve this question
add comment

2 Answers

I have tried a differen variety of solution with a similar data load -over 1B- but the better that I have found is this:

From mysql documentation

With some extra work, it is possible to make LOAD DATA INFILE run even faster for a MyISAM table when the table has many indexes. Use the following procedure:

  1. Execute a FLUSH TABLES statement or a mysqladmin flush-tables command.

  2. Use myisamchk --keys-used=0 -rq /path/to/db/tbl_name to remove all use of indexes for the table.

  3. Insert data into the table with LOAD DATA INFILE. This does not update any indexes and therefore is very fast.

  4. Re-create the indexes with myisamchk -rq /path/to/db/tbl_name. This creates the index tree in memory before writing it to disk, which is much faster that updating the index during LOAD DATA INFILE because it avoids lots of disk seeks. The resulting index tree is also perfectly balanced.

  5. Execute a FLUSH TABLES statement or a mysqladmin flush-tables command.

LOAD DATA INFILE performs the preceding optimization automatically if the MyISAM table into which you insert data is empty. The main difference between automatic optimization and using the procedure explicitly is that you can let myisamchk allocate much more temporary memory for the index creation than you might want the server to allocate for index re-creation when it executes the LOAD DATA INFILE statement.

In order to obtain better performance from the myisamchk you have to tune some params like :

--key_buffer_size --myisam_sort_buffer_size --read_buffer_size --write_buffer_size

Depending on your hw architecture

Note

When using LOCAL with LOAD DATA, a copy of the file is created in the server's temporary directory. This is not the directory determined by the value of tmpdir or slave_load_tmpdir, but rather the operating system's temporary directory, and is not configurable in the MySQL Server.

So, you you have this kind of problem and your file it's a csv, you can split you "huge" file into chunks

$ split -l (numbersofrowsinfile / ((filesize/tmpsize) + 1)) /path/to/your/<file>.csv

Then repeat your LOAD DATA LOCAL (step 3) for every chunk file.

share|improve this answer
add comment

I'll stay generic on this answer, as Cristian look to have already covered a significant number of MySQL specific considerations.

The general recommendation for bulk operations is definitely to remove and rebuild indexes afterwards. The amount of work to maintain the balance of the tree structures for each index is fairly high and depending on insert order can result in both significant index fragmentation (i.e. the location of pages in the data files) and the amount of space used (there may be a high proportion of part-used pages).

If you have no data already in the table then the generic advice is:

  1. Remove all indexes. Note that unique constraints are also indexes but I would leave them in place unless you are absolutely sure that incoming data is valid in that respect. PKs are indexes too, but you won't be able to drop them if FKs refer to them (and again you only should if the if you know the in coming data contains no duplicates). Contrary to popular belief foreign keys do not imply indexes in all RDBMSs (see http://www.sqlskills.com/blogs/kimberly/when-did-sql-server-stop-putting-indexes-on-foreign-key-columns/)
  2. Do the bulk insert.
  3. As the table is going to be mostly read-only rebuild the primary key (in many DBMS this will reduce fragmentation and space waste) with a high fill-factor (if your DBMS allows you to tweak this).
  4. Recreate the indexes, and unique constraints if you dropped those, again with a high fill-factor. If you have a clustered index then make sure this is (re)created before other indexes and constraints.

If you already have data in the table then this advice need careful reconsideration of course. I remember an old "rule of thumb" that it was worth dropping and recreating the indexes if you were adding or updating more than 70% of the table's content, otherwise not, but I don't know if that rule had any basis in properly run experiments or if it was plucked from thin air by one expert and repeated by everyone else! Also if there is existing data dropping index and constraints will be a problem if there are user actively using the system(s) the database backs.

Of course if time allows and you have a machine you can run tests on before importing the data in production, you could run your own benchmarks on the process each way around.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.