I'm bulk loading a few thousand CSVs totaling ~260GB and ~4.5 billion rows to MySQL using
mysqlimport calls on each CSV executed within a bash script. The MySQL installation and the data itself are on an external drive as I didn't have sufficient disk space on my machine to accommodate. My first attempt seemed to stall after a certain juncture (~24h), so I created another table without the indexes to see if processing sped up some (figured it was slowing things down to have to reindex the table after loading each CSV).
I'm now about 36h in and a little over 50% of the way complete, but stumped as to how I could speed up processing (or if it's even possible). Is there a better way to go about loading this much data to a MySQL table.
Edit: Processing seems to be slowing as it goes. Roughly 50% of data loaded in the first 36h, only an additional ~10% in the 24h since.
Edit2: I've updated
innodb_buffer_pool_size to 4x its default value (128MB --> 512MB) and loading appears to be humming along--a single file was taking 15min+ to load prior to tuning, individual files are loading within ~1min now. There's been a corresponding slight increase in RAM usage (~4.5G to ~5.6G).