As the title suggests I'm moving around a lot of data. This is in an environment with a 24 core CPU and 2TB of M.2 storage.
My first question is: given I have 16GB of ram, does that mean it's impossible for me to load a 30GB file into SQL (as the entire thing must be stored in ram?)
The second question: This is a fresh table I'm loading into, so I have the luxury of dictating which engine I can use. I've read that InnoDB is better for writing (in which I'm doing alot of). However, MySQL manual says that the
CONCURRENCY clause will work on MyISAM. So which engine should I use?
Any other helpful advice is appreciated.
My mysqld conf:
innodb_buffer_pool_size = 12G
innodb_log_file_size = 4G
innodb_write_io_threads = 18