If your PK is not AUTO_INCREMENT or data in csv file not sorted on PK than it might be affecting performance of dataload. Since table in MySQL is an index hence all data is stored in sorted order, if PK value is not on AUTO_INCREMENT than MySQL has to do lot of data shifting to get the data stored in sorted order. This is the reason for slower data load when table size start growing.
I am loading 91GB csv file with PK on AUTO_INCREMENT using LOAD DATA INFILE and I do not see any drop in my throughput. I am getting 145K140K to 150K145K inserts per seconds. Using Percona MySQL 5.6.38