Skip to main content
Database Administrators

Return to Answer

added 698 characters in body
Source Link
a_vlad
a_vlad
  • 3.7k
  • 2
  • 13
  • 17

What parameters are You use for Talend tMySQLOutput? What MySQL settings? It could not be faster then INFILE, but still really depend from settings.

You can post Your question on TelndForge forum with screen short of Job components, will try to help

For example one of my regular Talend-MySQL Jobs (billing prepare) - transfer from->to mysql 2M of records, 400Mb, it take 6-15 minutes depending from servers (we have 2 configuration)

edit, because always better 1 time to show, than 100 times explain

2.3M rows, 300Mb file, speed of tMySQL with 100 rows per insert: 100 rows per insert

same with 10 000 rows per insert 10 000 rows per insert

tBulkOutputExec: twice faster than previous

but then total time: original 10 000 rows: original

bulk: bulk just because after execution it wait commit transaction, single transaction

What parameters are You use for Talend tMySQLOutput? What MySQL settings? It could not be faster then INFILE, but still really depend from settings.

You can post Your question on TelndForge forum with screen short of Job components, will try to help

For example one of my regular Talend-MySQL Jobs (billing prepare) - transfer from->to mysql 2M of records, 400Mb, it take 6-15 minutes depending from servers (we have 2 configuration)

What parameters are You use for Talend tMySQLOutput? What MySQL settings? It could not be faster then INFILE, but still really depend from settings.

You can post Your question on TelndForge forum with screen short of Job components, will try to help

For example one of my regular Talend-MySQL Jobs (billing prepare) - transfer from->to mysql 2M of records, 400Mb, it take 6-15 minutes depending from servers (we have 2 configuration)

edit, because always better 1 time to show, than 100 times explain

2.3M rows, 300Mb file, speed of tMySQL with 100 rows per insert: 100 rows per insert

same with 10 000 rows per insert 10 000 rows per insert

tBulkOutputExec: twice faster than previous

but then total time: original 10 000 rows: original

bulk: bulk just because after execution it wait commit transaction, single transaction

Source Link
a_vlad
a_vlad
  • 3.7k
  • 2
  • 13
  • 17

What parameters are You use for Talend tMySQLOutput? What MySQL settings? It could not be faster then INFILE, but still really depend from settings.

You can post Your question on TelndForge forum with screen short of Job components, will try to help

For example one of my regular Talend-MySQL Jobs (billing prepare) - transfer from->to mysql 2M of records, 400Mb, it take 6-15 minutes depending from servers (we have 2 configuration)