Skip to main content
added 1076 characters in body
Source Link
Rick James
  • 79.4k
  • 5
  • 51
  • 117

Summarize the day's data each night. Then run the "report" against that table.

Note that a SUM is the SUM of sums; AVG is (SUM of sums / SUM of counts).

The summary table would probably have (in this order)

PRIMARY KEY(sname, usage, url, date)

plus columns like

sum_visits,
sum_price,
count_price,
sum_loc

More discussion: http://mysql.rjweb.org/doc.php/summarytables

I would expect this approach to turn "minutes" into "seconds", but not into "milliseconds". If you want milliseconds, we need to discuss more details; it may be possible.

(Initialization would take a long time to go through the existing 3B rows, but the nightly summarization would be much faster.)

Once you have built the summary table(s), you can and should drop the unnecessary indexes; they are bogging down that big Fact table.

If you plan to delete "old" data eventually (after, say, 90 days), you should add Partitioning now, before the table gets any bigger. See http://mysql.rjweb.org/doc.php/partitionmaint (The point is that DELETE of a million rows is very costly, whereas DROP PARTITION is virtually instant.)

Be careful to have the right datatypes to conserve space. (Smaller --> a little faster).

Probably a single node is fine and even has some room for growth.

12 inserts/second is not very fast, even on HDD. It's no problem on SSD.

While I recommend a nightly summarization, it is also possible to do the summarization as each row is inserted -- using IODKU. This would push HDD to the limit, but still leaving breathing room on SSD. And it would allow "realtime" reports.

Does your data come in bursts? Does it come through a single client? Can it be batched (multiple rows at once)? Some form of batching can stretch the 12/sec to a much higher throughput.

Summarize the day's data each night. Then run the "report" against that table.

Note that a SUM is the SUM of sums; AVG is (SUM of sums / SUM of counts).

The summary table would probably have

PRIMARY KEY(sname, usage, url, date)

plus columns like

sum_visits,
sum_price,
count_price,
sum_loc

More discussion: http://mysql.rjweb.org/doc.php/summarytables

I would expect this approach to turn "minutes" into "seconds", but not into "milliseconds".

(Initialization would take a long time to go through the existing 3B rows, but the nightly summarization would be much faster.)

Once you have built the summary table(s), you can and should drop the unnecessary indexes; they are bogging down that big Fact table.

Summarize the day's data each night. Then run the "report" against that table.

Note that a SUM is the SUM of sums; AVG is (SUM of sums / SUM of counts).

The summary table would probably have (in this order)

PRIMARY KEY(sname, usage, url, date)

plus columns like

sum_visits,
sum_price,
count_price,
sum_loc

More discussion: http://mysql.rjweb.org/doc.php/summarytables

I would expect this approach to turn "minutes" into "seconds", but not into "milliseconds". If you want milliseconds, we need to discuss more details; it may be possible.

(Initialization would take a long time to go through the existing 3B rows, but the nightly summarization would be much faster.)

Once you have built the summary table(s), you can and should drop the unnecessary indexes; they are bogging down that big Fact table.

If you plan to delete "old" data eventually (after, say, 90 days), you should add Partitioning now, before the table gets any bigger. See http://mysql.rjweb.org/doc.php/partitionmaint (The point is that DELETE of a million rows is very costly, whereas DROP PARTITION is virtually instant.)

Be careful to have the right datatypes to conserve space. (Smaller --> a little faster).

Probably a single node is fine and even has some room for growth.

12 inserts/second is not very fast, even on HDD. It's no problem on SSD.

While I recommend a nightly summarization, it is also possible to do the summarization as each row is inserted -- using IODKU. This would push HDD to the limit, but still leaving breathing room on SSD. And it would allow "realtime" reports.

Does your data come in bursts? Does it come through a single client? Can it be batched (multiple rows at once)? Some form of batching can stretch the 12/sec to a much higher throughput.

Source Link
Rick James
  • 79.4k
  • 5
  • 51
  • 117

Summarize the day's data each night. Then run the "report" against that table.

Note that a SUM is the SUM of sums; AVG is (SUM of sums / SUM of counts).

The summary table would probably have

PRIMARY KEY(sname, usage, url, date)

plus columns like

sum_visits,
sum_price,
count_price,
sum_loc

More discussion: http://mysql.rjweb.org/doc.php/summarytables

I would expect this approach to turn "minutes" into "seconds", but not into "milliseconds".

(Initialization would take a long time to go through the existing 3B rows, but the nightly summarization would be much faster.)

Once you have built the summary table(s), you can and should drop the unnecessary indexes; they are bogging down that big Fact table.