I have a 2+ billion row mysql table. I am also writing 1 million rows each day. The table contains some large columns like varchar(255)
(to hold long urls)
In order to perform analytics on this table I created 5 specific indices which really speed up the execution time. (From 25+ minutes to 2 minutes for some queries).
The problem still persists that 2 minutes is a lot of time for just one query. I would like to run multiple queries for analytics and reporting.
Also, this table is increasing rapidly on a daily basis and I am pretty sure the indexes are as optimised as they can be.
Is this the point where clustering would solve my issues? i.e. is it unusual that a table this size still runs on a single sql node?
or is it still possible to run queries in such large table in milliseconds?
An example query of mine is:
SELECT name, url, SUM(visits), AVG(price), AVG(loc) FROM mytable
WHERE sname IN ('white') AND usage IN ('three') AND date BETWEEN '2001-01-01' AND '2003-03-10'
GROUP BY name, url ORDER BY SUM(visits);
I am new to clustering and HPC in general any advice on what I should do here is appreciated.