Skip to main content
Source Link

Improving sql query execution time for 3 billion rows table

I have a 2+ billion row mysql table. I am also writing 1 million rows each day. The table contains some large columns like varchar(255) (to hold long urls)

In order to perform analytics on this table I created 5 specific indices which really speed up the execution time. (From 25+ minutes to 2 minutes for some queries).

The problem still persists that 2 minutes is a lot of time for just one query. I would like to run multiple queries for analytics and reporting.

Also, this table is increasing rapidly on a daily basis and I am pretty sure the indexes are as optimised as they can be.

Is this the point where clustering would solve my issues? i.e. is it unusual that a table this size still runs on a single sql node?

or is it still possible to run queries in such large table in milliseconds?

An example query of mine is:

SELECT name, url, SUM(visits), AVG(price), AVG(loc) FROM mytable
    WHERE sname IN ('white') AND usage IN ('three') AND date BETWEEN '2001-01-01' AND '2003-03-10'
    GROUP BY name, url ORDER BY SUM(visits);

I am new to clustering and HPC in general any advice on what I should do here is appreciated.