Optimize MySQL Server Performance

Question

I have a VPS Server at Vultr with 8GB Ram and 4vCPUs and am looking to get optimized mysql settings. I have a DB of around 170 InnoDB tables, but they mostly don't contain a lot of data. There are 2 MyISAM tables however with 150k rows each which I would like to increase to 350k.

The challenge here is that I do geo calculations in the table with long/lat data for every single calculation it has to go through the full 150k rows, determine which are the closest (within 5 miles) of distance and than give out only that data. This is obviously very resource hungry. Therefor increasing from 150k rows to 200k rows means the query takes 1sec longer while increasing to 350k rows takes an additional 3-4sec per query. I can't update the queries itself at this stage, but can only work on improving mysql as much as possible.

I already started updating my MySQL settings and greatly improved the performance. Here is the output from tuning-primer:

---

MySQL Version 5.5.55-0ubuntu0.14.04.1 x86_64

Uptime = 2 days 12 hrs 1 min 56 sec Avg. qps = 61 Total Questions = 2665033 Threads Connected = 3

SLOW QUERIES The slow query log is NOT enabled. Current long_query_time = 10.000000 sec. You have 0 out of 2665054 that take longer than 10.000000 sec. to complete Your long_query_time seems to be fine

BINARY UPDATE LOG The binary update log is NOT enabled. You will not be able to do point in time recovery See http://dev.mysql.com/doc/refman/5.5/en/point-in-time-recovery.html

WORKER THREADS Current thread_cache_size = 8 Current threads_cached = 5 Current threads_per_sec = 0 Historic threads_per_sec = 0 Your thread_cache_size is fine

MAX CONNECTIONS Current max_connections = 151 Current threads_connected = 3 Historic max_used_connections = 9 The number of used connections is 5% of the configured maximum. You are using less than 10% of your configured max_connections. Lowering max_connections could help to avoid an over-allocation of memory See "MEMORY USAGE" section to make sure you are not over-allocating

INNODB STATUS Current InnoDB index space = 35 M Current InnoDB data space = 56 M Current InnoDB buffer pool free = 25 % Current innodb_buffer_pool_size = 128 M Depending on how much space your innodb indexes take up it may be safe to increase this value to up to 2 / 3 of total system memory

MEMORY USAGE Max Memory Ever Allocated : 4.48 G Configured Max Per-thread Buffers : 1.53 G Configured Max Global Buffers : 4.39 G Configured Max Memory Limit : 5.92 G Physical Memory : 7.79 G Max memory limit seem to be within acceptable norms

KEY BUFFER Current MyISAM index space = 79 M Current key_buffer_size = 256 M Key cache miss rate is 1 : 9866 Key buffer free ratio = 79 % Your key_buffer_size seems to be fine

QUERY CACHE Query cache is enabled Current query_cache_size = 4.00 G Current query_cache_used = 889 M Current query_cache_limit = 4 M Current Query cache Memory fill ratio = 21.71 % Current query_cache_min_res_unit = 4 K Your query_cache_size seems to be too high. Perhaps you can use these resources elsewhere MySQL won't cache query results that are larger than query_cache_limit in size

SORT OPERATIONS Current sort_buffer_size = 2 M Current read_rnd_buffer_size = 256 K Sort buffer seems to be fine

JOINS Current join_buffer_size = 4.00 M You have had 72592 queries where a join could not use an index properly join_buffer_size >= 4 M This is not advised You should enable "log-queries-not-using-indexes" Then look for non indexed joins in the slow query log.

OPEN FILES LIMIT Current open_files_limit = 2209 files The open_files_limit should typically be set to at least 2x-3x that of table_cache if you have heavy MyISAM usage. Your open_files_limit value seems to be fine

TABLE CACHE Current table_open_cache = 1024 tables Current table_definition_cache = 1024 tables You have a total of 702 tables You have 1024 open tables. Current table_cache hit rate is 55% , while 100% of your table cache is in use You should probably increase your table_cache

TEMP TABLES Current max_heap_table_size = 16 M Current tmp_table_size = 16 M Of 162009 temp tables, 1% were created on disk Created disk tmp tables ratio seems fine

TABLE SCANS Current read_buffer_size = 3 M Current table scan ratio = 66078 : 1 read_buffer_size seems to be fine

TABLE LOCKING Current Lock Wait ratio = 1 : 696683 Your table locking seems to be fine

---

Thanks for your help. If you need more information please ask! Would it help to upgrade to mysql 5.6 or 5.7?

Answer 1

Step 1 in solving lat/lng performance problems:

INDEX(latitude),
INDEX(longitude)

If that does not work adequately, then see http://mysql.rjweb.org/doc.php/latlng

For more discussion, please provide SHOW CREATE TABLE and the SELECT that is doing the table scan.

innodb_buffer_pool_size = 128 M -- the most important performance tuning is to set that to 70% of available RAM (assuming all tables are InnoDB).

Why do you still have 2 MyISAM tables? Let's talk about converting them.

query_cache_size = 4.00 G -- This is really bad. Don't set it to more than 50M !

Answer 2

1. Read performance mostly depends on having proper indexes on tables and properly designed queries. There is no substitute to this.
2. This article provides some pointers for optimizing performance of InnoDB https://www.percona.com/blog/2007/11/01/innodb-performance-optimization-basics/

Please also review https://stackoverflow.com/questions/45001764/should-mysql-take-this-long-or-is-configuration-wrong/45001812?noredirect=1#comment76977777_45001812

Write performance substantially depends on the two settings (faster writes with these, but there is a catch relating to the first one. You will read in the article).

set global innodb_flush_log_at_trx_commit=2
set global general-log=0

Answer 3

Partially a database, partially a programming issue.

I do geo calculations in the table with long/lat data for every single calculation it has to go through the full 150k rows, determine which are the closest (within 5 miles)

To do that efficiently, you need indices on Longitute and Latitude.

And then some reforulation - unless MySQL supports that out of the box (like SQL Server).

YOu do NOT look for anything within a 5 mile RADIUS. YOu look for anything in a BOX that is 5 mile north, south, east and west of the central point. THAT hits the database.

Then you filter the radios in a second step.

Note how suddenly you have a query that is solvable by one of the indices? You will only do the actual distance calculation for a very minor part of the data.

That said, why do you not just create a SPATIAL INDEX? MySQL 5.7 supports that according to the documentation. Use htep roper data types and you should be golden.