1

Hello we use nextcloud hand use as backend a mysql-db on Debian 10. I run ./mysqltuner.pl which said

Temporary tables created on disk: 99% (48K on disk / 48K total) (I recently restart the mysql-server)

So far as I know disk is slower than RAM, so why is mysql doing this and how can I stop that? Here are some usefull things that might be helpfull

[mysqld]
performance_schema=ON
slow_query_log
slow_query_log_file=/var/log/mysql/mariadb-slow.log
long_query_time=5
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /nfs_nc_storage/mariadb
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
bind-address            = 127.0.0.1
innodb_buffer_pool_size = 16G
key_buffer_size         = 256M
max_allowed_packet      = 64M
thread_stack            = 192K
thread_cache_size       = 8
myisam_recover_options  = BACKUP
max_connections        = 150
table_cache            = 512
innodb_log_file_size = 1G
innodb_large_prefix=true
innodb_file_format=barracuda
innodb_file_per_table=1
query_cache_limit       = 6M
query_cache_size        = 36M
tmp_table_size = 512M
max_heap_table_size = 512M
join_buffer_size = 32M
log_error = /var/log/mysql/error.log
expire_logs_days        = 10
max_binlog_size   = 100M
sort_buffer_size = 32M
character-set-server  = utf8mb4
collation-server      = utf8mb4_general_ci

SELECT * FROM performance_schema.events_statements_summary_by_digest\G
[...]
*************************** 153. row ***************************
                SCHEMA_NAME: wolke7
                     DIGEST: bc0ef4c5d3c501a96bffdd90fe26b557
                DIGEST_TEXT: SELECT `share` . `id` , `share` . `share_type` , `share` . `share_with` , `share` . `uid_owner` , `share` . `uid_initiator` , `share` . `parent` , `share` . `item_type` , `share` . `item_source` , `share` . `item_target` , `share` . `file_source` , `share` . `file_target` , `share` . `pe
rmissions` , `share` . `stime` , `share` . `accepted` , `share` . `expiration` , `share` . `token` , `share` . `mail_send` , `share_circle` . `unique_id` AS `share_circle_unique_id` , `share_circle` . `name` AS `share_circle_name` , `share_circle` . `display_name` AS `share_circle_display_name` , `share_circle` . `s
anitized_name` AS `share_circle_sanitized_name` , `share_circle` . `source` AS `share_circle_source` , `share_circle` . `description` AS `share_circle_description` , `share_circle` . `settings` AS `share_circle_settings` , `share_circle` . `config` AS `share_circle_config` , `share_circle` . `contact_addressbook` AS
 `share_circle_contact_addressbook` , `share_circle` 
                 COUNT_STAR: 29025
             SUM_TIMER_WAIT: 115824079611000
             MIN_TIMER_WAIT: 771567000
             AVG_TIMER_WAIT: 3990493000
             MAX_TIMER_WAIT: 298412894000
              SUM_LOCK_TIME: 20403574000000
                 SUM_ERRORS: 0
               SUM_WARNINGS: 0
          SUM_ROWS_AFFECTED: 0
              SUM_ROWS_SENT: 0
          SUM_ROWS_EXAMINED: 85238
SUM_CREATED_TMP_DISK_TABLES: 29025
     SUM_CREATED_TMP_TABLES: 29025
       SUM_SELECT_FULL_JOIN: 0
 SUM_SELECT_FULL_RANGE_JOIN: 0
           SUM_SELECT_RANGE: 31
     SUM_SELECT_RANGE_CHECK: 0
            SUM_SELECT_SCAN: 4
      SUM_SORT_MERGE_PASSES: 0
             SUM_SORT_RANGE: 0
              SUM_SORT_ROWS: 0
              SUM_SORT_SCAN: 29025
          SUM_NO_INDEX_USED: 4
     SUM_NO_GOOD_INDEX_USED: 0
                 FIRST_SEEN: 2021-12-01 11:16:52
                  LAST_SEEN: 2021-12-01 12:21:23
[...]
*************************** 205. row ***************************
                SCHEMA_NAME: wolke7
                     DIGEST: 0a264a8134ee57157b1965548786ccdc
                DIGEST_TEXT: SELECT `share` . `id` , `share` . `share_type` , `share` . `share_with` , `share` . `uid_owner` , `share` . `uid_initiator` , `share` . `parent` , `share` . `item_type` , `share` . `item_source` , `share` . `item_target` , `share` . `file_source` , `share` . `file_target` , `share` . `pe
rmissions` , `share` . `stime` , `share` . `accepted` , `share` . `expiration` , `share` . `token` , `share` . `mail_send` , `share_circle_owner` . `circle_id` AS `share_circle_owner_circle_id` , `share_circle_owner` . `member_id` AS `share_circle_owner_member_id` , `share_circle_owner` . `single_id` AS `share_circl
e_owner_single_id` , `share_circle_owner` . `user_id` AS `share_circle_owner_user_id` , `share_circle_owner` . `instance` AS `share_circle_owner_instance` , `share_circle_owner` . `user_type` AS `share_circle_owner_user_type` , `share_circle_owner` . `level` AS `share_circle_owner_level` , `share_circle_owner` . `st
atus` AS `share_circle_owner_status` , 
                 COUNT_STAR: 5183
             SUM_TIMER_WAIT: 9465924638000
             MIN_TIMER_WAIT: 941140000
             AVG_TIMER_WAIT: 1826340000
             MAX_TIMER_WAIT: 16988737000
              SUM_LOCK_TIME: 1691111000000
                 SUM_ERRORS: 0
               SUM_WARNINGS: 0
          SUM_ROWS_AFFECTED: 0
              SUM_ROWS_SENT: 0
          SUM_ROWS_EXAMINED: 37606
SUM_CREATED_TMP_DISK_TABLES: 5183
     SUM_CREATED_TMP_TABLES: 5183
       SUM_SELECT_FULL_JOIN: 0
 SUM_SELECT_FULL_RANGE_JOIN: 0
           SUM_SELECT_RANGE: 2560
     SUM_SELECT_RANGE_CHECK: 0
            SUM_SELECT_SCAN: 0
      SUM_SORT_MERGE_PASSES: 0
             SUM_SORT_RANGE: 0
              SUM_SORT_ROWS: 0
              SUM_SORT_SCAN: 5183
          SUM_NO_INDEX_USED: 0
     SUM_NO_GOOD_INDEX_USED: 0
                 FIRST_SEEN: 2021-12-01 11:16:53
                  LAST_SEEN: 2021-12-01 12:21:22
[...]

SHOW global STATUS LIKE 'created_tmp%tables';
+-------------------------+-------+
| Variable_name           | Value |
+-------------------------+-------+
| Created_tmp_disk_tables | 47158 |
| Created_tmp_tables      | 47450 |
+-------------------------+-------+
2 rows in set (0.001 sec)

It is a VM with 12Cores and 32GB RAM, there is a running apache2

Thank you very much in advance!

EDIT: here is the output of tuning-primer and mysqltuner:

./tuning-primer.sh 
 
 -- MYSQL PERFORMANCE TUNING PRIMER --
      - By: Matthew Montgomery -

MySQL Version 10.3.31-MariaDB-0+deb10u1-log x86_64

Uptime = 12 days 23 hrs 22 min 14 sec
Avg. qps = 657
Total Questions = 736821161
Threads Connected = 6
 [...]

SLOW QUERIES
The slow query log is enabled.
Current long_query_time = 5.000000 sec.
Since startup, 383 out of 736821176 queries have taken longer than <long_query_time-when-they-were-executed> to complete.
Your long_query_time may be too high, I typically set this under 5 sec.

BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/10.3/en/point-in-time-recovery.html

WORKER THREADS
Current thread_cache_size = 8
Current threads_cached = 0
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine

MAX CONNECTIONS
Current max_connections = 150
Current threads_connected = 6
Historic max_used_connections = 73
The number of used connections is 48% of the configured maximum.
Your max_connections variable seems to be fine.

INNODB STATUS
Current InnoDB index space = 7.87 G
Current InnoDB data space = 8.96 G
Current InnoDB buffer pool free = 11 %
Current innodb_buffer_pool_size = 16.00 G
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory

MEMORY USAGE
Max Memory Ever Allocated : 27.74 G
Configured Max Per-thread Buffers : 23.51 G
Configured Max Global Buffers : 16.30 G
Configured Max Memory Limit : 39.82 G
Physical Memory : 31.36 G

Max memory limit exceeds 90% of physical memory

KEY BUFFER
Current MyISAM index space = 129 K
Current key_buffer_size = 256 M
Key cache miss rate is 1 : 139
Key buffer free ratio = 81 %
Your key_buffer_size seems to be fine

QUERY CACHE
Query cache is enabled
Current query_cache_size = 36 M
Current query_cache_used = 10 M
Current query_cache_limit = 6 M
Current Query cache Memory fill ratio = 27.94 %
Current query_cache_min_res_unit = 4 K
MySQL won't cache query results that are larger than query_cache_limit in size

SORT OPERATIONS
Current sort_buffer_size = 128 M
Current read_rnd_buffer_size = 256 K
Sort buffer seems to be fine

JOINS
Current join_buffer_size = 32.00 M
You have had 1269 queries where a join could not use an index properly
You have had 9928 joins without keys that check for key usage after each row
join_buffer_size >= 4 M
This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.

OPEN FILES LIMIT
Current open_files_limit = 8373 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.                                                                                                                                                                                          
Your open_files_limit value seems to be fine

TABLE CACHE
Current table_open_cache = 512 tables
Current table_definition_cache = 400 tables
You have a total of 198 tables
You have 376 open tables.
The table_cache value seems to be fine

TEMP TABLES
Current max_heap_table_size = 128 M
Current tmp_table_size = 128 M
Of 5427694 temp tables, 49% were created on disk
Perhaps you should increase your tmp_table_size and/or max_heap_table_size
to reduce the number of disk-based temporary tables
Note! BLOB and TEXT columns are not allow in memory tables.
If you are using these columns raising these values might not impact your                                                                                                                                                                    
ratio of on disk temp tables.                                                                                                                                                                                                                

TABLE SCANS
Current read_buffer_size = 128 K
Current table scan ratio = 16 : 1
read_buffer_size seems to be fine

TABLE LOCKING
Current Lock Wait ratio = 0 : 736822868
Your table locking seems to be fine

           


./mysqltuner.pl 
 >>  MySQLTuner 1.8.5 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.pl/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials from Debian maintenance account.
[OK] Currently running supported MySQL version 10.3.31-MariaDB-0+deb10u1-log
[OK] Operating on 64-bit architecture
 
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE 
[--] Data in InnoDB tables: 16.8G (Tables: 115)
[OK] Total fragmented tables: 0
 
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 12d 23h 22m 43s (736M q [657.360 qps], 8M conn, TX: 1796G, RX: 303G)
[--] Reads / Writes: 62% / 38%
[--] Binary logging is disabled
[--] Physical Memory     : 31.4G
[--] Max MySQL memory    : 49.5G
[--] Other process memory: 0B
[--] Total buffers: 16.6G global + 224.6M per thread (150 max threads)
[--] P_S Max memory usage: 103M
[--] Galera GCache Max memory usage: 0B
[!!] Maximum reached memory usage: 32.7G (104.15% of installed RAM)
[!!] Maximum possible memory usage: 49.5G (157.99% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (383/736M)
[OK] Highest usage of available connections: 48% (73/150)
[OK] Aborted connections: 0.00%  (0/8461870)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[OK] Query cache efficiency: 36.1% (250M cached / 694M selects)
[!!] Query cache prunes per day: 1138274
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 7M sorts)
[!!] Joins performed without indexes: 11197
[!!] Temporary tables created on disk: 98% (5M on disk / 5M total)
[--] Thread cache not used with thread_handling=pool-of-threads
[OK] Table cache hit rate: 99% (574M hits / 574M requests)
[OK] table_definition_cache(400) is upper than number of tables(276)
[OK] Open file limit used: 0% (60/8K)
[OK] Table locks acquired immediately: 100% (531 immediate / 531 locks)
 
-------- Performance schema ------------------------------------------------------------------------
[--] Memory used by P_S: 103.8M
[--] Sys schema isn't installed.
 
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 12 thread(s).
[--] Using default value is good enough for your version (10.3.31-MariaDB-0+deb10u1-log)
 
-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.2% (48M used / 268M cache)
[OK] Key buffer size / total MyISAM indexes: 256.0M/129.0K
[OK] Read Key buffer hit rate: 99.3% (278 cached / 2 reads)
[!!] Write Key buffer hit rate: 26.3% (365 cached / 96 writes)
 
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 16.0G/16.8G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (12.5 %): 1.0G * 2/16.0G should be equal to 25%
[!!] InnoDB buffer pool instances: 8
[--] Number of InnoDB Buffer Pool Chunk : 128 for 8 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 100.00% (20503518990 hits/ 20504286527 total)
[OK] InnoDB Write log efficiency: 93.13% (3088225654 hits/ 3315985043 total)
[OK] InnoDB log waits: 0.00% (0 waits / 227759389 writes)
 
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/0B
[!!] Aria pagecache hit rate: 87.8% (7M cached / 904K reads)
 
 
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Reduce your overall MySQL memory footprint for system stability
    Dedicate this server to your database for highest performance.
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
             See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
             (specially the conclusions at the bottom of the page).
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Consider installing Sys schema from https://github.com/mysql/mysql-sys for MySQL
    Consider installing Sys schema from https://github.com/FromDual/mariadb-sys for MariaDB
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_size (> 36M)
    join_buffer_size (> 32.0M, or always use indexes with JOINs)
    tmp_table_size (> 128M)
    max_heap_table_size (> 128M)
    innodb_buffer_pool_size (>= 16.8G) if possible.
    innodb_log_file_size should be (=2G) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
    innodb_buffer_pool_instances(=16)


11
  • Here's your problem: sort_buffer_size = 32M.
    – mustaccio
    Dec 1 '21 at 13:10
  • ok, so what do you recommend? because mysqltuner said its fine
    – colttt
    Dec 1 '21 at 13:14
  • 1
    Show us the query and SHOW CREATE TABLE. Adding a better index and/or reformulating the query are the best ways to speed up a query. (In some cases "temp table on disk" is not the real problem.)
    – Rick James
    Dec 1 '21 at 19:49
  • 1
    The query you showed is truncated.
    – Rick James
    Dec 1 '21 at 19:51
  • 1
    @WilsonHauck done and thanks for help!
    – colttt
    Dec 14 '21 at 13:34
0

Your query summary shows:

SUM_CREATED_TMP_DISK_TABLES: 29025
     SUM_CREATED_TMP_TABLES: 29025
...
              SUM_SORT_SCAN: 29025

and

SUM_CREATED_TMP_DISK_TABLES: 5183
     SUM_CREATED_TMP_TABLES: 5183
       SUM_SELECT_FULL_JOIN: 0
...
              SUM_SORT_SCAN: 5183

This means that your sorts spill, creating temporary tables on disk. Given the puny size of your sort buffer (sort_buffer_size = 32M) it's not surprising.

Start doubling your sort buffer size until the sort spills go away (or you hit memory constraints).

Also, do not run the database and web servers on the same machine -- it's a Bad Idea™.

2
  • Sorry, the sort buffer may spill over to use disk storage, but that is not technically a temporary table, and does not increment the temp table count. It increments the sort_merge_passes status variable when this happens. Dec 1 '21 at 15:02
  • sort_buffer_size = 32M is rather large -- 95% of servers set it lower than that. Do not set it bigger than, say, 1% of RAM size.
    – Rick James
    Dec 1 '21 at 19:57
0

The way to avoid temp tables on disk has to do with query design, not so much with tuning options.

You should read https://dev.mysql.com/doc/refman/8.0/en/internal-temporary-tables.html to understand all the details about how MySQL uses temp tables during queries. This page describes many types of queries that produce temp tables.

If a temp table is larger than tmp_table_size or max_heap_table_size, whichever is smaller, then the temp table must be stored on disk (in your case both are 512MB). This happens if your query results are large. That is, if they have many rows or the rows include large columns.

Regardless of the size of the query result, in a few cases temp tables must be stored on disk anyway. For example, if your query includes a BLOB or TEXT column, or a VARCHAR longer than 512 characters.

Sometimes queries can be redesigned to help avoid this problem. Bulky columns may not really be needed for a given query result, so you should change the ubiquitous SELECT * to specify only the columns you need. Or indexes can help to reduce the number of rows examined, and therefore the temp table will be smaller and fit within your RAM configuration. Or indexes may be used to avoid the temp table altogether.

To find out which queries are using temp tables, you can log them in the slow query log. MySQL 8.0 can optionally output extra information in this log to report statistics about the query, including whether it created temp tables in RAM or temp tables on disk.

If you use Percona Server or MariaDB, they have a similar feature to output optional extra information in the slow query log in older versions of the product.

0

Rate Per Second = RPS

Suggestions to consider improving your 10.3.31 MariaDB instance performance

max_heap_table_size=512M  # from 128M per tuning-primer
tmp_table_size=512M  # from 128M per tuning-primer to reduce 49% created on disk
key_buffer_size=128M  # from 256M per tuning-primer to reduce buffer free ratio of 81 percent
query_cache_min_res_unit=512  # from 4K for higher density use of QC
sort_buffer_size=2M  # from 128M a per-connection RAM request
join_buffer_size=2M  # from 32M a per-connection RAM request 
innodb_buffer_pool_size=22G  # from 16G to reduce innodb_data_reads RPS

The are many other opportunities for tuning. Please view profile and get in touch for additional suggestions.

1
  • @colttt Applied suggestions? Results positive or negative? Dec 25 '21 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.