1

Summary

My company recently deployed mysql innodb cluster as a backend database system using docker. To be exact, the cluster was deployed on Oct 3rd, so it has been almost two months. Recently, about 2 weeks ago, we observed a slowly decreasing in memory usage on our cluster servers, all from mysqld. Not sure why this would happen, below are the information about our production settings.

We also have zfs installed on our production servers as a backup and recovery solution

Facts
- RAM is decreasing on all three nodes about 250MB~ a day
- 3 servers
- each server has 68G of RAM

Cluster setup - 3 nodes
- single primary

Environment Setup
- mysql version: 5.7.19
- docker: 1.11
- container OS: debian:8.3
- host OS:
- Operating System: CentOS Linux 7 (Core)
- CPE OS Name: cpe:/o:centos:centos:7
- Kernel: Linux 3.10.0-514.26.2.el7.x86_64
- Architecture: x86-64
- zfs: ZFS: Loaded module v0.7.1-1, ZFS pool version 5000, ZFS filesystem version 5

mysqld.cnf (same on all three servers)

[mysqld] 

pid-file = /var/run/mysqld/mysqld.pid  
socket = /var/run/mysqld/mysqld.sock  
datadir = /data/mysql/data   
symbolic-links=0  

gtid-mode = OFF  
enforce-gtid-consistency = ON  
binlog-checksum = NONE  
log-bin = /data/mysql/bin-log/1  
server-id = 100  
master-info-repository = TABLE  
relay-log-info-repository = TABLE  
log-slave-updates = ON  
transaction-write-set-extraction = XXHASH64  

innodb_buffer_pool_size = 8G 
skip_name_resolve = ON  

innodb_data_home_dir = /data/mysql/system/  
innodb_log_group_home_dir = /data/mysql/logs/  
innodb_doublewrite=off  
innodb_file_per_table=off  

ZFS Settings

zfs create -o compression=lz4 -o recordsize=128k -o primarycache=metadata  tank/mysql && \  
zfs set mountpoint=/services/lemonwire/orchard/data/mysql tank/mysql && \  
zfs create tank/mysql/data && \  
zfs create -o recordsize=16k tank/mysql/system && \  
zfs create tank/mysql/logs && \  
zfs create tank/mysql/bin-log  

snapshots  

zfs list -H -o name -t filesystem ${POOLS} | while read fs; do  
zfs snapshot -r "${fs}@${snap}"  
done  

Current Memory State

Server 1 (primary):

$ ps aux  

polkitd 151255 0.0 0.0 20144 1756 ? Ss Oct03 0:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mysqld  
polkitd 155477 42.8 57.8 40800012 37945140 ? Sl Oct03 31425:52 mysqld  

$ cat /proc/meminfo 

MemTotal: 65555696 kB  
MemFree: 17297404 kB  
MemAvailable: 17665336 kB  
Buffers: 171900 kB   
Cached: 656136 kB  
SwapCached: 0 kB  
Active: 38398360 kB  
Inactive: 439888 kB  
Active(anon): 38071836 kB  
Inactive(anon): 72160 kB  
Active(file): 326524 kB  
Inactive(file): 367728 kB  
Unevictable: 21544 kB  
Mlocked: 21544 kB  
SwapTotal: 16777212 kB  
SwapFree: 16777212 kB  
Dirty: 8 kB  
Writeback: 0 kB  
AnonPages: 38031756 kB  
Mapped: 98692 kB  
Shmem: 126340 kB  
Slab: 2101840 kB  
SReclaimable: 116228 kB  
SUnreclaim: 1985612 kB  
KernelStack: 11296 kB  
PageTables: 82188 kB  
NFS_Unstable: 0 kB  
Bounce: 0 kB  
WritebackTmp: 0 kB  
CommitLimit: 49555060 kB  
Committed_AS: 10545048 kB  
VmallocTotal: 34359738367 kB  
VmallocUsed: 1221784 kB  
VmallocChunk: 34324739108 kB  
HardwareCorrupted: 0 kB  
AnonHugePages: 37781504 kB  
HugePages_Total: 0  
HugePages_Free: 0   
HugePages_Rsvd: 0  
HugePages_Surp: 0  
Hugepagesize: 2048 kB  
DirectMap4k: 232140 kB  
DirectMap2M: 4782080 kB  
DirectMap1G: 63963136 kB  

Server 2 (secondary):

$ ps aux 

polkitd 151255 0.0 0.0 20144 1756 ? Ss Oct03 0:00 /bin/bash  /usr/local/bin/docker-entrypoint.sh mysqld  
polkitd 155477 42.8 57.8 40800012 37945140 ? Sl Oct03 31425:52 mysqld  

$ cat /proc/meminfo  

MemTotal: 65555696 kB  
MemFree: 14707936 kB  
MemAvailable: 15072196 kB   
Buffers: 175252 kB  
Cached: 657644 kB  
SwapCached: 0 kB  
Active: 38376748 kB  
Inactive: 427300 kB  
Active(anon): 38033584 kB   
Inactive(anon): 71308 kB   
Active(file): 343164 kB  
Inactive(file): 355992 kB  
Unevictable: 21608 kB  
Mlocked: 21608 kB  
SwapTotal: 16777212 kB  
SwapFree: 16777212 kB  
Dirty: 0 kB  
Writeback: 0 kB  
AnonPages: 37993088 kB  
Mapped: 103960 kB  
Shmem: 126296 kB  
Slab: 2978004 kB  
SReclaimable: 99080 kB  
SUnreclaim: 2878924 kB  
KernelStack: 12688 kB  
PageTables: 81040 kB  
NFS_Unstable: 0 kB  
Bounce: 0 kB  
WritebackTmp: 0 kB  
CommitLimit: 49555060 kB  
Committed_AS: 10528076 kB  
VmallocTotal: 34359738367 kB  
VmallocUsed: 1227144 kB  
VmallocChunk: 34324725020 kB  
HardwareCorrupted: 0 kB  
AnonHugePages: 22595584 kB  
HugePages_Total: 0  
HugePages_Free: 0  
HugePages_Rsvd: 0  
HugePages_Surp: 0  
Hugepagesize: 2048 kB  
DirectMap4k: 248524 kB  
DirectMap2M: 5814272 kB  
DirectMap1G: 62914560 kB  

Server 3 (secondary):

$ ps aux

polkitd 44072 0.0 0.0 20120 1652 ? Ss Nov09 0:00 /bin/bash  /usr/local/bin/docker-entrypoint.sh mysqld  
polkitd 44687 11.9 33.3 24242188 21854496 ? Sl Nov09 2361:58 mysqld  

$ cat /proc/meminfo 

MemTotal: 65555696 kB  
MemFree: 31669528 kB  
MemAvailable: 31629280 kB  
Buffers: 103536 kB  
Cached: 372840 kB  
SwapCached: 7568 kB  
Active: 22103988 kB  
Inactive: 299484 kB  
Active(anon): 21976708 kB 
Inactive(anon): 151872 kB  
Active(file): 127280 kB  
Inactive(file): 147612 kB  
Unevictable: 21560 kB  
Mlocked: 21560 kB  
SwapTotal: 16777212 kB  
SwapFree: 16753276 kB  
Dirty: 0 kB  
Writeback: 0 kB  
AnonPages: 21941816 kB  
Mapped: 90424 kB  
Shmem: 194020 kB  
Slab: 2818220 kB  
SReclaimable: 138592 kB  
SUnreclaim: 2679628 kB  
KernelStack: 13488 kB  
PageTables: 49272 kB  
NFS_Unstable: 0 kB  
Bounce: 0 kB  
WritebackTmp: 0 kB  
CommitLimit: 49555060 kB  
Committed_AS: 10982096 kB  
VmallocTotal: 34359738367 kB  
VmallocUsed: 1182512 kB  
VmallocChunk: 34324730200 kB  
HardwareCorrupted: 0 kB  
AnonHugePages: 21760000 kB  
HugePages_Total: 0  
HugePages_Free: 0  
HugePages_Rsvd: 0  
HugePages_Surp: 0  
Hugepagesize: 2048 kB  
DirectMap4k: 262860 kB  
DirectMap2M: 6848512 kB  
DirectMap1G: 61865984 kBmonwire 2  
  • Did you find a cure? Did it eat up all RAM? What is the state? – Rick James May 19 '18 at 2:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.