Summary
My company recently deployed mysql innodb cluster as a backend database system using docker. To be exact, the cluster was deployed on Oct 3rd, so it has been almost two months. Recently, about 2 weeks ago, we observed a slowly decreasing in memory usage on our cluster servers, all from mysqld. Not sure why this would happen, below are the information about our production settings.
We also have zfs installed on our production servers as a backup and recovery solution
Facts
- RAM is decreasing on all three nodes about 250MB~ a day
- 3 servers
- each server has 68G of RAM
Cluster setup
- 3 nodes
- single primary
Environment Setup
- mysql version: 5.7.19
- docker: 1.11
- container OS: debian:8.3
- host OS:
- Operating System: CentOS Linux 7 (Core)
- CPE OS Name: cpe:/o:centos:centos:7
- Kernel: Linux 3.10.0-514.26.2.el7.x86_64
- Architecture: x86-64
- zfs: ZFS: Loaded module v0.7.1-1, ZFS pool version 5000, ZFS filesystem version 5
mysqld.cnf (same on all three servers)
[mysqld]
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = /data/mysql/data
symbolic-links=0
gtid-mode = OFF
enforce-gtid-consistency = ON
binlog-checksum = NONE
log-bin = /data/mysql/bin-log/1
server-id = 100
master-info-repository = TABLE
relay-log-info-repository = TABLE
log-slave-updates = ON
transaction-write-set-extraction = XXHASH64
innodb_buffer_pool_size = 8G
skip_name_resolve = ON
innodb_data_home_dir = /data/mysql/system/
innodb_log_group_home_dir = /data/mysql/logs/
innodb_doublewrite=off
innodb_file_per_table=off
ZFS Settings
zfs create -o compression=lz4 -o recordsize=128k -o primarycache=metadata tank/mysql && \
zfs set mountpoint=/services/lemonwire/orchard/data/mysql tank/mysql && \
zfs create tank/mysql/data && \
zfs create -o recordsize=16k tank/mysql/system && \
zfs create tank/mysql/logs && \
zfs create tank/mysql/bin-log
snapshots
zfs list -H -o name -t filesystem ${POOLS} | while read fs; do
zfs snapshot -r "${fs}@${snap}"
done
Current Memory State
Server 1 (primary):
$ ps aux
polkitd 151255 0.0 0.0 20144 1756 ? Ss Oct03 0:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mysqld
polkitd 155477 42.8 57.8 40800012 37945140 ? Sl Oct03 31425:52 mysqld
$ cat /proc/meminfo
MemTotal: 65555696 kB
MemFree: 17297404 kB
MemAvailable: 17665336 kB
Buffers: 171900 kB
Cached: 656136 kB
SwapCached: 0 kB
Active: 38398360 kB
Inactive: 439888 kB
Active(anon): 38071836 kB
Inactive(anon): 72160 kB
Active(file): 326524 kB
Inactive(file): 367728 kB
Unevictable: 21544 kB
Mlocked: 21544 kB
SwapTotal: 16777212 kB
SwapFree: 16777212 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 38031756 kB
Mapped: 98692 kB
Shmem: 126340 kB
Slab: 2101840 kB
SReclaimable: 116228 kB
SUnreclaim: 1985612 kB
KernelStack: 11296 kB
PageTables: 82188 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49555060 kB
Committed_AS: 10545048 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 1221784 kB
VmallocChunk: 34324739108 kB
HardwareCorrupted: 0 kB
AnonHugePages: 37781504 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 232140 kB
DirectMap2M: 4782080 kB
DirectMap1G: 63963136 kB
Server 2 (secondary):
$ ps aux
polkitd 151255 0.0 0.0 20144 1756 ? Ss Oct03 0:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mysqld
polkitd 155477 42.8 57.8 40800012 37945140 ? Sl Oct03 31425:52 mysqld
$ cat /proc/meminfo
MemTotal: 65555696 kB
MemFree: 14707936 kB
MemAvailable: 15072196 kB
Buffers: 175252 kB
Cached: 657644 kB
SwapCached: 0 kB
Active: 38376748 kB
Inactive: 427300 kB
Active(anon): 38033584 kB
Inactive(anon): 71308 kB
Active(file): 343164 kB
Inactive(file): 355992 kB
Unevictable: 21608 kB
Mlocked: 21608 kB
SwapTotal: 16777212 kB
SwapFree: 16777212 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 37993088 kB
Mapped: 103960 kB
Shmem: 126296 kB
Slab: 2978004 kB
SReclaimable: 99080 kB
SUnreclaim: 2878924 kB
KernelStack: 12688 kB
PageTables: 81040 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49555060 kB
Committed_AS: 10528076 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 1227144 kB
VmallocChunk: 34324725020 kB
HardwareCorrupted: 0 kB
AnonHugePages: 22595584 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 248524 kB
DirectMap2M: 5814272 kB
DirectMap1G: 62914560 kB
Server 3 (secondary):
$ ps aux
polkitd 44072 0.0 0.0 20120 1652 ? Ss Nov09 0:00 /bin/bash /usr/local/bin/docker-entrypoint.sh mysqld
polkitd 44687 11.9 33.3 24242188 21854496 ? Sl Nov09 2361:58 mysqld
$ cat /proc/meminfo
MemTotal: 65555696 kB
MemFree: 31669528 kB
MemAvailable: 31629280 kB
Buffers: 103536 kB
Cached: 372840 kB
SwapCached: 7568 kB
Active: 22103988 kB
Inactive: 299484 kB
Active(anon): 21976708 kB
Inactive(anon): 151872 kB
Active(file): 127280 kB
Inactive(file): 147612 kB
Unevictable: 21560 kB
Mlocked: 21560 kB
SwapTotal: 16777212 kB
SwapFree: 16753276 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 21941816 kB
Mapped: 90424 kB
Shmem: 194020 kB
Slab: 2818220 kB
SReclaimable: 138592 kB
SUnreclaim: 2679628 kB
KernelStack: 13488 kB
PageTables: 49272 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49555060 kB
Committed_AS: 10982096 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 1182512 kB
VmallocChunk: 34324730200 kB
HardwareCorrupted: 0 kB
AnonHugePages: 21760000 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 262860 kB
DirectMap2M: 6848512 kB
DirectMap1G: 61865984 kBmonwire 2
