Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

A redis server v2.8.4 is running on a Ubuntu 14.04 VPS with 8 GB RAM and 16 GB swap space (on SSDs). However htop shows that redis alone is taking up 22.4 G of memory!

redis-server eventually crashed due to out of memeory. Mem and Swp both hits 100% then redis-server is killed along with other services.

From dmesg:

[165578.047682] Out of memory: Kill process 10155 (redis-server) score 834 or sacrifice child
[165578.047896] Killed process 10155 (redis-server) total-vm:31038376kB, anon-rss:5636092kB, file-rss:0kB

Restarting redis-server from etiher a OOM crash or a service redis-server force-reload causes memory usage to drop to <100MB.

Question: Why does redis-server occupy more and more memory till it crashes? How can we prevent this?

Is it true that setting maxmemory will not work because once redis hits the maxmemory limit, it will start removing data?

enter image description here enter image description here

After restarting redis-server

enter image description here enter image description here

Redis version: Redis server v=2.8.4 sha=00000000:0 malloc=jemalloc-3.4.1 bits=64 build=a44a05d76f06a5d9


Update

When htop reports the memory usage of redis-server to be 4.4G RAM and 22.6G Swap, the amount of space taken up by all the keys in redis is only 60.59636307 MB, as reported by rdbtools. This is also the amount of RAM taken up by redis-server right after it restarts.

INFO ALL when redis-server is taking up tons of memory

mem_fragmentation_ratio:0.19

127.0.0.1:6379> INFO all

# Server
redis_version:2.8.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:a44a05d76f06a5d9
redis_mode:standalone
os:Linux 3.13.0-24-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.2
process_id:26858
run_id:4d4a507b325e567d5ada203a0c65891bcf4d02de
tcp_port:6379
uptime_in_seconds:100011
uptime_in_days:1
hz:10
lru_clock:165668
config_file:/etc/redis/redis.conf

# Clients
connected_clients:60
client_longest_output_list:768774
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:23973468008
used_memory_human:22.33G
used_memory_rss:4563857408
used_memory_peak:24083474760
used_memory_peak_human:22.43G
used_memory_lua:33792
mem_fragmentation_ratio:0.19
mem_allocator:jemalloc-3.4.1

# Persistence
loading:0
rdb_changes_since_last_save:127835154
rdb_bgsave_in_progress:0
rdb_last_save_time:1406716479
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok

# Stats
total_connections_received:110
total_commands_processed:386765263
instantaneous_ops_per_sec:3002
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:1385878
keyspace_misses:23655
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:82

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:10547.48
used_cpu_user:8240.36
used_cpu_sys_children:201.83
used_cpu_user_children:914.86

# Commandstats
cmdstat_del:calls=136,usec=1407,usec_per_call=10.35
cmdstat_exists:calls=161428,usec=1391252,usec_per_call=8.62
cmdstat_zadd:calls=64149642,usec=936323882,usec_per_call=14.60
cmdstat_zrem:calls=137,usec=2131,usec_per_call=15.55
cmdstat_zremrangebyscore:calls=2293,usec=111905082,usec_per_call=48802.91
cmdstat_zrange:calls=7925,usec=285907448,usec_per_call=36076.65
cmdstat_zrangebyscore:calls=921434,usec=292731002,usec_per_call=317.69
cmdstat_zcount:calls=8,usec=172,usec_per_call=21.50
cmdstat_zrevrange:calls=191184,usec=965447,usec_per_call=5.05
cmdstat_zcard:calls=5180,usec=13502,usec_per_call=2.61
cmdstat_zscore:calls=29856,usec=576044,usec_per_call=19.29
cmdstat_hset:calls=64145124,usec=199407095,usec_per_call=3.11
cmdstat_hget:calls=248487,usec=501220,usec_per_call=2.02
cmdstat_hincrby:calls=128339355,usec=2071112929,usec_per_call=16.14
cmdstat_hgetall:calls=193747,usec=1608260,usec_per_call=8.30
cmdstat_select:calls=1,usec=5,usec_per_call=5.00
cmdstat_rename:calls=134,usec=1090,usec_per_call=8.13
cmdstat_keys:calls=4503,usec=4997628,usec_per_call=1109.84
cmdstat_bgsave:calls=2,usec=20012,usec_per_call=10006.00
cmdstat_type:calls=603,usec=2736,usec_per_call=4.54
cmdstat_multi:calls=64181979,usec=383633610,usec_per_call=5.98
cmdstat_exec:calls=64181979,usec=4403181204,usec_per_call=68.60
cmdstat_info:calls=126,usec=28675,usec_per_call=227.58

# Keyspace
db0:keys=2109,expires=0,avg_ttl=0
share|improve this question

2 Answers 2

  1. Use the maxmemory to set a limit to how much your Redis database can grow too. Failing to do so, Redis will grow until the OS will kill it once memory is exhausted (per your current experience).
  2. The usage of maxmemory should be coupled with maxmemory-policy - you can choose from different eviction policies depending on your use case's requirements. For example, if you use the allkey-lru eviction policy, Redis will indeed start evicting (the least recently used) data once maxmemory has been reached. Alternatively, you can instruct Redis to evict only expire-able data with the volatile-lru or volatile-random policies. Lastly, you can set the policy to noeviction but that would mean that once memory has been exhausted, Redis will deny further writes with an OOM message.

Edit:

First disable swap - Redis and swap don't mix easily and this can certainly cause slowness.

Also do free -m instead of top for the full picture of your RAM's state (http://www.linuxatemyram.com/).

share|improve this answer
    
Thank you, I'm confused as to why the memory usage keeps growing, yet doing a bgsave and restarting redis-server causes the memory usage to drop to a more reasonable value of 70 MB. Could this be a memory leak? –  Nyxynyx Jul 30 at 18:42
    
Possible but unlikely (or other people would have reported it)... More likely a fragmentation issue. Next time that happens, post the output of your Redis' INFO ALL. If my guess is correct, the mem_fragmentation_ratio will by sky-high. –  Itamar Haber Jul 30 at 18:46
    
redis-server hogs up all the memory and crashes everyday. It's about to use up all the memory now, so I've captured the output of INFO ALL and added to the OP. mem_fragmentation_ratio:0.19 –  Nyxynyx Jul 30 at 19:30
    
If the redis datasets do not exceed 250MB and maxmemory is set to 1 GB, does this mean that when redis's mem usage hits 1GB, eviction will still remove data? Since redis's mem_fragmentation_ratio is 0.19, does it mean that theres too much fragmentation, or too much is stored in swap, or both? Any way to reduce the fragmentation? –  Nyxynyx Jul 30 at 19:36
    
When redis-server is about to crash due to OOM, rdbtools shows that the keys in redis only takes up 60MB. This looks like extremely serious fragmentation? Considering its taking up 4.4GB of RAM and 22.4G of Swap. –  Nyxynyx Jul 30 at 19:53

This is almost certainly memory fragmentation, as redis is well-known and loved in production and you probably haven't found a memory leak.

The recommendations about setting the size of the pool won't help fragmentation. You'll have to specifically lower the Redis size - lower than your actual memory size - because Redis can't account for fragmentation - but, in terms of a short answer, you'll have to do that, and start plan on restarting your servers frequently.

My rule of thumb working with a variety of operating systems and in-memory databases is you need 2x your actual memory, and the memory size will stabilize in about 2 weeks.

However, that depends on your actual allocation patterns, and the memory allocator you're using.

Right now, the best memory allocator I've found for servers is JEMalloc. We use it at Aerospike now to reduce (nearly remove) long-term memory fragmentation. JEMalloc has a feature that allows you to create a memory "arena" (pool), and on any allocation, choose which pool, thus giving you like-size allocations and to manage similar memory lifetime allocations. It's been a big win for us in the kind of cases you're discussing.

The Zend PHP engine is sophisticated in this regard, because all allocations inside the engine are either in per-transaction memory, or global memory. Per transaction memory is freed at one-fell-swoop at the end of the transaction, and thus can be very efficient.

If you're in Linux, the kernel memory allocator (Clib) has taken a number of twists and turns, and which version you're on will dramatically determine the amount of fragmentation, as will the actual application pattern. For example, some allocators are much better when you are slightly growing objects, some are much worse. Regrettably, even discussing with other Redis users means talking about which OS, and which OS version, you're using.

The fact that you can restart the server (from persistence) and get your memory back could mean a leak, but more likely points to fragmentation.

  1. Disallow swap (it's better to OOM than to swap, for redis)
  2. Decrease redis' memory size
  3. Restart on a schedule
share|improve this answer
    
How would you decrease the memory size, by adjusting maxmemory? –  Nyxynyx Aug 23 at 20:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.