How to monitor memcached
There are several ways to retrieve stats for your memcached daemon, and to confirm all is well with how it works. Most of the time you’ll want to monitor cache HITS and MISSES, so we’ll primarily focus on this aspect.
Pick your preferred monitoring tool
Whether you don’t want to install any additional package or fancy more detailed statistics, your preferred monitoring tool will vary.
Let’s make a few assumptions first to better understand the context of this post:
- Your database server is running Ubuntu/Debian.
- You have one or several web nodes and at least one database server (alternatively you could have a primary/replica).
- memcached is installed on the remote database server.
- The database server’s IPv4 address is 203.0.113.12
- memcached is listening on its default port (11211)
- netcat is installed
Using netcat
In most Ubuntu-based distributions and Debian, netcat will be available by default. This allows to run commands such as:
$ echo stats | nc 203.0.113.12 11211
STAT pid 71
STAT uptime 6949
STAT time 1486379640
STAT version 1.4.21
STAT libevent 2.0.21-stable
STAT pointer_size 64
(snipped)
ENDThis gives you extensive stats (uptime, version, HITS, MISSES, connections, evictions…) about the memcached daemon. If you wish to retrieve HITS in real-time, you can build upon this command and do the following.
$ watch -td '(echo stats ; echo quit) | nc 203.0.113.12 11211 | grep get_hits '
STAT get_hits 1987The number will then increase as soon as there’s data coming in through the memcached daemon.
Using memcstat
Installing the libmemcached-tools package will give you access to the memcstat command, which displays the operating status of a single or group of memcached servers. Usage is very simple.
$ memcstat --servers=203.0.113.12 11211 | grep get_hits
get_hits: 1207Once again we’ve filtered stats out to retrieve memcached HITS only. You get the point.
Using telnet
Telnet is usually installed on Ubuntu-based distributions but not on Debian. It’s a network protocol that can help you with interacting with the memcached daemon.
Since we’re primarily interested into high-level memcached monitoring here, we’ll simply use the stats command here.
telnet 203.0.113.12 11211
Trying 203.0.113.12...
Connected to 203.0.113.12.
Escape character is '^]'.
stats
STAT pid 71
STAT uptime 10791
STAT time 1486383482
STAT version 1.4.21
STAT libevent 2.0.21-stable
STAT pointer_size 64
STAT rusage_user 0.356000
STAT rusage_system 0.652000
STAT curr_connections 12
STAT total_connections 804
STAT connection_structures 13
(snipped)Using memcached-top
memcached-top was originally hosted on Google Code and now lives on Github. It’s a perl file that allows you to watch the traffic and stats in real-time. It wasn’t updated in a while but still seems to work fine.
$ perl memcache-top --instance=203.0.113.12
memcache-top v0.7 (default port: 11211, color: on, refresh: 3 seconds)
INSTANCE USAGE HIT % CONN TIME ITEMS EVICT/s READ/s WRITE/s
203.0.113.12:11211 5.3% 90.7% 10 0.6ms 867 0.0 9207 118.2K
AVERAGE: 5.3% 93.4% 10 0.7ms 867 0.0 9831 114.9K
TOTAL: 3.4MB/64.0MB 10 0.6ms 867 0.0 8999 117.2K
(ctrl-c to quit.)Using mcstat
I find mcstat more interesting because it’s also more advanced. It wasn’t updated in a while so it might behave inconsistently as memcached evolves.
It reports on memcached daemon statistics, but in real-time, and arbitrarily returns only the stats that really matter to you when evaluating if memcached is doing its job or not. When you exit it, it immediately returns session statistics.
$ mcstat 203.0.113.12 11211
Memcache version 1.4.21
Max Memory Size: 64.0M Pointer size: 64 bit
time cnct used gets hits miss sets evic miss% evic% in out
10:23:23 10 2.7M 1.3k 1.2k 87 6.1k 0 6.72 0.00 24.8M 11.0M
10:23:28 10 2.7M 0 0 0 0 0 0.00 0.00 7 1.1k
10:23:33 10 2.7M 209 206 3 7 0 1.44 0.00 20.0k 1.4M
10:23:38 10 2.7M 202 178 24 37 0 11.88 0.00 95.8k 1.6M
10:23:43 10 2.7M 209 175 34 43 0 16.27 0.00 181.9k 2.2M
Session Statistics:
Statistic Total Session
uptime 1h 5m 12s 20s
libevent 2.0.21-stable 0
rusage_user 0.24s 0.01s
rusage_system 0.45s 0.03s
curr_connections 10
total_connections 644 10
connection_structures 12 0
reserved_fds 20 0
cmd_get 1.3k 620
cmd_set 6.1k 87
cmd_flush 0 0
cmd_touch 0 0
get_hits 1.2k 559
get_misses 87 61
delete_misses 3 1
delete_hits 18 1
incr_misses 0 0
incr_hits 0 0
decr_misses 0 0
decr_hits 0 0
cas_misses 0 0
cas_hits 0 0
cas_badval 0 0
touch_hits 0 0
touch_misses 0 0
auth_cmds 0 0
auth_errors 0 0
bytes_read 24.8M 297.7k
bytes_written 11.0M 5.2M
accepting_conns 1 0
listen_disabled_num 0 0
threads 4 0
conn_yields 0 0
hash_power_level 16 0
hash_bytes 524.3k 0
hash_is_expanding 0 0
malloc_fails 0 0
bytes 2.7M 74.2k
curr_items 603-649
total_items 6.1k 87
expired_unfetched 0 0
evicted_unfetched 0 0
evictions 0 0
reclaimed 0 0
crawler_reclaimed 0 0
lrutail_reflocked 0 0
Connection closed
Exiting.As you can see, there are a lot of tools available for you to easily monitor memcached. Not all are created equal so it’s really up to you what’s the best tool for the job. My advice is: keep it simple until you need to dive deeper.