bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

pm-memcache-amo02 is not serving any GETs according to munin

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
--
minor
RESOLVED FIXED
8 years ago
3 years ago

People

(Reporter: jbalogh, Assigned: jabba)

Tracking

Details

(URL)

(Reporter)

Description

8 years ago
This is presumably just a munin issue, but pm-memcache-amo02 is not showing any GETs (and thus no hits).  According to the metrics we're getting from memcache in our app, this is not true.

pm-memcache-amo02: http://munin.mozilla.org/munin/app-memcache/pm-memcache-amo02.mozilla.org/memcached_rates.html

all memcache boxes: http://munin.mozilla.org/munin/app-memcache/comparison-week.html#memcache
munin's busted.

[root@pm-memcache-amo02 ~]# telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
stats
STAT pid 6090
STAT uptime 669486
STAT time 1273598799
STAT version 1.4.5
STAT pointer_size 64
STAT rusage_user 21266.972928
STAT rusage_system 125085.660089
STAT curr_connections 1887
STAT total_connections 4558755
STAT connection_structures 3524
STAT cmd_get 4728828753
STAT cmd_set 249430698
STAT cmd_flush 0
STAT get_hits 4533326456
STAT get_misses 195502297
STAT delete_misses 54287
STAT delete_hits 184830
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 1080066631167
STAT bytes_written 14014359699404
STAT limit_maxbytes 7516192768
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 233773
STAT bytes 6153756686
STAT curr_items 10328895
STAT total_items 249423587
STAT evictions 238026
STAT reclaimed 45970894
END

And I ran it a second later :

STAT get_hits 4533453894
STAT get_misses 195506745
(Assignee)

Updated

8 years ago
Assignee: server-ops → jdow
(Assignee)

Comment 2

8 years ago
I think this is related to bug 565232. I just wanted to update that I am working on it, even though I haven't updated this bug in a while. Sorry for the delay.
(Assignee)

Comment 3

8 years ago
I'm going to call this one closed. It started working again two days ago after what seems like the cache getting cleared on the box. Also, I will track general munin/memcache issues in bug 565232.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.