980218 - Run contrib/clear-memcached.pl after each push

Status: VERIFIED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[:glob ✱]

currently memcached entries are not being cleared while running checksetup.

this is because the cache is cleared by checksetup, which is run on the admin node.  either the admin node doesn't have flows to all memcache clusters, or it's missing /etc/hosts entries.

similar to bug 934521, please run control/clear-memcached.pl on a webhead to clear the cache.  this doesn't have to run on each webhead, but there's no real harm in doing so.

Comment 1

[Kendall Libby [:fubar] (he/him)]

It's missing flows, will submit bug to fix. Memcache cleared from web5.

web5.bugs.scl3# ./contrib/clear-memcached.pl 
memcached cleared

Comment 2

[Kendall Libby [:fubar] (he/him)]

Added explicit hosts entries for memcache nodes across the board:

e.g., scl3:
10.22.82.16	memcache1-prod
10.22.85.16	memcache1-stage
10.22.82.17	memcache2-prod
10.22.85.17	memcache2-dev

(only prod in phx1)

Comment 3

[:glob ✱]

  dev: changed "memcache2:11211" to "memcache2-dev:11211"
stage: changed "memcache1:11211" to "memcache1-stage:11211"
 prod: changed "memcache1:11211 memcache2:11211" to "memcache1-prod:11211 memcache2-prod:11211"

Comment 4

[Kendall Libby [:fubar] (he/him)]

Old entries removed in puppet.

Comment 5

[Kendall Libby [:fubar] (he/him)]

I think we're all set here; if not, re-open and we'll see if we can sort out any issues.

Comment 6

[Ashish Vijayaram [:ashish]]

(In reply to Kendall Libby [:fubar] from comment #5)
> I think we're all set here; if not, re-open and we'll see if we can sort out
> any issues.

You'd missed the original topic :P

Added this to the push script to run contrib/clear-memcached.pl from bugzillaadm in sysadmins r85953:

> +  # Run contrib/clear-memcached.pl, bug 980218
> +  logecho "+ contrib/clear-memcached.pl"; contrib/clear-memcached.pl 2>&1 | /root/bin/timestamp | tee -a "$logfile"

Comment 7

[Ashish Vijayaram [:ashish]]

Verified during today's push:

+ contrib/clear-memcached.pl
[23:13:32] memcached cleared