688035 - -dev redis is all masters and no slaves

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wil Clouser [:clouserw]]

We're trying to get -dev back online (see bug 685669 for some backstory) and the current best theory is that redis is all out of sync.  https://addons-dev.allizom.org/services/monitor says all 4 redis hosts are masters when 2 should be slaves (see https://addons.mozilla.org/services/monitor for the correct configuration).

Comment 1

[Jake Maul [:jakem]]

What are you looking at on those pages? For addons-dev, I see cache, cache_slave, master, and slave in the Redis section.

Comment 2

[Wil Clouser [:clouserw]]

That's correct.  "slave" and "cache_slave" should be "role: slave" in their details but they are "role: master"

Comment 3

[Jake Maul [:jakem]]

Ah, thanks, now I see what you're referring to.

I found this in settings_local.py:

REDIS_BACKEND = 'redis://redis-dev-master:6379?socket_timeout=0.1'
REDIS_BACKENDS = {
    'cache': REDIS_BACKEND,
    'cache_slave': 'redis://redis-dev-master:6379?socket_timeout=0.1',
    'master': 'redis://redis-dev-master:6380?socket_timeout=0.1',
    'slave': 'redis://redis-dev-master:6380?socket_timeout=0.1',
}
CACHE_MACHINE_USE_REDIS = True

Note that cache and cache_slave are the same, and master/slave are the same. This leads me to believe that as a shortcut to setting up 4 full Redis instances, we simply set up 2, and then pointed both the 'master' and 'slave' names of each at each one... and set up no actual master/slave relationships.

It'd be easy to verify this- just insert something into cache or master, and see if you can read it from cache_slave or slave, respectively. I suspect it will work properly.

Comment 4

[Wil Clouser [:clouserw]]

-> normal.  Sounds like this might be a red herring if oremj set it up that way for dev.  The site is back, so no longer a critical issue.

Comment 5

[Jake Maul [:jakem]]

I'm going to kick this to oremj just to confirm that this was intentional. He's on leave for another week or so, but I'm pretty sure it was, and like you said the site is working again.

Comment 6

[Jeremy Orem [:oremj]]

The config looks like this:

REDIS_BACKEND = 'redis://redis-dev-master:6379?socket_timeout=0.1'
REDIS_BACKENDS = {
    'cache': REDIS_BACKEND,
#    'cache_slave': REDIS_BACKEND,
#    'master': REDIS_BACKEND,
#    'slave': REDIS_BACKEND,
    'cache_slave': 'redis://redis-dev-master:6379?socket_timeout=0.1',
    'master': 'redis://redis-dev-master:6380?socket_timeout=0.1',
    'slave': 'redis://redis-dev-master:6380?socket_timeout=0.1',
}

We just don't have a second redis server for dev, so slave and master point to the same redis instance. Luckily, it is impossible that they are out of sync.

I'm 99% positive that zamboni doesn't actually use the redis slaves, so it shouldn't really matter either way.

Jeff, can you confirm the lack of redis slave use and mark this verified?

Comment 7

[Jeff Balogh (:jbalogh)]

(In reply to Jeremy Orem [[e-mail]] from comment #6)
> I'm 99% positive that zamboni doesn't actually use the redis slaves, so it
> shouldn't really matter either way.
> 
> Jeff, can you confirm the lack of redis slave use and mark this verified?

The only thing using the slave is the script that cleans out expired keys. It only uses the slave so it doesn't lock up the master asking for all the keys.

AMO doesn't care about master vs. slave for redis. That could change in the future, but right now all masters is fine.