1187295 - aus4 read and write webheads out of sync; blocking release work

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Problems, task)

Description

[[e-mail] (:bhearsum)]

The aus4 read and write webheads are out of sync, which means any changes we make on the admin host are not reflected on the public facing notes. Effectively, we cannot make any changes to updates. This is blocking the ongoing Beta as well as nightlies from being published. I suspect this is fallout from bug 1175652.


From the admin host (uses aus4-rw-vip.db.phx1.mozilla.com)
mysql> select rule_id, mapping, channel, product from rules where rule_id=45;
+---------+-----------------------+--------------+---------+
| rule_id | mapping               | channel      | product |
+---------+-----------------------+--------------+---------+
|      45 | Firefox-40.0b7-build1 | beta-cdntest | Firefox |
+---------+-----------------------+--------------+---------+
1 row in set (0.00 sec)


And on a public webhead (uses db-aus-prod-ro, which appears to be 10.8.70.21):
mysql> select rule_id, mapping, channel, product from rules where rule_id=45;
+---------+-----------------------+--------------+---------+
| rule_id | mapping               | channel      | product |
+---------+-----------------------+--------------+---------+
|      45 | Firefox-40.0b6-build1 | beta-cdntest | Firefox |
+---------+-----------------------+--------------+---------+

Comment 1

[[e-mail] (:bhearsum)]

Probably also worth noting that the admin webhead is ausadm.private.phx1.mozilla.com and the public facing nodes are aus[1-6].webapp.phx1.mozilla.com.

Comment 2

[Sheeri Cabral [:sheeri]]

The load balancers have been set so both reads and writes go to the master. I will resync the slave while it's offline. You should be unblocked now.

Comment 3

[[e-mail] (:bhearsum)]

All looks good to me now, thanks folks!

Comment 4

[Chris AtLee [:catlee]]

What was the cause for the read slaves to be out of sync?

Comment 5

[Sheeri Cabral [:sheeri]]

improper backup restoration technique. We have fixed that issue.