Try repository broken: pushlog empty - Try tree closed, same for Try-comm-central
Categories
(Developer Services :: Mercurial: hg.mozilla.org, defect)
Tracking
(Not tracked)
People
(Reporter: aryx, Assigned: sheehan)
Details
https://hg.mozilla.org/try shows no commits right now, Try pushes don't show up on treeherder for this reason.
Will close Try tree for that.
|
Reporter | |
Updated•6 years ago
|
|
||
Comment 1•6 years ago
|
||
try-comm-central seems to have the same affliction. I've closed it too.
|
|
Reporter | |
Updated•6 years ago
|
|
Assignee | |
Comment 3•6 years ago
|
||
There was an abandoned transaction on the try repo for one of the hgweb mirrors in us-west-2. This caused an abort loop for the replication consumers on the try partition, which includes both try and try-comm-central. Since the hosts in us-west-2 are in the pushdataaggregator_groups file, the consumer being stuck on that host means the entire cluster of web heads will not serve any changesets that have not replicated to that host.
I ran hg recover on the try repo on that host, which un-jammed the consumer, and things are replicating as normal again.
The delay in reacting to this bug has two root causes. The first is that the issue happened to arise about 20 minutes after I left the hotel at the Whistler All Hands to go on a bike ride with some other Mozillians. Had this happened at literally any other time this week, I could have fixed it in minutes. We have alerting that pointed to the root cause, and once I got to my computer it was a simple fix.
The second is that I have not given enough people access to the hgweb mirrors so that someone else could jump on and fix the problem in my absence. I will open another bug to get some people the required access, and train the people who already have access on how to debug and fix issues like this.
I sincerely apologize if this bug blocked anyone's work, especially during the all hands.
Description
•