1405126 - Autoland stuck on repeatedly failing MozReviewUpdate

Status: RESOLVED FIXED

(Conduit Graveyard :: Transplant, defect)

Description

[Steven MacLeod [:smacleod]]

I was notified on irc that autoland was failing to land a change to try after quite a while. Upon investigation the change had landed, but MozReview was still displaying as in progress as it had not received the update from autoland.

The autoland logs show that transplant request 37310 is repeatedly failing in the MozReview update queue holding everything up:

> Oct 02 20:46:12 autoland.mozilla.org python[31401]: 20:46:12,790 autoland INFO failed: 404 - {"stat": "fail", "err": {"msg": "Object does not exist", "code": 100}}

The associated row in the update table is as follows:
>  103311 |         37310 | {"landed": true, "trysyntax": "", "result": "ddfc127526b2", "request_id": 37310, "rev": "4a06eedec30f1aedae83297d89071389daa7e617", "destination": "ssh://hg.mozilla.org/integration/autoland", "tree": "gecko", "error_msg": ""}

This update request appears to fail because MozReview itself doesn't seem to know about the landing request (see 404 above ^). This can also be seen by viewing the review requests associated with this landing request [1][2]. I'd like to dig further into how this happened, but the initiator of the landing request (:ahal) is now on PTO.

[1] https://reviewboard.mozilla.org/r/183638/
[2] https://reviewboard.mozilla.org/r/183636/

Comment 1

[Steven MacLeod [:smacleod]]

I'm going to delete the problematic row from the update table to unstick things, rather than try to craft a proper history of the landing request in the RB database. This will be easier and should get things rolling again.

Comment 2

[Steven MacLeod [:smacleod]]

:ahal, do you remember anything strange from when you autolanded in Bug 1403222? It might help with investigation into how this ended up happening.

Comment 3

[Steven MacLeod [:smacleod]]

Ran the following to unstick the queue:
> DELETE FROM MozreviewUpdate WHERE id = 103311;

The queue has now emptied, things are moving again. Verified by checking there are 0 rows in the update table and the originally reported review request[1] has been updated.

[1] https://reviewboard.mozilla.org/r/185628/

Comment 4

[Andrew Halberstadt [:ahal]]

Yeah sorry, I did see some kind of error message (I think it was a popup that either disappeared or I dismissed), but then I noticed the push went through so I didn't end up filing a bug.