1542247 - Patch stuck in "landing queued" state, even though it already landed

Status: RESOLVED FIXED

(Conduit :: Lando, defect, P1)

Description

[Andrew McCreight [:mccr8]]

Yesterday, I used Lando to land some patches. And they landed:
https://bugzilla.mozilla.org/show_bug.cgi?id=1541684#c5

Then they got backed out. I reopened the revisions and uploaded my fixed up patches. Everything looks fine in Differential.

However, when I go to Lando, it still thinks my landing from yesterday is in the queue, so I can't reland my patches:
https://lando.services.mozilla.com/D26071/

It says "Landing Queued On Thu, April 4, 2019, 3:03 PM PDT", which is probably when I pushed them yesterday.

Comment 1

[Boris Zbarsky [:bzbarsky]]

I have the same thing going on with https://lando.services.mozilla.com/D26007/ -- it's landed and the phabricator revisions are closed and all, but it still thinks it's queued. I thought that was because the Decision task failed, but it succeeded on Andrew's push...

Comment 2

[Andrew McCreight [:mccr8]]

Note that this shouldn't get landed yet, because I need to rebase the patch again, but it would be nice if I could at least get this out of the stuck state so I can land it once I do that.

Comment 3

[:glob ✱]

lando sent a request to transplant which it promptly forgot about, blocking transplant's processing of the queue:

: INFO trying to post lando update to: https://api-private.lando.services.mozilla.com/landings/update for request: 71041
: INFO failed: None - {
: "detail": "The requested Landing does not exist",
: "status": 404,
: "title": "Landing not found",
: "type": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404"
: }

working on unblocking the queue and trying to figure out what caused lando's forgetfulness.

Comment 4

[:glob ✱]

looks like we lost 71041, 71042, and 71043.

i've cleared the responses from the queue and lando is processing landings again.

the trigger appears to be transplant taking longer than the 10 seconds timeout to process the initial request (not the landing itself). we're unsure what the cause of the timeouts were - we're thinking noisy neighbour in aws.

we'll increase the timeout in lando from 10 to 30 seconds.