1854004 - ./mach try again --push-to-lando is failing due to a 502 Gateway Error from Lando

Status: RESOLVED FIXED

(Conduit :: Lando, defect, P2)

Description

[Sylvestre Ledru [:Sylvestre]]

Error running mach: mach try again --push-to-landoThe error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke ./mach busted to check if this issue is already on file. If it
isn't, please use ./mach busted file try to report it. If ./mach busted is
misbehaving, you can also inspect the dependencies of bug 1543241.If filing a bug, please include the full output of mach, including this error
message.The details of the failure are as follows:json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1) File "/home/sylvestre/dev/mozilla/mozilla-unified.hg/tools/tryselect/mach_commands.py", line 391, in try_again
return run(command_context, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sylvestre/dev/mozilla/mozilla-unified.hg/tools/tryselect/mach_commands.py", line 196, in run
return mod.run(**kwargs)
^^^^^^^^^^^^^^^^^
File "/home/sylvestre/dev/mozilla/mozilla-unified.hg/tools/tryselect/selectors/again.py", line 149, in run
return push_to_try(
^^^^^^^^^^^^
File "/home/sylvestre/dev/mozilla/mozilla-unified.hg/tools/tryselect/push.py", line 231, in push_to_try
push_to_lando_try(vcs, commit_message)
File "/home/sylvestre/dev/mozilla/mozilla-unified.hg/tools/tryselect/lando.py", line 273, in push_to_lando_try
post_try_push_to_lando(
File "/home/sylvestre/dev/mozilla/mozilla-unified.hg/tools/tryselect/lando.py", line 178, in post_try_push_to_lando
response_json = response.json()
^^^^^^^^^^^^^^^
File "/home/sylvestre/dev/mozilla/mozilla-unified.hg/third_party/python/requests/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/json/init.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
./mach try again --push-to-lando 6,44s user 2,13s system 5% cpu 2:44,86 total

Comment 1

[Connor Sheehan [:sheehan]]

This failure was caused by a stalled deploy to Lando-API dev, and as a result Lando-API wasn't sending valid JSON in the response. We should handle a failure from Lando more gracefully in mach try.

Perhaps we could also fall back to a VCS push when a push to Lando is having problems.

Comment 2

[Connor Sheehan [:sheehan]]

Even after un-stalling the deploy I mentioned in comment 1, I ran into this problem a few times today. The updated version of D187641 will response.raise_for_status() and display the true problem to the user, which is a recurring 502 Gateway Error. I've made jbuck aware of the problem and we're working to debug and resolve it.

Comment 3

[Connor Sheehan [:sheehan]]

Seems like this is resolved.