1467078 - FF 52.8.1esr buildbot bridge exceptions

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

eg BRzuZpv8TdC-MVbxSe9mDA - firefox mozilla-esr52 linux l10n repack 1/10:

Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener bbb.tcutils - task BRzuZpv8TdC-MVbxSe9mDA: run 0: couldn't upload artifact to s3
Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener bbb.services - buildrequest 155891979: failed to handle finished event 
Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener Traceback (most recent call last):
Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener   File "/builds/bbb/lib/python2.7/site-packages/bbb/services.py", line 170, in handleFinished
Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener     self._handleFinishedRequest(brid, properties, results)
Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener   File "/builds/bbb/lib/python2.7/site-packages/statsd/client.py", line 37, in _wrapped
Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener     return_value = f(*args, **kwargs)
Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener   File "/builds/bbb/lib/python2.7/site-packages/bbb/services.py", line 207, in _handleFinishedRequest
Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener     createJsonArtifact(self.tc_queue, taskid, runid, "public/properties.json", properties, expires)
Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener   File "/builds/bbb/lib/python2.7/site-packages/bbb/tcutils.py", line 40, in createJsonArtifact
Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener     raise IOError("couldn't upload artifact to s3")
Jun 06 16:51:21 bm82 python: buildbot_bridge_bblistener IOError: couldn't upload artifact to s3 

Tasks ends up staying in Running state in the TC graph, despite having finished in buildbot.

I suspect this is related to bug 1467067, because createJsonArtifact() sents an int value for the Content-Length header and we have requests v2.18.4 since the recent puppet improvements.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: buildbot-bridge PR

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: puppet PR

Not sure why I'm not getting test runs on these PRs :/

Comment 3

[[e-mail] (:bhearsum)]

I believe this is fixed now. I ended up kicking along the release with cancel/rerun/complete of the "running" tasks, and killing all of the buildbot jobs that spawned because of it (to avoid them overwriting any existing artifacts).

Before doing the above I tried to directly complete the tasks with my own credentials and the buildbot bridge credentials - neither of which worked. After consulting with #taskcluster I couldn't think of any other options than the above that didn't require a rebuild.

Thanks for all your work here, Nick, and sorry.