50.0b1 failed release-runner because TC build finished after buildbot

RESOLVED FIXED

Status

Release Engineering
Release Automation
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: nthomas, Assigned: rail)

Tracking

(Blocks: 1 bug)

unspecified

Firefox Tracking Flags

(firefox50 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Reporter)

Description

a year ago
TC and buildbot jobs are publishing to the same index on beta. For rev 91faf7ec36cd18a8ebdc0e2edac966e5bbb15da2 the linux64 buildbot build [1] finished first, and TC build second [2], so TC was what the index pointed to [3]. The other platforms were OK (linux build finished in the other order, mac failed on TC, and no windows TC builds).

Since we don't sign TC builds we don't have an artifact matching checksums.asc, and can't continue. bug 1304259 to handle that  more gracefully, but the real issue for release automation is we should only use buildbot builds. Perhaps we've just been lucky so far, or perhaps something changed in the recent merge to beta. We shouldn't be vulnerable to timing though.

I'm attempting to work around this by re-running the buildbot job.


[1] https://tools.taskcluster.net/task-inspector/#HD5JE_paQ-SwH1_fit1ugQ/0
[2] https://tools.taskcluster.net/task-inspector/#DWkAFJPNQb6I1QtJwmB2ug/0
[3] https://tools.taskcluster.net/index/#gecko.v2.mozilla-beta.revision.91faf7ec36cd18a8ebdc0e2edac966e5bbb15da2.firefox/gecko.v2.mozilla-beta.revision.91faf7ec36cd18a8ebdc0e2edac966e5bbb15da2.firefox.linux64-opt returned DWkAFJPNQb6I1QtJwmB2ug
(Reporter)

Updated

a year ago
See Also: → bug 1304259
(Reporter)

Comment 1

a year ago
(In reply to Nick Thomas [:nthomas] from comment #0)
> TC and buildbot jobs are publishing to the same index on beta. For rev
> 91faf7ec36cd18a8ebdc0e2edac966e5bbb15da2 the linux64 buildbot build [1]
> finished first, and TC build second [2], so TC was what the index pointed to
> [3]. 

Specifically, the upload-files action in buildbot finished before the whole TC job.
(Reporter)

Comment 2

a year ago
(In reply to Nick Thomas [:nthomas] from comment #0)
> I'm attempting to work around this by re-running the buildbot job.

That worked, the index points to https://tools.taskcluster.net/task-inspector/#PcbuNN7uQpyXtA7dC4fFpQ/ now. The release was then restarted.

Leaving this open for now. Could use this bug or a more general one for the tc vs buildbot race.
hmm, are TC builds tier1 now? If not, we may need to reopen bug 1274311.
https://queue.taskcluster.net/v1/task/DWkAFJPNQb6I1QtJwmB2ug is missing "tier" in the definition. According to https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/task/legacy.py#493-498 this won't set the rank to 0, so TC wins in this case.

Maybe we should reverse the logic and make it set the rank to 0 if no "tier" is set?
Flags: needinfo?(mshal)
Comment hidden (mozreview-request)
The patch is for beta only.
(In reply to Rail Aliiev [:rail] from comment #4)
> Maybe we should reverse the logic and make it set the rank to 0 if no "tier"
> is set?

Yeah, when I added the rank-0 code for tier-2+, for some reason I made it default to tier-1 if no tier was present. However, dustin made this more sensible in bug 1286075 by defaulting to tier-3 (rank 0). That isn't on beta yet, though.

I think your approach of just setting the builds you don't want in the index as tier-2 should be sufficient.
Flags: needinfo?(mshal)

Comment 10

a year ago
mozreview-review
Comment on attachment 8793275 [details]
Bug 1304260 - opt-linux64 should be tier 2

https://reviewboard.mozilla.org/r/80066/#review78848
Attachment #8793275 - Flags: review?(dustin) → review+
Comment on attachment 8793275 [details]
Bug 1304260 - opt-linux64 should be tier 2

This is required to prevent Tier 2 Taskcluster builds overwrite Buildbot "uploads". The patch if for beta only.
The patch is trivial enough, so I didn't even tested it. :) What can possibly go wrong - linux32 config contains the line I added in this patch.

[Risks and why]: Very low, this affects only TC builds and this is reimplemented in 51.

[String/UUID change made/needed]: None.
Attachment #8793275 - Flags: approval-mozilla-beta?

Updated

a year ago
status-firefox50: --- → affected
Comment on attachment 8793275 [details]
Bug 1304260 - opt-linux64 should be tier 2

NPOTB, Beta50+
Attachment #8793275 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Assignee: nobody → rail
status-firefox50: affected → fixed
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
(Reporter)

Updated

a year ago
See Also: → bug 1307326
You need to log in before you can comment on or make changes to this bug.