Closed Bug 1307326 Opened 3 years ago Closed 3 years ago

50.0b4 failed release-runner because TC build finished but not buildbot

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: nthomas, Assigned: mtabara)

References

Details

(Whiteboard: toverify)

Attachments

(1 file)

Similar to bug 1304260 but not quite. At the time 50.0b4 was started in ship-it the TC build had finished but not the buildbot one. So even though TC was Tier 2 (and therefore Rank 0 in the index) we hit

2016-10-03 18:43:18,552 - DEBUG - "GET /K1d4PZltTnOmctA4OV3SWw/0/public/build/target.checksums HTTP/1.1" 200 6804
2016-10-03 18:43:18,555 - INFO - Attempt to validate signatures ...
...
2016-10-03 18:43:18,631 - ERROR - Failed to start release "Firefox-50.0b4-build1" promotion for graph nTbouKvWTMS_NAJ-nVysAw. Error(s): local variable 'signature' referenced before assignment
Traceback (most recent call last):
  File "release-runner.py", line 455, in main
    validate_graph_kwargs(queue, gpg_key_path, **kwargs)
  File "release-runner.py", line 259, in validate_graph_kwargs
    sanitize_en_US_binary(queue, task_id, gpg_key_path)
  File "release-runner.py", line 224, in sanitize_en_US_binary
    validate_signatures(checksums, signature, tempdir, gpg_key_path)
UnboundLocalError: local variable 'signature' referenced before assignment

Should we require the rank in the index be > 0 before accepting a taskID ? Or live with it until taskcluster task are better next cycle ?

Working around by restarting the release after buildbot is done.
This happened again today :/

We need to teach http://hg.mozilla.org/build/tools/file/tip/lib/python/kickoff/build_status.py#l67 to ignore tasks with task.extra.index.rank == 0. This may require an extra queue.task() call.
Something like https://gist.github.com/rail/2b1c0f8d8520e5cb87451f0842c11586 should work. Just need to make the tests happier.
Assignee: nobody → mtabara
Comment on attachment 8801120 [details]
Bug 1307326 - consider only tasks with rank greater than 0.

https://reviewboard.mozilla.org/r/85914/#review84570
Attachment #8801120 - Flags: review?(rail) → review+
Comment on attachment 8801120 [details]
Bug 1307326 - consider only tasks with rank greater than 0.

http://hg.mozilla.org/build/tools/rev/78748858da19
Attachment #8801120 - Flags: checked-in+
deployed to bm8{3,5}
Whiteboard: toverify
Changes worked in 50.0b8 \o/

From the release runner log:
...
2016-10-17 16:15:46,455 - DEBUG - Ignoring task EOGKtI74S8i4M3AB5aBH7w because the rank is set to 0
2016-10-17 16:15:46,455 - INFO - Builds are not completed yet, skipping release "Firefox-50.0b8-build1" for now
2016-10-17 16:15:46,455 - INFO - updating status for Firefox-50.0b8-build1 to Waiting for builds to be completed
2016-10-17 16:15:46,455 - INFO - Resetting dropped connection: ship-it.mozilla.org
...
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.