Closed Bug 1307326 Opened 3 years ago Closed 3 years ago

50.0b4 failed release-runner because TC build finished but not buildbot


(Release Engineering :: Release Automation: Other, defect)

Not set


(Not tracked)



(Reporter: nthomas, Assigned: mtabara)



(Whiteboard: toverify)


(1 file)

Similar to bug 1304260 but not quite. At the time 50.0b4 was started in ship-it the TC build had finished but not the buildbot one. So even though TC was Tier 2 (and therefore Rank 0 in the index) we hit

2016-10-03 18:43:18,552 - DEBUG - "GET /K1d4PZltTnOmctA4OV3SWw/0/public/build/target.checksums HTTP/1.1" 200 6804
2016-10-03 18:43:18,555 - INFO - Attempt to validate signatures ...
2016-10-03 18:43:18,631 - ERROR - Failed to start release "Firefox-50.0b4-build1" promotion for graph nTbouKvWTMS_NAJ-nVysAw. Error(s): local variable 'signature' referenced before assignment
Traceback (most recent call last):
  File "", line 455, in main
    validate_graph_kwargs(queue, gpg_key_path, **kwargs)
  File "", line 259, in validate_graph_kwargs
    sanitize_en_US_binary(queue, task_id, gpg_key_path)
  File "", line 224, in sanitize_en_US_binary
    validate_signatures(checksums, signature, tempdir, gpg_key_path)
UnboundLocalError: local variable 'signature' referenced before assignment

Should we require the rank in the index be > 0 before accepting a taskID ? Or live with it until taskcluster task are better next cycle ?

Working around by restarting the release after buildbot is done.
This happened again today :/

We need to teach to ignore tasks with task.extra.index.rank == 0. This may require an extra queue.task() call.
Something like should work. Just need to make the tests happier.
Assignee: nobody → mtabara
Comment on attachment 8801120 [details]
Bug 1307326 - consider only tasks with rank greater than 0.
Attachment #8801120 - Flags: review?(rail) → review+
Comment on attachment 8801120 [details]
Bug 1307326 - consider only tasks with rank greater than 0.
Attachment #8801120 - Flags: checked-in+
deployed to bm8{3,5}
Whiteboard: toverify
Changes worked in 50.0b8 \o/

From the release runner log:
2016-10-17 16:15:46,455 - DEBUG - Ignoring task EOGKtI74S8i4M3AB5aBH7w because the rank is set to 0
2016-10-17 16:15:46,455 - INFO - Builds are not completed yet, skipping release "Firefox-50.0b8-build1" for now
2016-10-17 16:15:46,455 - INFO - updating status for Firefox-50.0b8-build1 to Waiting for builds to be completed
2016-10-17 16:15:46,455 - INFO - Resetting dropped connection:
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.