1256070 - Re-enable release-sanity.py in release-runner.py

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Rail Aliiev [:rail]]

We disabled release sanity at some point, but it would be better to enable it back to catch possible mistakes.

http://hg.mozilla.org/build/tools/file/old-release-runner/buildfarm/release/release-runner.py#l423 is how it is invoked in old world.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

We had some more release sanity related issues in 48.0b3 - https://github.com/mozilla/releasewarrior/blob/master/releases/firefox-beta-48.0b3.md

* we should block the build if one of the ship-it partials build did not ship successfully to prevent situations like this 
(48.0b2build1 failed on one particular win64 chunk, we had a follow-up build2, but for 48.0b3 we used 48.0b2build1 which ended up serbing complete mars for users, instead of partial mars
* maybe consider blocking the build if there is follow-up for one of the specified partial builds (if used 48.0b2build1, fail and suggest there is a follow-up build2 or build3 to be used)
* ensure all previous mar files exist for each platform based on partial list submitted in ship-it

Comment 2

[Rail Aliiev [:rail]]

Would be also great to check if RC has both beta and release versions in partials, see bug 1265579

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

This is done now as per bug 1282959. Release sanity in the non-relpro world from http://hg.mozilla.org/build/tools/file/old-release-runner/buildbot-helpers/release_sanity.py has been migrated to http://hg.mozilla.org/build/tools/file/tip/lib/python/kickoff/sanity.py.

Closing this for now all old sanity + some other checks have been migrated and turned on.

Comment 4

[Rail Aliiev [:rail]]

Awww: :)

Traceback (most recent call last):
  File "release-runner.py", line 413, in main
    validate_graph_kwargs(queue, gpg_key_path, **kwargs)
  File "release-runner.py", line 225, in validate_graph_kwargs
    sanitize_en_US_binary(queue, task_id, gpg_key_path)
  File "release-runner.py", line 190, in sanitize_en_US_binary
    validate_signatures(checksums, signature, tempdir, gpg_key_path)
UnboundLocalError: local variable 'signature' referenced before assignment
2016-07-18 18:54:23,077 - DEBUG - Releasing lock: /builds/releaserunner/tools/buildfarm/release/.release-runner.lock

Comment 5

[Rail Aliiev [:rail]]

Aactually the issue wasn't that code, we found that add-on-devel builds were overriding win*-opt build indexes and those builds have not checksums, see bug 1287665.

however, the message is misleading. Probably we should rearrange the code to get rid of UnboundLocalError.

Feel free to close this bug (enough!) and address it in a separate bug.

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

I'll close this bug and add a TODO for bug 1287343 which is more suitable to address the aforementioned issue.
Thanks rail for brining this up - will take care of it as part of moving the en-US binary sanity into the new shiny main one.