Closed Bug 1462120 Opened 2 years ago Closed 2 years ago

esr60 final update verification fails, and update-verify doesn't run

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set

Tracking

(firefox62 fixed)

RESOLVED FIXED
Tracking Status
firefox62 --- fixed

People

(Reporter: mtabara, Assigned: nthomas)

References

Details

Attachments

(2 files)

For "Unable to download config file(s) or config files are missing from repo". More here[1].

[1]: https://taskcluster-artifacts.net/Qqs8HC4VTbySKXjBbcwJPQ/0/public/logs/live_backing.log
I took a look into this. The update verify config generation failed upstream of final verify, which means we also didn't run update verify from 60.0 to 60.0.1.

The config gen failed because it didn't find 60.0esr, from [1]:
 INFO - retry: Calling _urlopen with args: (), kwargs: {'url': 'https://product-details.mozilla.org/1.0/firefox.json'}, attempt #1
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-9.0b6
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-9.0b5
....
 INFO - Skipping release that's behind the last watershed: firefox-7.0
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-61.0b5
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-61.0b4
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-61.0b3
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b9
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b8
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b7
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b6
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b5
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b4
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b3
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b16
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b15
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b14
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b13
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b12
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b11
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b10
 INFO - Skipping release that's behind the last watershed: firefox-60.0
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-6.0b5
 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-6.0b4
...
FATAL - Didn't find any update paths, cannot continue
FATAL - Running post_fatal callback...
FATAL - Exiting -1
 INFO - [mozharness: 2018-05-16 06:07:15.612524Z] Finished gather-info step (failed)

So this comes back to ship-it's product-details code.


[1] https://taskcluster-artifacts.net/cozae-D_QRizRojtryvL0A/0/public/logs/live_backing.log)
Summary: esr60 final update verification fails → esr60 final update verification fails, and update-verify doesn't run
Assignee: nobody → nthomas
Blocks: esr60
Merged and deployed to ship-it: https://github.com/mozilla-releng/ship-it/commit/90794c58730ec20d00a184399f12b04cadc45484

Munged up a copy of the 60.0.1esr graph for verify config and u.v. tasks at https://tools.taskcluster.net/groups/TiiSjWAFRluTqpVupwoZUQ but it's still failing.

Last watershed is set to 60.0esr so we get
[task 2018-05-24T00:12:26.283Z] 00:12:26     INFO - Skipping release that's behind the last watershed: firefox-60.0esr

Thinks 60.0.1 should be included but fails to look up shipped-locales:
[task 2018-05-24T00:12:26.285Z] 00:12:26     INFO - retry: Calling _urlopen with args: (), kwargs: {'url': 'https://archive.mozilla.org/pub/firefox/candidates/60.0.1-candidates/build1/win64_info.txt'}, attempt #1
[task 2018-05-24T00:12:26.866Z] 00:12:26     INFO - retry: Calling _urlopen with args: (), kwargs: {'url': 'https://hg.mozilla.org/releases/mozilla-esr60/raw-file/FIREFOX_60_0_1_RELEASE/browser/locales/shipped-locales'}, attempt #1
[task 2018-05-24T00:12:27.106Z] 00:12:27     INFO - retry: attempt #1 caught HTTPError exception: HTTP Error 404: Not Found
.... [run out of retries] ...
[task 2018-05-24T00:24:28.935Z] 00:24:28     INFO - [mozharness: 2018-05-24 00:24:28.935426Z] Finished gather-info step (failed)

Here's the call to the config creation script:

[task 2018-05-24T00:12:25.755Z] 00:12:25     INFO - Run as testing/mozharness/scripts/release/update-verify-config-creator.py --config internal_pypi.py --product firefox --stage-product firefox --app-name browser --platform win64 --to-version 60.0.1esr --to-app-version 60.0.1 --to-build-number 1 --to-buildid 20180516032417 --to-revision efbcc205a0d3562691bf62bb0cb55bb7e891e8ff --output-file update-verify.cfg --repo-path releases/mozilla-esr60 --partial-version 60.0esr --updater-platform linux-x86_64 --channel default --archive-prefix https://archive.mozilla.org/pub --aus-server https://aus5.mozilla.org --include-version ^\d+\.\d+(\.\d+)?$ --last-watershed 60.0esr

The default channel differs from what we do on other branches.
The Firefox 61.0b8 release tasks tried to include 60.0esr as well, so I've backed out the change in ship-it until I can up with a solution for the config generator and input args that works across all branches:

https://github.com/mozilla-releng/ship-it/commit/0097b55e7faae10ee2356a406e22ed557dfc5e71
This patch solves a few issues with ESR60
* we should include firefox-60.0esr in firefox.json in product-details, so that we can generate update verify configs including it
* if we do that we can't tell the difference between a 60.0 on release and esr, due to trimming the esr off the release name in the config gen script (eg, comment #4). This cross-talk will also bite us if we do a 60.0.2 on release or esr, because 60.0.1esr is already in firefox.json
* so leave the esr in the release name when looking for releases to consider, and the existing regexes will ignore esr releases on beta & release. Add a new regex to handle the esr case
* we also needed to set the channel to esr-localtest for update verify on esr60, otherwise it ends up as default

Testing:
* redid the 60.0.1esr update verify config and update verify jobs https://tools.taskcluster.net/groups/VrKJf36lQZegfM3ea84UUw (munged up graph/tasks from release, and a copy of firefox.json with 60.0esr added)
* local testing to make sure update verify configs from Fx & Deved 61.0b8, 60.0.1, 60.0 on beta & release are not modified

Deployment:
* land this patch everywhere. It'll need a merge on mozilla-esr60 because include-version is set to nonbeta there
* new PR to reland https://github.com/mozilla-releng/ship-it/pull/222 and push that out
Attachment #8981379 - Flags: review?(bhearsum)
Comment on attachment 8981379 [details] [diff] [review]
[gecko] Fix up ESR60 handling

Review of attachment 8981379 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for fixing this up.
Attachment #8981379 - Flags: review?(bhearsum) → review+
Pushed by nthomas@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7998b3ccc005
Update verify fixes for ESR60 branch, r=bhearsum  DONTBUILD
https://github.com/mozilla-releng/ship-it/pull/224 to re-add 60.0esr to firefox.json.
https://hg.mozilla.org/mozilla-central/rev/7998b3ccc005
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Merged #224 of shipit and deployed to stage+prod. We should be all done here.
You need to log in before you can comment on or make changes to this bug.