Update verify tests: pass OS versions to balrog URLs

RESOLVED FIXED

Status

Release Engineering
Release Automation
RESOLVED FIXED
16 days ago
15 days ago

People

(Reporter: jlorenzo, Assigned: bhearsum)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

16 days ago
57.0 has a lot of rules[1] that depend on what OS Firefox runs on. At the moment, UV tests are failing, because the URL used defines "default" as the OS version[2]. "default" is not a supported keyword in our balrog rules. Then we need to change this URL[3] to be either Linux, Darwin, or Windows_NT (instead of default).

[1] https://docs.google.com/spreadsheets/d/17u3x092rGPB0rTwManlJ3nN2WdNwy2DgXifW8RwtpFA/edit#gid=0
[2] For instance:  https://aus5.mozilla.org/update/3/Firefox/55.0.3/20170824053838/Darwin_x86_64-gcc3-u-i386-x86_64/sr/release-localtest/default/default/default/update.xml?force=1 . The OS Version is the "default" right after "release-localtest"[2a].
[2a] https://github.com/mozilla/balrog/blob/814a66e17b57f522a1259da4052cc6bc5cb8311e/auslib/web/public/swagger/api.yml#L73
[3] https://github.com/mozilla/build-tools/blob/409cce5a0aa943b52dbfb4c67c824bc561806fc5/release/updates/verify.sh#L127
(Reporter)

Updated

16 days ago
Blocks: 1415172
(Assignee)

Comment 1

16 days ago
Created attachment 8926454 [details] [diff] [review]
set OS_VERSION in update verify; update release configs to remove watersheds

This is a hack, but it seems to work. I haven't been able to test on OS X or Windows (because they run in buildbot, and I can't force them to use my tools repo).

Because OS_VERSION causes us to hit different rules in Balrog, I've had to alter the configs to remove all of the Linux & Windows previous versions locked behind a watershed (eg: everything below 56.0 on windows stops at 56.0). This patch is probably also going to cause other branches to fail similarly, so my suggestion is to land this, re-tag for the release you want to test (eg: FIREFOX_57_0_RELEASE), and backout immediately. This will need to be done for each RC, too.
Attachment #8926454 - Flags: review?(jlorenzo)
(Reporter)

Comment 2

16 days ago
Comment on attachment 8926454 [details] [diff] [review]
set OS_VERSION in update verify; update release configs to remove watersheds

Review of attachment 8926454 [details] [diff] [review]:
-----------------------------------------------------------------

Good to see some old update paths cleaned up. r+ modulo the question on 55.0.3.

::: release/updates/release-firefox-win32.cfg
@@ -1,4 @@
>  release="56.0.2" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20171024165158" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete partial" from="/firefox/releases/56.0.2/win32/%locale%/Firefox Setup 56.0.2.exe" aus_server="https://aus5.mozilla.org" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub" to="/firefox/candidates/57.0-candidates/build1/win32/%locale%/Firefox Setup 57.0.exe" to_build_id="20171106194249" to_display_version="57.0" to_app_version="57.0"
>  release="56.0.1" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20171002220106" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da dsb el en-GB en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete"
>  release="56.0" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20170926190823" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete partial" from="/firefox/releases/56.0/win32/%locale%/Firefox Setup 56.0.exe" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub"
> -release="55.0.3" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20170824053622" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete partial" from="/firefox/releases/55.0.3/win32/%locale%/Firefox Setup 55.0.3.exe" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub" to_build_id="20170926190823" to_display_version="56.0" to_app_version="56.0"

Should we keep this one and make it point to 56.0? We changed that balrog rule to apply to Windows only. It's a good sanity check to do, IMO.

::: release/updates/release-firefox-win64.cfg
@@ -1,4 @@
>  release="56.0.2" product="Firefox" platform="WINNT_x86_64-msvc-x64" build_id="20171024165158" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete partial" from="/firefox/releases/56.0.2/win64/%locale%/Firefox Setup 56.0.2.exe" aus_server="https://aus5.mozilla.org" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub" to="/firefox/candidates/57.0-candidates/build1/win64/%locale%/Firefox Setup 57.0.exe" to_build_id="20171106194249" to_display_version="57.0" to_app_version="57.0"
>  release="56.0.1" product="Firefox" platform="WINNT_x86_64-msvc-x64" build_id="20171002220106" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da dsb el en-GB en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete"
>  release="56.0" product="Firefox" platform="WINNT_x86_64-msvc-x64" build_id="20170926190823" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete partial" from="/firefox/releases/56.0/win64/%locale%/Firefox Setup 56.0.exe" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub"
> -release="55.0.3" product="Firefox" platform="WINNT_x86_64-msvc-x64" build_id="20170824053622" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete partial" from="/firefox/releases/55.0.3/win64/%locale%/Firefox Setup 55.0.3.exe" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub" to_build_id="20170926190823" to_display_version="56.0" to_app_version="56.0"

Same comment as win32

::: release/updates/verify.sh
@@ +105,5 @@
>  
> +  os_version="default"
> +  case `echo $platform | cut -c-12` in
> +    Linux*)
> +      os_version="Linux%204.4.0-83-generic%20(GTK 3.18.9,libpulse 8.0.0)"

Already discussed on IRC: Should we make this os_version more generic? I'm afraid we may write a rule for GTK, and the test will fire back at us.

That's not a blocker for 57.0, though.
Attachment #8926454 - Flags: review?(jlorenzo) → review+
(Assignee)

Comment 3

16 days ago
(In reply to Johan Lorenzo [:jlorenzo] from comment #2)
> Comment on attachment 8926454 [details] [diff] [review]
> set OS_VERSION in update verify; update release configs to remove watersheds
> 
> Review of attachment 8926454 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Good to see some old update paths cleaned up. r+ modulo the question on
> 55.0.3.
> 
> ::: release/updates/release-firefox-win32.cfg
> @@ -1,4 @@
> >  release="56.0.2" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20171024165158" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete partial" from="/firefox/releases/56.0.2/win32/%locale%/Firefox Setup 56.0.2.exe" aus_server="https://aus5.mozilla.org" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub" to="/firefox/candidates/57.0-candidates/build1/win32/%locale%/Firefox Setup 57.0.exe" to_build_id="20171106194249" to_display_version="57.0" to_app_version="57.0"
> >  release="56.0.1" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20171002220106" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da dsb el en-GB en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete"
> >  release="56.0" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20170926190823" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete partial" from="/firefox/releases/56.0/win32/%locale%/Firefox Setup 56.0.exe" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub"
> > -release="55.0.3" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20170824053622" locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk km kn ko lij lt lv mai mk ml mr ms my nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk ur uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete partial" from="/firefox/releases/55.0.3/win32/%locale%/Firefox Setup 55.0.3.exe" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub" to_build_id="20170926190823" to_display_version="56.0" to_app_version="56.0"
> 
> Should we keep this one and make it point to 56.0? We changed that balrog
> rule to apply to Windows only. It's a good sanity check to do, IMO.

This won't work, unfortunately. "to" is a "global" key to the code that does the chunking, so you can't have different values per line (https://hg.mozilla.org/build/tools/file/tip/lib/python/release/updates/verify.py#l17).

The purpose of update verify is to ensure that the MARs apply correctly, so I don't think it's important that we tested updates *to* 56.0 in it. My update status checks should verify the <=55.0.3 -> 56.0 update paths (and we verified that those MARs apply correctly when we shipped 56.0).

> ::: release/updates/verify.sh
> @@ +105,5 @@
> >  
> > +  os_version="default"
> > +  case `echo $platform | cut -c-12` in
> > +    Linux*)
> > +      os_version="Linux%204.4.0-83-generic%20(GTK 3.18.9,libpulse 8.0.0)"
> 
> Already discussed on IRC: Should we make this os_version more generic? I'm
> afraid we may write a rule for GTK, and the test will fire back at us.
> 
> That's not a blocker for 57.0, though.

Let's postpone this discussion :). I think it is related to other discussions about what we want to do with overall update testing.
(Assignee)

Comment 4

16 days ago
Comment on attachment 8926454 [details] [diff] [review]
set OS_VERSION in update verify; update release configs to remove watersheds

➜  tools hg out -r upstream ssh://hg.mozilla.org/build/tools
comparing with ssh://hg.mozilla.org/build/tools
searching for changes
changeset:   8100:a35b0a76164b
tag:         FIREFOX_57_0_BUILD1
tag:         FIREFOX_57_0_RELEASE
parent:      8096:9a582d523507
user:        Ben Hearsum <bhearsum@mozilla.com>
date:        Wed Nov 08 12:52:54 2017 -0500
summary:     bug 1415557: set OS_VERSION in update verify; update release configs to remove watersheds. r=jlorenzo

changeset:   8101:60ffcfd939d9
user:        Ben Hearsum <bhearsum@mozilla.com>
date:        Wed Nov 08 12:53:08 2017 -0500
summary:     Added tag FIREFOX_57_0_BUILD1 for changeset a35b0a76164b

changeset:   8102:bb8713ddb389
user:        Ben Hearsum <bhearsum@mozilla.com>
date:        Wed Nov 08 12:53:11 2017 -0500
summary:     Added tag FIREFOX_57_0_RELEASE for changeset a35b0a76164b

changeset:   8103:932786cb9755
bookmark:    upstream
tag:         tip
user:        Ben Hearsum <bhearsum@mozilla.com>
date:        Wed Nov 08 12:54:11 2017 -0500
summary:     Backout bug 1415557.

➜  tools hg push -r upstream ssh://hg.mozilla.org/build/tools
pushing to ssh://hg.mozilla.org/build/tools
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 4 changesets with 12 changes to 6 files
remote: recorded push in pushlog
remote: 
remote: View your changes here:
remote:   https://hg.mozilla.org/build/tools/rev/a35b0a76164bd1b6f6cfef4049e3f3b06f87bc46
remote:   https://hg.mozilla.org/build/tools/rev/60ffcfd939d98be9b7c76024c2ebd6b7b1b59e96
remote:   https://hg.mozilla.org/build/tools/rev/bb8713ddb389b28a374629b4a9dfe827687a40f4
remote:   https://hg.mozilla.org/build/tools/rev/932786cb9755d72b92b130607902d8b4c01326e8
remote: recorded changegroup in replication log in 0.002s
Attachment #8926454 - Flags: checked-in+
(Assignee)

Comment 5

16 days ago
Created attachment 8926513 [details] [diff] [review]
set OS_VERSION & SYSTEM_CAPABILITIES

The previous patch mostly worked - except it ended up uncovering a subtle bug in Balrog (https://bugzilla.mozilla.org/show_bug.cgi?id=1415649). The problem occurs because update verify sends what is not really a valid update URL -- it uses /update/3 with 56.0.1, which means SYSTEM_CAPABILITIES is not in the update URL. Because some of the Rules we're using are looking for JAWS and memory fields, we end up with an internal server error.

We can workaround this for now by using /update/6 for update verify. As with the previous patch, my intent is to land, retag, and backout immediately.
Attachment #8926513 - Flags: review?(bugspam.Callek)

Updated

16 days ago
Attachment #8926513 - Flags: review?(bugspam.Callek) → review+
(Assignee)

Comment 6

15 days ago
With my second patch we managed to get full green update verify for 57.0 RC1. We'll need to do another land/tag/backout cycle for any additional RCs. The update verify config diffs probably won't apply correctly - but the removals are easy enough to do by hand.
Status: NEW → RESOLVED
Last Resolved: 15 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.