1174174 - Use --update-target-buildid for the Firefox UI updates

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[Armen [:armenzg]]

This is a follow up from bug 1148546.

When we test Firefox UI update tests, we simply test that we've updated and not that we've updated to the right build. Passing the extra parameter of --update-target-buildid would take care of it.

We can probably simply do something like this in here: [1]
-  retcode = self._run_test(installer_path, self.channel)
+ retcode = self._run_test(
+       installer_path=installer_path,
+       update_channel=self.channel,
+       build_id=rel_info['build_id'])

http://hg.mozilla.org/build/mozharness/file/ecd1d24223d3/scripts/firefox_ui_updates.py#l315

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This is definitely a blocker for us before we can stop running the Mozmill/Marionette tests on our own.

Comment 2

[Armen [:armenzg]]

Ben, out of curiosity, do we test that updates point users to the right target? (with the headless tests)

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Armen Zambrano Gasparnian [:armenzg] from comment #2)
> Ben, out of curiosity, do we test that updates point users to the right
> target? (with the headless tests)

Yes, but in a different way than looking at the buildid. After applying the update we diff that against the unpacked installer. This checks that user got the right update, and also that applying the MAR results in the same appdir as a fresh install.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So I had a look into the mozharness script and the CLI option is supported! 

python scripts/firefox_ui_updates.py --firefox-ui-branch master --tools-tageloper_config.py --installer-path firefox.tar.bz2 --update-target-buildid 20150821030204

Armen, so this is only a question in how you invoke the script on the build machine right now. Can you point me to that code? Does it include the --update-target-buildid option and the appropriate buildid?

Comment 5

[Armen [:armenzg]]

Yes, finding what value to pass is the difficult part.
I assume we want to pass the buildid of the release which is currently being built and the one we're updating people to.
However, the tools config files, IIUC, the most recent buildid they contain is the buildid of the previous release rather the one being generated (see line 1):
http://hg.mozilla.org/build/tools/file/default/release/updates/mozBeta-firefox-linux64.cfg

I also assume that not all locales and all platforms are supposed to upload to the latest release.

The code where we call fx ui tests is in here:
http://hg.mozilla.org/build/buildbotcustom/file/default/process/release.py#l1328
Notice that when we try to schedule on testers we might not use that code path.

There's buildid values flying around that file, however, I wouldn't want to use those values before talking it with bhearsum.

whimboo: does this make sense?

Comment 6

[Robert Kaiser]

(In reply to Armen Zambrano Gasparnian [:armenzg] from comment #5)
> I also assume that not all locales and all platforms are supposed to upload
> to the latest release.

Wrong assumption, then. In fact all supported locales and platforms are required to update to the same build ID of the new build.

Comment 7

[Armen [:armenzg]]

(In reply to Robert Kaiser (:[e-mail]) from comment #6)
> (In reply to Armen Zambrano Gasparnian [:armenzg] from comment #5)
> > I also assume that not all locales and all platforms are supposed to upload
> > to the latest release.
> 
> Wrong assumption, then. In fact all supported locales and platforms are
> required to update to the same build ID of the new build.

Don't we sometimes make a release only for Windows? Or certain locales we skip them on a beta because they're not ready?
I'm happy to be wrong but I thought this was the case.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Whereby it can happen that platforms can have different buildids. But then as Robert said, all locales will have the same buildid for the same platform. Btw. it may also be good to get failures for those locales which are no longer supported in the new version, or?

Comment 9

[Robert Kaiser]

(In reply to Armen Zambrano Gasparnian [:armenzg] from comment #7)
> Don't we sometimes make a release only for Windows? Or certain locales we
> skip them on a beta because they're not ready?

We always build for all of them. If we do not put the updates up for some platform or locale, we actually *want* to see the failure as it's confirmation that we are off the norm (and in the correct way).

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We never finished the project and already landed code will be removed. If we still need such a thing we would implement it via jobs in TC nowadays.