1347107 - Different BuildIDs for Firefox 53 Beta, for Linux VS. Mac and Win

Status: RESOLVED WORKSFORME

(Release Engineering :: Release Automation, enhancement)

Description

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

For Firefox 53 Beta 1 and Beta 2, we got different BuildIDs for Linux builds versus Mac and Windows builds:
- 53 Beta 2 - Linux (x86, x64) - "buildid": "20170313224914"
- 53 Beta 2 - Mac & Win (x86, x64) - "buildid": "20170313154936"

* for 53 Beta 1 * 
- 53 Beta 1 - Linux (x86, x64) - "buildid": "20170307144748"
- 53 Beta 1 - Mac & Win (x86, x64) - "buildid": "20170307064827"

The difference in BuildIDs causes the ondemand update tests to fail for Mac and Windows, because they use the Linux BuildID to check that the correct version is installed after update.

Comment 1

[Johan Lorenzo [:jlorenzo]]

After some investigation, this is related to tc-migration. Firefox 53.0 on linux is the first version entirely built on Taskcluster.

The main problem is the following: Taskcluster and Buildbot both generate a BuildID on the CI builds. This hasn't been an issue before, because builds weren't promoted until they reached Beta.

More precisely, we pass different values to the task parameters
* On Buildbot, BuildIDs are passed via the parameter `buildid`: https://treeherder.mozilla.org/logviewer.html#?job_id=83504342&repo=mozilla-beta&lineNumber=5 
* On Tasluster, CI builds gets their build IDs via `MOZ_BUILD_DATE`: https://treeherder.mozilla.org/logviewer.html#?job_id=83504374&repo=mozilla-beta&lineNumber=517 . As you can see it's slightly different from the one just above

Back when Linux was built on Buildbot, they did share the same values:
* https://treeherder.mozilla.org/logviewer.html#?job_id=83523540&repo=mozilla-release&lineNumber=5
* https://treeherder.mozilla.org/logviewer.html#?job_id=83523566&repo=mozilla-release&lineNumber=5

The fix should be to pass the value in `buildid` to `MOZ_BUILD_DATE`. Callek, do you think that's a good approach?

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

I might be off the tracks, but I think this is somehow expected. Back in the 53 cycle on mozilla-central when we switched the nightlies for Linux platforms from BB to Taskcluster, the triggering/schedule also changed. For Firefox desktop, we'd have two sets of nightlies being generated - the old ones off of BB for Mac/Windows and the new TC-based ones for Linux. The same happened when 53 hit aurora during following cycle. You can see for example that the nightlies running for central/aurora have different buildIds across different timings and there are separte entries for Mac/Win vs Linux. Sample for 2017-03-10:

Mac/Win:
http://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-10-03-02-05-mozilla-central/
http://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-10-06-54-28-mozilla-central/
http://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-10-09-01-16-mozilla-central/

Linux
http://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-10-11-02-48-mozilla-central/
http://archive.mozilla.org/pub/firefox/nightly/2017/03/2017-03-10-14-55-45-mozilla-central/ 

I think this happens because the scheduling mechanism is different too. For Mac/Win we use the old way of scheduling things whereas for TC builds we use the TC hooks via cron.yml jobs. Also, even though they are scheduled at the same time (give or take few mins), BB jobs are PST whilst TC builds are UTC, hence the similarity between the buildIs, sans the 8h difference in hours time.

Coming back to original question, I think this buildID difference might just be the new "standard" in the new workd where TC-based builds are promoted. Until we switch mac/win to Taskcluster as well.
Hopefully I'm right and I'm not adding more confusion to this. Would love to have a confirmation on this.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Please also see the following issues for mozmill-ci with the different file naming schemes on archive.mozilla.org:
https://github.com/mozilla/mozmill-ci/issues/828

If it turns out we continue as of now with different build ids we will have to get a fix in place for it.

Comment 4

[Rail Aliiev [:rail]]

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #0)
> The difference in BuildIDs causes the ondemand update tests to fail for Mac
> and Windows, because they use the Linux BuildID to check that the correct
> version is installed after update.

Can we fix this assumption? Is this something the jenkins wrapper does?

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The following PR for mozdownload will fix the retrieval of the json file from the release candidates folder for Windows and Mac: https://github.com/mozilla/mozdownload/pull/429

I will have to get a new version released and then check with mozmill-ci if some more work is necessary in this tool.

Comment 6

[Rail Aliiev [:rail]]

Thank you! \o/

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

A fix for mozmill-ci should be in place later today. Once Florin run some tests in staging, we could also deploy to production later.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It's on staging now. Florin please respond and let us know if all works well. Thanks.

Comment 9

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

(In reply to Henrik Skupin (:whimboo) from comment #8)
> It's on staging now. Florin please respond and let us know if all works
> well. Thanks.

Tested on staging, and we no longer have an issue with this, as the tests pass now - https://treeherder.allizom.org/#/jobs?repo=mozilla-beta&revision=a5ccc32310c9&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta(

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks Florin. I deployed the code changes to production so that we are ready for the beta 3 tomorrow.

Closing as WFM given that the issue was outside of RelEng.