1343339 - Linux buildids are incorrect on mozilla-central nightlies

Status: RESOLVED INVALID

(Firefox Build System :: General, defect)

Description

[Bill Gianopoulos [:WG9s]]

This was filed from comments on bug 1342942.

Comment 1

[Bill Gianopoulos [:WG9s]]

As shown in bug 1342952 comment 16:

Nightlies have appeared on ftp.mozilla.org (at this bug's URL) dated 2017-02-28 for all five platforms, as follows:

linux-i686
20170228110308
https://hg.mozilla.org/mozilla-central/rev/1bc2ad020aee2830e0a7941f10958dbec108c254

linux-x86_64
20170228110308
https://hg.mozilla.org/mozilla-central/rev/1bc2ad020aee2830e0a7941f10958dbec108c254

mac
20170228030203
https://hg.mozilla.org/mozilla-central/rev/1bc2ad020aee2830e0a7941f10958dbec108c254

win32
20170228030203
https://hg.mozilla.org/mozilla-central/rev/1bc2ad020aee2830e0a7941f10958dbec108c254

win64
20170228030203

As you can see not only are the Linux bugs 8 hours int he future but the minutes and seconds components do not match those for the other platforms.

https://hg.mozilla.org/mozilla-central/rev/1bc2ad020aee2830e0a7941f10958dbec108c254

Comment 2

[Bill Gianopoulos [:WG9s]]

My understanding of how the official nightly builds are intended to work is that even accounting for incorrect timezone configuration, that would still leave a discrepancy between the correct buildid of 20170228030203 and 20170228030308.  It is my understanding that all official Firefox mozilla-central nightly builds for all platforms are supposed to be triggered by a single source that provides both the changset to use for checkout and the timestamp to use for the buildid.

Comment 3

[Bill Gianopoulos [:WG9s]]

One other comment copied from the other bug:

The buildid on official builds is supposed to be in mozilla standard time. not GMT or UTC or any other locale.

Comment 4

[Bill Gianopoulos [:WG9s]]

And also this:

Myself, my build systems are in the US/Eastern timezone, but to keep the buildids consistent with the official nightlies, I do a "export TZ=PST8PDT,M3.2.0,M11.1.0" in my build procedure.

I need the M3.2.0,M11.1.0 because of stupidities in Microsoft Windows not interpreting PST8PDT correctly.

Comment 5

[Bill Gianopoulos [:WG9s]]

(In reply to Bill Gianopoulos [:WG9s] from comment #4)
> And also this:
> 
> Myself, my build systems are in the US/Eastern timezone, but to keep the
> buildids consistent with the official nightlies, I do a "export
> TZ=PST8PDT,M3.2.0,M11.1.0" in my build procedure.
> 
> I need the M3.2.0,M11.1.0 because of stupidities in Microsoft Windows not
> interpreting PST8PDT correctly.

OR perhaps more accurately msys.

Comment 6

[Bill Gianopoulos [:WG9s]]

And I feel compelled, less I get stupid comments, that this issue predates the S3 issues.

Comment 7

[Chris AtLee [:catlee]]

Sorry, there's nothing to do here. buildids are not guaranteed to have any particular value, nor are they guaranteed to be the same between platforms. They just happened to have been using the time (in the Pacific time zone) of when the nightlies were scheduled. Now that we've started to migrate to TaskCluster, those builds are using a UTC based time. Once Windows and OSX switch over to TaskCluster as well, those platforms will share the same buildid as Linux most of the time.

Comment 8

[Bill Gianopoulos [:WG9s]]

It is fine if that is the direction going forward,  I actually filed a bug earlier to do this, but it got rejected.  go figure. I wanted to change mozilla standard time to be UTC time. to fix issues we had every daylight savings time change because nightlights were scheduled to run at 2 AM.

Comment 9

[Bill Gianopoulos [:WG9s]]

My bug was what resulted in the 3:02 AM build time.

Comment 10

[Bill Gianopoulos [:WG9s]]

Please point to a bug that supports this change from all builds have the same buildid before closing this bug as invalid.

Comment 11

[Tony Mechelynck [:tonymec]]

The advantage of having Build IDs reflect the build start time in some known timezone (traditionally California time) is that it allows humans a quick way of telling which of two builds was made before the other, and by how much. It also and for the same reason allows saying quickly whether the currently running nightly is out of date, and by how much.

There is already another ID which has "no particular value" and that is the changeset ID, which is a 40-nybble hash and thus cannot be pinpointed in time without going back to the pushlog. If the build ID is changed to some ID with no particular value, it is even less useful than the CSID, which at least can be found by means of standard hg commands in a Mercurial clone, and defines unequivocally exactly which mozilla-central source was used for the build.

Comment 12

[Bill Gianopoulos [:WG9s]]

I realize that but in the past an effort was made to synchronize these especially since the crash reporter likes to group things by buildid so that synchronizing buildid on different platforms make this behavior make sense if we are deviating from this then crash reports are going to be harder to debug.

Comment 13

[Justin Wood (:Callek)]

(In reply to Bill Gianopoulos [:WG9s] from comment #10)
> Please point to a bug that supports this change from all builds have the
> same buildid before closing this bug as invalid.

I don't have one (not sure if one existed). But this was a concious choice by releng. We consulted with Product owners and Release Management, and discussed with those involved with Socorro and related systems.

The bottom line is that much of the infrastructure that had hardcoded assumptions about buildID doesn't have that anymore, and that we are comfortable with the potential/likelihood/etc of these values being diverged.

There is no intended *direct* meaning behind BUILDID's these days other than "newer for newer code builds" (even different buildid's for the same code is 'ok')

The historic meaning, and timezone/time of day should not be assumed going forward.

If there is an actual reason this causes problems, other than merely human expectation/confusion, please do file a new bug, feel free to CC on these comments and we can revisit with whatever product/process owner to figure out the path forward.

Comment 14

[Bill Gianopoulos [:WG9s]]

Actually I prefer them to be in UTC rather than California time.  It prevents collisions that could occur on Fall Back day when the hour from 1AM to 2am is repeated.  Two builds with the same buildid from the same platform on same branch make the crash reporter assume they are the same build.