1315695 - build system doesn't cope with DST changes

Status: NEW

(Firefox Build System :: General, defect)

Description

[Chris AtLee [:catlee]]

Probably WONTFIX, but just in case this is a real bug...

We had an Aurora nightly build active during the DST change this weekend:
https://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-06-00-40-04-mozilla-aurora/mozilla-aurora-win64-nightly-bm72-build1-build6.txt.gz

The change happened between these lines:
01:55:06     INFO -  mozmake.EXE[6]: Leaving directory 'c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/obj-firefox/js/src/shell'
01:12:42     INFO -  warning C4742: '`string'' has different alignment in 'c:\builds\moz2_slave\m-aurora-w64-ntly-000000000000\build\src\obj-firefox\media\webrtc\trunk\webrtc\base\base_rtc_base_approved\unified_cpp_trunk_webrtc_base0.cpp' and 'c:\builds\moz2_slave\m-aurora-w64-ntly-000000000000\build\src\xpcom\components\nscategorymanager.cpp': 1 and 2

Somehow this resulted in xul.dll being twice as large as normal (about 133MB vs 65MB), and busted updates (bug 1315530).

Is this a windows-ism where file system times are stored as timezone-less times, and so switching from DST to standard time would break something in the build system?

Comment 1

[Gregory Szorc [:gps]]

Some more context:

01:52:59     INFO -  Generating code
01:52:59     INFO -  Finished generating code
01:52:59     INFO -  touch -t `date +%Y%m%d%H%M.%S -d 'now+5seconds'` pgo.relink
01:52:59     INFO -  c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/config/nsinstall.py -t -m 755 'gdb-tests.exe' '../../../dist/bin'
01:52:59     INFO -  mozmake.EXE[6]: Leaving directory 'c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/obj-firefox/js/src/gdb'
01:53:19     INFO -  Generating code
01:53:19     INFO -  Finished generating code
01:53:19     INFO -  touch -t `date +%Y%m%d%H%M.%S -d 'now+5seconds'` pgo.relink
01:53:19     INFO -  c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/config/nsinstall.py -t -m 755 'jsapi-tests.exe' '../../../dist/bin'
01:53:19     INFO -  mozmake.EXE[6]: Leaving directory 'c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/obj-firefox/js/src/jsapi-tests'
01:55:05     INFO -  fallible.obj : MSIL .netmodule or module compiled with /GL found; restarting link with /LTCG; add /LTCG to the link command line to improve linker performance
01:55:05     INFO -  Generating code
01:55:05     INFO -  Finished generating code
01:55:05     INFO -  touch -t `date +%Y%m%d%H%M.%S -d 'now+5seconds'` pgo.relink
01:55:05     INFO -  c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/config/nsinstall.py -t -m 755 'js.exe' '../../../dist/bin'
01:55:06     INFO -  mozmake.EXE[6]: Leaving directory 'c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/obj-firefox/js/src/shell'
\moz2_slave\m-aurora-w64-ntly-000000000000\build\src\xpcom\components\nscategorymanager.cpp': 1 and 2
01:12:42     INFO -     Creating library xul.lib and object xul.exp
01:12:42     INFO -  Generating code
01:12:42     INFO -  Finished generating code
01:12:42     INFO -  test "$(dumpbin -exports xul.dll | grep _NSModule@@ | sort -k 3 | sed -n 's/^.*?\([^@]*\)@@.*$/\1/;1p;$p' | xargs echo)" != "start_kPStaticModules_NSModule end_kPStaticModules_NSModule" && echo "NSModules are not ordered appropriately" && exit 1 || exit 0
01:12:43     INFO -  touch -t `date +%Y%m%d%H%M.%S -d 'now+5seconds'` pgo.relink
01:12:43     INFO -  chmod +x xul.dll
01:12:43     INFO -  c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/config/nsinstall.py -t -m 644 'xul.dll' '../../dist/bin'
01:12:43     INFO -  c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/m-aurora-w64-ntly-000000000000/build/src/config/nsinstall.py -t -m 644 'xul.lib' '../../dist/sdk/lib'

DST change happened in the middle of PGO.

The "touch -t `date +%Y%m%d%H%M.%S -d 'now+5seconds'`" seem highly relevant, as it is using a timezone-less value. `man touch` says -t only accepts a "[[CC]YY]MMDDhhmm[.ss]" value, so it just plain doesn't handle timezones.

The fix here is likely to change that `touch` to some Python code that does math on UNIX time integers.

Since this can impact releases, I'll code up a fix.

Comment 2

[Gregory Szorc [:gps]]

Attachment: Bug 1315695 - Use Python to adjust pgo.relink mtime;

`touch -t` may not correctly handle timezones across DST changes
because it accepts a timezone-less date string. Work around this by
using Python and UNIX timestamps to adjust the mtime of the
pgo.relink file.

If this doesn't fix the issue, then signs would seem to point to
a bug in MSVC's PGO process not working correctly when run across
timezones.

Review commit: https://reviewboard.mozilla.org/r/91136/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/91136/

Comment 3

[Bill Gianopoulos [:WG9s]]

Well I did suggest once in the past, that in order to be less US centric and embrace our international contributors rather than using Mountain View time as Mozilla Standard time, we switch to using UTC for our servers and therefore become immune form all lame Daylight Saving (Summertime for you UK brethren) transitions.

Comment 4

[Gregory Szorc [:gps]]

(In reply to Bill Gianopoulos [:WG9s] from comment #3)
> Well I did suggest once in the past, that in order to be less US centric and
> embrace our international contributors rather than using Mountain View time
> as Mozilla Standard time, we switch to using UTC for our servers and
> therefore become immune form all lame Daylight Saving (Summertime for you UK
> brethren) transitions.

It is a best practice to set server timezones to UTC. Pretty much every new server I see spun up does this. These machines, however, have their configuration rooted from the "dark ages" when practically everything at Mozilla was set to California time. Since these machines are being phased out in favor of TaskCluster, it probably doesn't make sense to change them.

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 8808286 [details]
Bug 1315695 - Use Python to adjust pgo.relink mtime;

https://reviewboard.mozilla.org/r/91136/#review93500

This is terrible, but the whole pgo.relink thing is sorta terrible.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a77fb6a85fc8
Use Python to adjust pgo.relink mtime; r=ted

Comment 7

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/530bbb41afc4
Backed out changeset a77fb6a85fc8 for breaking PGO

Comment 8

[Kai Engert (:KaiE:)]

NSPR is a separate product, and Firefox should only use released snapshots of NSPR.

Please don't make local changes to the copy of NSPR in directory /nsprpub (except when you make a try build).

Instead, please suggest your changes to the NSPR component, and ask for a new NSPR snapshot to get uplifted.

Comment 9

[Julien Cristau [:jcristau]]

Too late for firefox 52, mass-wontfix.

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months, so the assignee is being reset.