932211 - [Meta] Nightly builds should be identical to standard builds in automation

Status: RESOLVED DUPLICATE of bug 1352113

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

Time and time again we end up with some nightly-only bustage on mozilla-central (or even release branches) that did not occur on the standard dep/clobber builds on the same push. These are really not helpful since we then have a 24hour regression range, rather than per-push.

I would imagine these differences are due to:
* Differences in mozconfig.
* Build system conditionals that explicitly only enable certain features if the channel is set to nightly.

I imagine the reason for some of the above is due to concern that standard (non-nightly channel builds) either in automation or on a developers own machine, will end up submitting telemetry (or similar) to production, skewing results. However:
1) In automation, we should be substituting out production server URLs in about:config or else disabling the feature at runtime (so we can at least know it compiles).
2) To protect against submissions from local developer builds, we can just add something to the in-tree mozconfigs (that get used by both nightly and standard builds) that local the dev won't have on their local machine's mozconfig (eg like we do for crashreporter).

I'll file bugs under this one as and when we find instances to fix.

Comment 1

[Ed Morley [:emorley]]

We have just been bitten on aurora by nightly-only bustage (bug 932206).

Comment 2

[Ed Morley [:emorley]]

http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/updater/Makefile.in#103
100 ifneq (,$(filter beta release esr,$(MOZ_UPDATE_CHANNEL)))
101 RCFLAGS += -DMAR_SIGNING_RELEASE_BETA=1
102 else
103 ifneq (,$(filter nightly aurora nightly-elm nightly-profiling nightly-oak,$(MOZ_UPDATE_CHANNEL)))
104 RCFLAGS += -DMAR_SIGNING_AURORA_NIGHTLY=1
105 endif
106 endif

-> Seems fine, is only used here:
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/updater/updater.rc#51


http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/Makefile.in#5
5 ifeq (,$(filter aurora beta release esr,$(MOZ_UPDATE_CHANNEL)))
6 DEFINES += -DMOZ_COMPATIBILITY_NIGHTLY=1
7 endif

-> This treats nightly and standard builds the same \o/


http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/test/xpcshell/head_addons.js#35
35 function isNightlyChannel() {
36   var channel = "default";
37   try {
38     channel = Services.prefs.getCharPref("app.update.channel");
39   }
40   catch (e) { }
41 
42   return channel != "aurora" && channel != "beta" && channel != "release" && channel != "esr";
43 }

-> This treats nightly and standard builds the same \o/


http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/test/browser/head.js#45
45   if (channel != "aurora" &&
46     channel != "beta" &&
47     channel != "release") {
48     var version = "nightly";
49   } else {

-> This treats nightly and standard builds the same \o/ (although it doesn't check for esr?)


http://mxr.mozilla.org/mozilla-central/source/browser/components/search/test/browser_google.js#25
23 #if MOZ_UPDATE_CHANNEL == beta
24 const GOOGLE_CLIENT = "firefox-beta";
25 #elif MOZ_UPDATE_CHANNEL == aurora
26 const GOOGLE_CLIENT = "firefox-aurora";
27 #elif MOZ_UPDATE_CHANNEL == nightly
28 const GOOGLE_CLIENT = "firefox-nightly";
29 #else
30 const GOOGLE_CLIENT = "firefox-a";
31 #endif

-> This only affects what is submitted to Google (http://mxr.mozilla.org/mozilla-central/source/browser/components/search/test/browser_google.js#141) so this is fine.


http://mxr.mozilla.org/mozilla-central/source/accessible/src/shared.mozbuild#11
11 if CONFIG['MOZ_UPDATE_CHANNEL'] not in ('aurora', 'beta', 'release', 'esr'):
12     a11y_log = 1

-> This treats nightly and standard builds the same \o/


http://mxr.mozilla.org/mozilla-central/source/accessible/src/Makefile.in#9
9 ifeq (,$(filter aurora beta release esr,$(MOZ_UPDATE_CHANNEL)))
10   A11Y_LOG = 1

-> This treats nightly and standard builds the same \o/


http://mxr.mozilla.org/mozilla-central/source/toolkit/modules/debug.js#42
40   try {
41     switch (defB.getCharPref("app.update.channel")) {
42       case "nightly":
43       case "beta":
44       case "default":
45         releaseBuild = false;
46     }
47   } catch(ex) {}

-> Treats all channels the same anyway ?!


http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/content/updates.js#959
959     if (update.channel == "nightly") {
960       updateName = gUpdates.getAUSString("updateNightlyName",
961                                          [gUpdates.brandName,
962                                           update.displayVersion,
963                                           update.buildID]);
964     }

-> Guess this is fine, since the updater isn't used in standard builds & this is UI string only.


http://mxr.mozilla.org/mozilla-central/source/configure.in#3608
3613 case "$GRE_MILESTONE" in
3614   *a1*)
3615       NIGHTLY_BUILD=1
3616       AC_DEFINE(NIGHTLY_BUILD)
3617       ;;
3618   *a*)
3619       ;;
3620   *)
3621       RELEASE_BUILD=1
3622       AC_DEFINE(RELEASE_BUILD)
3623       ;;
3624 esac

-> This is fine, since both nightly and standard builds will have the same version number :-)


http://mxr.mozilla.org/mozilla-central/source/browser/confvars.sh#15
15     if test "$MOZ_UPDATE_CHANNEL" = "nightly" -o \
16             "$MOZ_UPDATE_CHANNEL" = "aurora" -o \
17             "$MOZ_UPDATE_CHANNEL" = "beta" -o \
18             "$MOZ_UPDATE_CHANNEL" = "release"; then
19       if ! test "$MOZ_DEBUG"; then
20         MOZ_STUB_INSTALLER=1
21       fi
22     fi

-> This treats nightly and standard builds differently.

Comment 3

[[e-mail] (:bhearsum)]

There's some things on the Buildbot that are different too. Here's my (hopefully complete) list below, with comments:
- Signing servers used (dep vs nightly). We're not allowed to change this per opsec policy.
- Update generation. Doesn't make sense to do this on dep builds.
- L10n repack triggering. Doesn't make sense to do this on dep builds.
- Build symbol uploading. Probably can't do this for disk space reasons.

Comment 4

[Ed Morley [:emorley]]

Thank you :-)

Also the reason for the recent aurora nightly only bustage, is that on release branches the dep builds = channel default, whereas the nightly = channel {aurora,beta,release,esr}. This means that we actually inadvertently switch some trunk-only features back on for release branch dep build - ooops.

Assigning this to me to track things down/file bugs under this for fixing each case.

Comment 5

[Aki Sasaki (not active)]

I believe we have two conflicting questions:

- is my commit good (prioritize speed; dep / ci)
  - depend
  - non-pgo
  - non-updatable
  - generic branding
- is this binary shippable (prioritize correctness; nightly / release)
  - clobber
  - mozconfig
  - updatable
  - branding
  - pgo
  - nightly/release promotable

I'm currently leaning towards:

- dep/ci on try and all integration branches
- pgo/nightly/release builds become "shippable" builds, which can be backfilled easily.
  - these run periodically on integration, are choosable on try
  - train branches don't run dep/ci at all for opt, only shippable builds.

Comment 6

[u589863]

(In reply to Aki Sasaki [:aki] from comment #5)
> I believe we have two conflicting questions:
> 
> - is my commit good (prioritize speed; dep / ci)
>   - depend
>   - non-pgo
>   - non-updatable
>   - generic branding
> - is this binary shippable (prioritize correctness; nightly / release)
>   - clobber
>   - mozconfig
>   - updatable
>   - branding
>   - pgo
>   - nightly/release promotable
> 
> I'm currently leaning towards:
> 
> - dep/ci on try and all integration branches
> - pgo/nightly/release builds become "shippable" builds, which can be
> backfilled easily.
>   - these run periodically on integration, are choosable on try
>   - train branches don't run dep/ci at all for opt, only shippable builds.

So when will this bug be landing and ready for testing?

Comment 7

[Chris AtLee [:catlee]]

I think this is addressed by the shippable builds proposal in bug 1352113.