386338 - Diff Release mozconfig with depend/nightly mozconfigs at start of release automation

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Currently, we bring forward a version of mozconfig from a release branch, and manually merge in any changes from what is used for nightlies.

Can we just use what is in nightly, and not manually change/update anything?

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

1) I'd like to review whats used by nightlies vs release, and for any differences, confirm if they are really intentional differences.

2) for any differences that are really needed, if we can think of some way to deal with them programatically, and not keep needing manual reviews, that would be great.

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to comment #1)
> 1) I'd like to review whats used by nightlies vs release, and for any
> differences, confirm if they are really intentional differences.

We should do this on every release.

> 2) for any differences that are really needed, if we can think of some way to
> deal with them programatically, and not keep needing manual reviews, that would
> be great.

There are several things that must be different, e.g. for an alpha:

ac_add_options --enable-update-channel=beta
ac_add_options --with-branding=browser/branding/unofficial
mk_add_options MOZ_MAKE_FLAGS="-j1"

For a nightly:

# tests, e.g.:
ac_add_options --enable-codesighs
ac_add_options --enable-update-channel=nightly
mk_add_options MOZ_MAKE_FLAGS="-j5"


We have automated the whole tinder-config/version bumping process (using Bootstrap/Config->Bump()), so it's certainly doable. So there are intentional differences, and it doesn't take a lot of time so I'm not sure how worth automating it is right now, compared to the higher-profile items on our plate.

Just to be clear though, it was a conscious decision to split the release and nightly configs. The old way of doing this was to edit the nightly config, make some changes, and change it back when done (of course something would always be missed and the first nightly would always be busted; or we'd miss something in the release).

Comment 3

[Benjamin Smedberg]

Because mozconfig is just a shell script, we could theoretically do something like this:

if test -n "$MOZ_RELEASE_FOR_REAL";
mk_add_options MOZ_MAKE_FLAGS="-j1"
else
mk_add_options MOZ_MAKE_FLAGS="-j4"
fi

However, you still have the problem then of telling "something" to set MOZ_RELEASE_FOR_REAL.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

During F3.0b3, we just now discovered that rc1 was built with a mozconfig out of sync with the nightly mozconfig. The nightly mozconfig had been modified on 13dec07 as part of bug#407794, and we never noticed to make the corresponding change to the release machines.

For now, we have added a manual "diff the mozconfig" step to the manual build instructions before starting a build. However, generating release mozconfig programatically from the nightly mozconfig would have avoided this problem, and avoided the entire respin.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

From triage meeting, we're uncomfortable with auto-generating mozconfig. 

However, it seems useful to:
- diff release mozconfig with default/nightly mozconfig
- exclude any expected/known differences
- fail out if there are any remaining differences, as these are by definition unexpected.

Tweaking summary to match.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found during triage. Even though we dont hit this often, its bad news when we
do, so this still feels like a valid bug.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Tweaked summary to clarify.

Basically, we want to make sure that any changes to depend/nightly mozconfigs
are also in the release mozconfigs. There are some known diffs, but unexpected
diffs should be flagged before the release builds start.

Comment 8

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

triaged - p5

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: proof of concept script to diff mozconfigs

This program takes two files and diffs them.  It ignores things that are be known to be different on a line by line basis.  This script does not currently understand sourcing other mozconfigs and I don't know if it should because that would require moving both mozconfigs to the topsrcdir in order to source relative file paths properly or having a full mozilla-central checkout present.  For mozconfigs that have logic (e.g. tryserver and universal build mozconfigs) it will do a line by line diff.  Two methods for accounting for this are to set the allowed_differences list to have the entire blob of script, or run it in a shell.  If we do decide to run the mozconfig through a shell, we can define 

function ac_add_option {
  echo ac_add_option $@
}

function mk_add_option {
  echo mk_add_option $@
}

to get the processed mozconfig output.  This is out of scope for the purpose of this bug (nightly vs release).

Sample output:
python diff-mozconfig.py buildbot-configs/mozilla2/win32/mozilla-central/nightly/mozconfig buildbot-configs/mozilla2/win32/mozilla-central/release/mozconfig 
There are issues that need investigation:
  release: |. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags|


Things that still need to be done:
-OptionParser to make the script nicer to use
-Unit tests -- seems important
-hook into release process
-potentially break the large diff_files into smaller functions (low priority)

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: updates

jhford-wifi:diff-mozconfigs jhford$ python diff-mozconfig.py buildbot-configs/mozilla2/linux/mozilla-central/nightly/mozconfig buildbot-configs/mozilla2/linux/mozilla-central/release/mozconfig -l nightly -r release
There are issues that need investigation:
  only in nightly: |ac_add_options --enable-codesighs|
  only in nightly: |CC="ccache /tools/gcc-4.3.3/installed/bin/gcc"|
  only in nightly: |CXX="ccache /tools/gcc-4.3.3/installed/bin/g++"|
  only in nightly: |# more options for 1.9 vs mozilla-central perf comparisons|
  only in nightly: |# shouldn't do anything - we don't do profiled builds on linux|
  only in nightly: |# Enable parallel compiling|
  only in release: |CC=/tools/gcc-4.3.3/installed/bin/gcc|
  only in release: |CXX=/tools/gcc-4.3.3/installed/bin/g++|

We could fix this by using something like:

REAL_CC=/tools/gcc-4.3.3/installed/bin/gcc
CC="ccache $REAL_CC"
REAL_CXX=/tools/gcc-4.3.3/installed/bin/g++
CXX="ccache $REAL_CXX"

in the nightly, and

REAL_CC=/tools/gcc-4.3.3/installed/bin/gcc
CC="$REAL_CC"
REAL_CXX=/tools/gcc-4.3.3/installed/bin/g++
CXX="$REAL_CXX"

in the release mozconfig.  This means that we can ignore the difference between CC="REAL_CC" and CC="ccache REAL_CC" and the same for CXX.  This seperates the differences on a line by line basis between which compiler we use and how it is invoked.

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

Here are some issues that I have found (diffing release vs nightly):

|--enable-codesighs| is only on linux,linux64,macosx,macosx64 nightlies for mozilla-central, mozilla-1.9.2 and mozilla-1.9.1

|--disable-install-strip| is only on macosx,macosx64 nightlies for mozilla-central, mozilla-1.9.2

|. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags| is only on win32 nightlies for mozilla-central,mozilla-1.9.2 and mozilla-1.9.1

|ac_add_options --enable-debug-symbols="-gdwarf-2"| is only on macosx mozilla-1.9.2 nightlies

|ac_add_options --with-crashreporter-enable-percent=10| is only on mozilla-1.9.1 win32 release builds

|ac_add_options --disable-tests| is only on win32 mozilla-1.9.1 release builds.

Not sure which of these differences are harmless and which (if any) are actual differences.  I can account for the differences in my patch if they are valid differences.

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

also

|export ac_cv_have_visibility_class_bug=yes| is only on mozilla-central linux64 nightly builds

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbot-configs patch v1

This patch adds a mozconfig verification script to the buildbot-configs repository.  It is currently only useful for comparing nightly and release mozconfigs.  If desired, a new bug to do the same for tryserver and nightly/dep builds could be done.

I am fine with moving the ok_diffs*.dat files somewhere else if it is felt to be an issue.  Ideally, the reason for the ok_diffs*.dat files should be resolved.  I can write a README document if desired.

Comment 15

[John Ford [:jhford] CET/CEST Berlin Time]

thanks aki, for pointing out the change to let the build system use ccache

http://mxr.mozilla.org/mozilla-central/source/configure.in#7428

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

which is defined by http://mxr.mozilla.org/mozilla-central/source/build/autoconf/altoptions.m4#108 and looking at configure.in looks like there is a safe fallback for ccache.

filed bug 590607 to track switching our mozconfigs to use --with-cache=<wherever it is> instead of CC='ccache gcc...'

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 465009 [details] [diff] [review]
buildbot-configs patch v1

These comments apply to verify_mozconfigs.py only, since the configs and mozconfigs are going to change for ccache.

>diff --git a/verify_mozconfigs.py b/verify_mozconfigs.py
>+        elif i.startswith('? '):
>+            not_in_either.append(i)

Uh, what's this ?

>+    def find_allowed(file_list, file_title):
>+        for f in file_list:

These are really matching lines, so using file_list and f is a little confusing.

>+    allowed_differences = []
>+    allowed_differences.extend(parse_dat_file('ok_diffs.dat'))
>+    for b in branches:
>+        allowed_differences.extend(parse_dat_file('ok_diffs_%s.dat' % b))
>+    for p in platforms:
>+        allowed_differences.extend(parse_dat_file('ok_diffs_%s.dat' % p))
>+    for b in branches:
>+        for p in platforms:
>+            allowed_differences.extend(parse_dat_file('ok_diffs_%s_%s.dat' % (b,p)))

This appears to remove the connection with the specific branch or platform, so that any rule may apply to any mozconfig. Surely not what you intended.

>+    except AssertionError, e:
>+        print 'ERROR: ', e.args[0]

Never actually write out the differences in non-verbose mode ?

Comment 18

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: checks mozconfigs against whitelist (see attached) as part of release_sanity

New approach - run this as part of release_sanity. Checks nightly/release mozconfigs and compares the diff to a whitelist per platform (see attached for example).

This can be tested locally using the following:

PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk -V 7.0b4 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py -c release-fennec-mozilla-beta.py --dryrun --products firefox,fennec localhost:9018 -b -m -w mozconfig_whitelist

Comment 19

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: sample mozconfig whitelist

Comment 20

[Chris AtLee [:catlee]]

Comment on attachment 557720 [details] [diff] [review]
checks mozconfigs against whitelist (see attached) as part of release_sanity

Review of attachment 557720 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildbot-helpers/release_sanity.py
@@ +79,5 @@
>          log.error("Repo does not exist with required revision. Check again, or use -b to bypass")
>          success = False
>      return success
>  
> +def verify_mozconfigs(branch, version, hghost, product, platforms, whitelist=None):

is None really a valid option? won't readConfig(None, ['whitelist']) fail?

@@ +97,5 @@
> +            mozconfigs.append(urllib2.urlopen(url).readlines())
> +        diffInstance = difflib.Differ()
> +        diffList = list(diffInstance.compare(mozconfigs[0],mozconfigs[1]))
> +        for line in diffList:
> +            if line[0] == '-' and len(line.strip()) > 1 and not '#' in line:

should we be checking new lines as well as deleted lines?

also, I think you need to be more careful about ignoring lines with comments. it's possible for a line to have a comment after real code.

@@ +101,5 @@
> +            if line[0] == '-' and len(line.strip()) > 1 and not '#' in line:
> +                # compare these to the whitelist
> +                line = line.replace('-','', 1).strip()
> +                if mozconfigWhitelist.has_key(platform) and line in mozconfigWhitelist[platform]:
> +                    pass

nit: use continue instead of pass here. then you can un-indent the block below

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: (v.2) checks mozconfigs against whitelist (see attached) as part of release_sanity

will attach an updated whitelist too.

Comment 22

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: (v.2) mozconfig_whitelist

Comment 23

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: (v.2) checks mozconfigs against whitelist (see attached) as part of release_sanity

added an else: mozconfigWhitelist = {} so that it doesn't fail out later on has_key()

Comment 24

[Chris AtLee [:catlee]]

Comment on attachment 557902 [details] [diff] [review]
(v.2) checks mozconfigs against whitelist (see attached) as part of release_sanity

Review of attachment 557902 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildbot-helpers/release_sanity.py
@@ +100,5 @@
> +            mozconfigs.append(urllib2.urlopen(url).readlines())
> +        diffInstance = difflib.Differ()
> +        diffList = list(diffInstance.compare(mozconfigs[0],mozconfigs[1]))
> +        for line in diffList:
> +            clean_line = re.sub(r'[-+]','',line,1).strip()

how about
clean_line = line[1:].strip()

what you have will strip out the first +/- from the line, regardless of the line starts with it or not.

@@ +111,5 @@
> +                    and line.replace('-','',1).strip() in mozconfigWhitelist['release'][platform]:
> +                        continue
> +                if line.startswith('+') and mozconfigWhitelist['nightly'].has_key(platform) \
> +                    and line.replace('+','',1).strip() in mozconfigWhitelist['nightly'][platform]:
> +                        continue

and then these can be replaced with, e.g.

if line[0] == '-' and mozconfigWhitelist.get('release', {}).has_key(platform) and clean_line in mozconfigWhitelist['release'][platform]:

what you've got will fail with KeyError in the case where no mozconfigWhitelist is passed in.

Comment 25

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: (v.3) checks mozconfigs against whitelist (see attached) as part of release_sanity

thanks for the tip - this one runs through with or without a whitelist file now.

Comment 26

[Chris AtLee [:catlee]]

Comment on attachment 558509 [details] [diff] [review]
(v.3) checks mozconfigs against whitelist (see attached) as part of release_sanity

Review of attachment 558509 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildbot-helpers/release_sanity.py
@@ +111,5 @@
> +                    and line[1:].strip() in mozconfigWhitelist['release'][platform]:
> +                        continue
> +                if line[0] == '+' and mozconfigWhitelist.get('release', {}).has_key(platform) \
> +                    and line[1:].strip() in mozconfigWhitelist['nightly'][platform]:
> +                        continue

I think you can use clean_line here and above instead of line[1:].strip()

also, the second if should use 'nightly' in both places instead of 'release', right?

@@ +273,5 @@
>              help="specify the release-config files (the first is primary)")
>      parser.add_option("-p", "--products", dest="products",
>              help="coma separated list of products")
> +    parser.add_option("-m", "--mozconfig", dest="mozconfig", action="store_true",
> +            help="just do a diff of mozconfigs")

looks like there's no way to disable this? it defaults to True above.

Comment 27

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: (v.4) checks mozconfigs against whitelist (see attached) as part of release_sanity -- cleaned up output & comments addressed

Output from running with/without a whitelist in testing:

Running with whitelist...
2011-09-08 18:01:44,396 : INFO : Comparing fennec nightly/release mozconfigs...
2011-09-08 18:01:44,830 : INFO : Comparing firefox nightly/release mozconfigs...
2011-09-08 18:01:45,318 : INFO : command: START
2011-09-08 18:01:45,319 : INFO : command: buildbot --version
2011-09-08 18:01:45,319 : INFO : command: cwd: /Users/lsb/mozilla/code/build-master/master
2011-09-08 18:01:45,320 : INFO : command: output:
Buildbot version: 0.8.1
Twisted version: 8.2.0
2011-09-08 18:01:45,526 : INFO : command: END (0.21s elapsed)
2011-09-08 18:01:45,527 : INFO : Tests Passed! Did not run reconfig/sendchange. Rerun without `-d`


Running withOUT whitelist...
2011-09-08 18:01:45,873 : INFO : Comparing fennec nightly/release mozconfigs...
2011-09-08 18:01:45,961 : ERROR : FAIL mozconfig diff: not in whitelist: linux-maemo5-gtk/release --mk_add_options MOZ_MAKE_FLAGS=-j4
2011-09-08 18:01:45,961 : ERROR : FAIL mozconfig diff: not in whitelist: linux-maemo5-gtk/nightly --ac_add_options --enable-official-branding
2011-09-08 18:01:46,214 : ERROR : FAIL mozconfig diff: not in whitelist: linux-android/release --mk_add_options MOZ_MAKE_FLAGS=-j4
2011-09-08 18:01:46,214 : ERROR : FAIL mozconfig diff: not in whitelist: linux-android/nightly --ac_add_options --enable-update-channel=beta
2011-09-08 18:01:46,214 : ERROR : FAIL mozconfig diff: not in whitelist: linux-android/release --ac_add_options --enable-update-channel=nightly
2011-09-08 18:01:46,215 : ERROR : FAIL mozconfig diff: not in whitelist: linux-android/nightly --ac_add_options --enable-official-branding
2011-09-08 18:01:46,296 : ERROR : FAIL mozconfig diff: not in whitelist: linux-mobile/nightly --CC=/tools/gcc-4.3.3/installed/bin/gcc
2011-09-08 18:01:46,296 : ERROR : FAIL mozconfig diff: not in whitelist: linux-mobile/release --CC=/tools/gcc-4.5/bin/gcc
2011-09-08 18:01:46,296 : ERROR : FAIL mozconfig diff: not in whitelist: linux-mobile/nightly --CXX=/tools/gcc-4.3.3/installed/bin/g++
2011-09-08 18:01:46,296 : ERROR : FAIL mozconfig diff: not in whitelist: linux-mobile/release --CXX=/tools/gcc-4.5/bin/g++
2011-09-08 18:01:46,297 : ERROR : FAIL mozconfig diff: not in whitelist: linux-mobile/release --ac_add_options --enable-stdcxx-compat
2011-09-08 18:01:46,434 : ERROR : Error verifying mozconfigs
2011-09-08 18:01:46,436 : INFO : Comparing firefox nightly/release mozconfigs...
2011-09-08 18:01:46,504 : ERROR : FAIL mozconfig diff: not in whitelist: linux/nightly --ac_add_options --enable-update-channel=beta
2011-09-08 18:01:46,505 : ERROR : FAIL mozconfig diff: not in whitelist: linux/release --ac_add_options --enable-update-channel=nightly
2011-09-08 18:01:46,505 : ERROR : FAIL mozconfig diff: not in whitelist: linux/nightly --ac_add_options --enable-official-branding
2011-09-08 18:01:46,505 : ERROR : FAIL mozconfig diff: not in whitelist: linux/release --ac_add_options --enable-codesighs
2011-09-08 18:01:46,505 : ERROR : FAIL mozconfig diff: not in whitelist: linux/release --export MOZILLA_OFFICIAL=1
2011-09-08 18:01:46,505 : ERROR : FAIL mozconfig diff: not in whitelist: linux/release --export MOZ_TELEMETRY_REPORTING=1
2011-09-08 18:01:46,505 : ERROR : FAIL mozconfig diff: not in whitelist: linux/nightly --export MOZILLA_OFFICIAL=1
2011-09-08 18:01:46,505 : ERROR : FAIL mozconfig diff: not in whitelist: linux/release --mk_add_options MOZ_MAKE_FLAGS="-j4"
2011-09-08 18:01:46,505 : ERROR : FAIL mozconfig diff: not in whitelist: linux/nightly --export MOZ_TELEMETRY_REPORTING=1
2011-09-08 18:01:46,505 : ERROR : FAIL mozconfig diff: not in whitelist: linux/release --ac_add_options --with-ccache=/usr/bin/ccache
2011-09-08 18:01:46,583 : ERROR : FAIL mozconfig diff: not in whitelist: linux64/nightly --ac_add_options --enable-update-channel=beta
2011-09-08 18:01:46,583 : ERROR : FAIL mozconfig diff: not in whitelist: linux64/release --ac_add_options --enable-update-channel=nightly
2011-09-08 18:01:46,583 : ERROR : FAIL mozconfig diff: not in whitelist: linux64/nightly --ac_add_options --enable-official-branding
2011-09-08 18:01:46,583 : ERROR : FAIL mozconfig diff: not in whitelist: linux64/release --ac_add_options --enable-codesighs
2011-09-08 18:01:46,583 : ERROR : FAIL mozconfig diff: not in whitelist: linux64/release --export MOZILLA_OFFICIAL=1
2011-09-08 18:01:46,584 : ERROR : FAIL mozconfig diff: not in whitelist: linux64/release --export MOZ_TELEMETRY_REPORTING=1
2011-09-08 18:01:46,584 : ERROR : FAIL mozconfig diff: not in whitelist: linux64/nightly --export MOZILLA_OFFICIAL=1
2011-09-08 18:01:46,584 : ERROR : FAIL mozconfig diff: not in whitelist: linux64/release --mk_add_options MOZ_MAKE_FLAGS="-j4"
2011-09-08 18:01:46,584 : ERROR : FAIL mozconfig diff: not in whitelist: linux64/nightly --export MOZ_TELEMETRY_REPORTING=1
2011-09-08 18:01:46,584 : ERROR : FAIL mozconfig diff: not in whitelist: linux64/release --ac_add_options --with-ccache=/usr/bin/ccache
2011-09-08 18:01:46,653 : ERROR : FAIL mozconfig diff: not in whitelist: win32/nightly --ac_add_options --enable-update-channel=beta
2011-09-08 18:01:46,653 : ERROR : FAIL mozconfig diff: not in whitelist: win32/release --ac_add_options --enable-update-channel=nightly
2011-09-08 18:01:46,653 : ERROR : FAIL mozconfig diff: not in whitelist: win32/nightly --ac_add_options --enable-official-branding
2011-09-08 18:01:46,653 : ERROR : FAIL mozconfig diff: not in whitelist: win32/release --. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags
2011-09-08 18:01:46,743 : ERROR : FAIL mozconfig diff: not in whitelist: macosx64/nightly --ac_add_options --enable-update-channel=beta
2011-09-08 18:01:46,743 : ERROR : FAIL mozconfig diff: not in whitelist: macosx64/release --ac_add_options --enable-update-channel=nightly
2011-09-08 18:01:46,744 : ERROR : FAIL mozconfig diff: not in whitelist: macosx64/nightly --ac_add_options --enable-official-branding
2011-09-08 18:01:46,744 : ERROR : FAIL mozconfig diff: not in whitelist: macosx64/release --ac_add_options --enable-codesighs
2011-09-08 18:01:46,744 : ERROR : FAIL mozconfig diff: not in whitelist: macosx64/release --ac_add_options --disable-install-strip
2011-09-08 18:01:46,744 : ERROR : FAIL mozconfig diff: not in whitelist: macosx64/release --mk_add_options MOZ_MAKE_FLAGS="-j4"
2011-09-08 18:01:46,744 : ERROR : Error verifying mozconfigs
2011-09-08 18:01:46,744 : INFO : command: START
2011-09-08 18:01:46,746 : INFO : command: buildbot --version
2011-09-08 18:01:46,746 : INFO : command: cwd: /Users/lsb/mozilla/code/build-master/master
2011-09-08 18:01:46,746 : INFO : command: output:
Buildbot version: 0.8.1
Twisted version: 8.2.0
2011-09-08 18:01:46,961 : INFO : command: END (0.21s elapsed)

2011-09-08 18:01:46,961 : CRITICAL : Tests Failed! Not running sendchange!

Comment 28

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: (v.5) checks mozconfigs against whitelist (included) as part of release_sanity

Updated to include the whitelist in tools/buildbot-helpers/mozconfig_whitelist so that this is all one patch.  I have tested this with the current production release configs for all three release branches with the following script:

echo "Running for a beta...\n++++++++++++++\n"
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk -V 8.0b3 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py -c release-fennec-mozilla-beta.py --dryrun --products firefox,fennec localhost:9018 -w ../tools/buildbot-helpers/mozconfig_whitelist
echo "Running for a release whitelist...\n++++++++++++++\n"
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk -V 7.0.1 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py -c release-fennec-mozilla-release.py --dryrun --products firefox,fennec localhost:9018  -w mozconfig_whitelist
echo "Running for a 3.6 release whitelist...\n++++++++++++++\n"
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk -V 3.6.23 --branch mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --dryrun --products firefox localhost:9018  -w ../tools/buildbot-helpers/mozconfig_whitelist

I also added in an error_tally set because the amount of info scrolling by is so much that it would be great to have a summary at the end of which test steps failed.

Comment 29

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 568143 [details] [diff] [review]
(v.5) checks mozconfigs against whitelist (included) as part of release_sanity

http://hg.mozilla.org/build/tools/rev/40a4916e512c

updated default for whitelist='mozconfig_whitelist'

Comment 30

[Rail Aliiev [:rail]]

release_sanity failed with

==================================
(build1)[cltbld@buildbot-master08 master]$ PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u rail -pfirefox,fennec -V 8.0b5 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py -c release-fennec-mozilla-beta.py --dryrun localhost:9001Traceback (most recent call last):
  File "../tools/buildbot-helpers/release_sanity.py", line 349, in <module>
    options.whitelist
  File "../tools/buildbot-helpers/release_sanity.py", line 93, in verify_mozconfigs
    mozconfigWhitelist = readConfig(whitelist, ['whitelist'])
  File "/builds/buildbot/build1/tools/lib/python/release/info.py", line 71, in readConfig
    execfile(configfile, c)
IOError: [Errno 2] No such file or directory: 'mozconfig_whitelist'

==================================

As a workaround I did this:

(build1)[cltbld@buildbot-master08 master]$ ln -s ../tools/buildbot-helpers/mozconfig_whitelist ./

Comment 31

[Lukas Blakk [:lsblakk] use ?needinfo]

Thanks for pointing that out, I have updated the default path (and tested to be sure it still worked). http://hg.mozilla.org/build/tools/rev/afc184e156cd