1188493 - tracking bug for build and release of Firefox and Fennec 41.0

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Comment 1

[Jordan Lund (:jlund)]

tracking wiki: https://wiki.mozilla.org/Releases/Firefox_41.0b1/BuildNotes

Comment 2

[Jordan Lund (:jlund)]

release-runner failed on first attempt of beta1 build1 post ship-it:
2015-08-11 14:30:20,302 - INFO - command: output:
2015-08-11 14:30:20,303 - INFO - e928b851d8d9
2015-08-11 14:30:20,304 - INFO - mark as failed Fennec-41.0b1-build1
2015-08-11 14:30:20,304 - DEBUG - Request to https://ship-it.mozilla.org/releases/Fennec-41.0b1-build1
2015-08-11 14:30:20,305 - DEBUG - Data sent: {'ready': False, 'status': "Failed: KeyError('nextVersion',)", 'csrf_token': '20150811220015##a6621c41e288d1dfd091141212e7a62fb439a0e6'}
2015-08-11 14:30:20,305 - DEBUG - attempt 1/5
2015-08-11 14:30:20,306 - DEBUG - retry: Calling <function _req at 0x1205578> with args: (), kwargs: {}, attempt #1
2015-08-11 14:30:20,356 - DEBUG - "POST /releases/Fennec-41.0b1-build1 HTTP/1.1" 200 0
2015-08-11 14:30:20,357 - INFO - mark as failed Firefox-41.0b1-build1
2015-08-11 14:30:20,357 - DEBUG - Request to https://ship-it.mozilla.org/releases/Firefox-41.0b1-build1
2015-08-11 14:30:20,357 - DEBUG - Data sent: {'ready': False, 'status': "Failed: KeyError('nextVersion',)", 'csrf_token': '20150811220015##a6621c41e288d1dfd091141212e7a62fb439a0e6'}
2015-08-11 14:30:20,357 - DEBUG - attempt 1/5
2015-08-11 14:30:20,358 - DEBUG - retry: Calling <function _req at 0x138d488> with args: (), kwargs: {}, attempt #1
2015-08-11 14:30:20,402 - DEBUG - "POST /releases/Firefox-41.0b1-build1 HTTP/1.1" 200 0
2015-08-11 14:30:20,403 - DEBUG - Releasing lock: /builds/releaserunner/tools/buildfarm/release/.release-runner.lock
Traceback (most recent call last):
  File "release-runner.py", line 482, in <module>
    main(options)
  File "release-runner.py", line 404, in main
    ssh_username=hg_username, ssh_key=hg_ssh_key)
  File "/builds/releaserunner/tools/lib/python/util/hg.py", line 634, in apply_and_push
    changer(localrepo, 1)
  File "release-runner.py", line 385, in process_configs
    sendMailRD(smtp_server, notify_from, "%s/mozilla/%s" % (configs_workdir, cfgFile), release)
  File "release-runner.py", line 215, in sendMailRD
    release_config = readReleaseConfig(cfgFile)
  File "/builds/releaserunner/tools/lib/python/release/info.py", line 69, in readReleaseConfig
    return readConfig(configfile, keys=['releaseConfig'], required=required)
  File "/builds/releaserunner/tools/lib/python/release/info.py", line 87, in readConfig
    execfile(configfile, c)
  File "buildbot-configs/mozilla/release-fennec-mozilla-beta.py", line 46, in <module>
    'nextVersion': releaseConfig['nextVersion']
KeyError: 'nextVersion'

The full log is available on buildbot-master81 in /builds/releaserunner/release-runner.log
I'll sleep for 60 seconds before retry

nthomas is landing: https://bugzilla.mozilla.org/attachment.cgi?id=8646602

and we are going to reconfig and start again

Comment 3

[Jordan Lund (:jlund)]

Attachment: 150811_1188493_whitelist_beta_fix.patch

how do I test this?

Comment 4

[Jordan Lund (:jlund)]

Attachment: 150811_1188493_whitelist_beta_fix-2.patch

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Fix whitelist for 41.0b1

I tried the patch and it was still, which I think is due to the comparitor tracking which mozconfig each line is in. So here's a counter-offer which passes for me.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8646700 [details] [diff] [review]
[tools] Fix whitelist for 41.0b1

http://hg.mozilla.org/build/tools/rev/87d69be3cd53

Comment 7

[Jordan Lund (:jlund)]

bouncer submitter errors for fennec:
18:53:12     INFO - Checking if Fennec-41.0b1 already exists
18:53:12     INFO - retry: Calling _api_call with args: ('product_show?product=Fennec-41.0b1', None), kwargs: {}, attempt #1
18:53:12     INFO - Submitting to https://bounceradmin.mozilla.com/api/product_show?product=Fennec-41.0b1
18:53:13     INFO - Server response
18:53:13     INFO - <?xml version="1.0" encoding="utf-8"?><products/>
18:53:13     INFO - Products found: 0
18:53:13     INFO - Adding installer...
18:53:13     INFO - retry: Calling _api_call with args: ('product_add/', {'languages': ['ach', 'af', 'ak', 'ar', 'an', 'as', 'az', 'be', 'bn-IN', 'br', 'ca', 'cak', 'cs', 'cy', 'da', 'de', 'dsb', 'el', 'en-GB', 'en-ZA', 'eo', 'es-AR', 'es-ES', 'es-MX', 'et', 'eu', 'fa', 'ff', 'fi', 'fr', 'fy-NL', 'ga-IE', 'gd', 'gl', 'gu-IN', 'he', 'hi-IN', 'hr', 'hsb', 'hu', 'hy-AM', 'id', 'is', 'it', 'ja-JP-mac', 'ja', 'kk', 'km', 'kn', 'ko', 'lg', 'lij', 'lt', 'lv', 'mai', 'ml', 'mr', 'ms', 'my', 'nb-NO', 'nl', 'nn-NO', 'nso', 'or', 'pa-IN', 'pl', 'pt-BR', 'pt-PT', 'ro', 'ru', 'sk', 'sl', 'son', 'sq', 'sr', 'sv-SE', 'sw', 'ta', 'te', 'th', 'tr', 'tsz', 'uk', 'uz', 'vi', 'wo', 'xh', 'zh-CN', 'zh-TW', 'zu'], 'product': 'Fennec-41.0b1'}), kwargs: {}, attempt #1
18:53:13     INFO - POST data: languages=ach&languages=af&languages=ak&languages=ar&languages=an&languages=as&languages=az&languages=be&languages=bn-IN&languages=br&languages=ca&languages=cak&languages=cs&languages=cy&languages=da&languages=de&languages=dsb&languages=el&languages=en-GB&languages=en-ZA&languages=eo&languages=es-AR&languages=es-ES&languages=es-MX&languages=et&languages=eu&languages=fa&languages=ff&languages=fi&languages=fr&languages=fy-NL&languages=ga-IE&languages=gd&languages=gl&languages=gu-IN&languages=he&languages=hi-IN&languages=hr&languages=hsb&languages=hu&languages=hy-AM&languages=id&languages=is&languages=it&languages=ja-JP-mac&languages=ja&languages=kk&languages=km&languages=kn&languages=ko&languages=lg&languages=lij&languages=lt&languages=lv&languages=mai&languages=ml&languages=mr&languages=ms&languages=my&languages=nb-NO&languages=nl&languages=nn-NO&languages=nso&languages=or&languages=pa-IN&languages=pl&languages=pt-BR&languages=pt-PT&languages=ro&languages=ru&languages=sk&languages=sl&languages=son&languages=sq&languages=sr&languages=sv-SE&languages=sw&languages=ta&languages=te&languages=th&languages=tr&languages=tsz&languages=uk&languages=uz&languages=vi&languages=wo&languages=xh&languages=zh-CN&languages=zh-TW&languages=zu&product=Fennec-41.0b1
18:53:13     INFO - Submitting to https://bounceradmin.mozilla.com/api/product_add/
18:53:13  WARNING - Cannot access https://bounceradmin.mozilla.com/api/product_add/
18:53:13  WARNING - Returned page source:
18:53:13  WARNING - <?xml version="1.0" encoding="utf-8"?><error number="103">invalid language code(s)</error>

this seems to be because we have new locales that are not in bouncer:
https://hg.mozilla.org/releases/mozilla-beta/file/FENNEC_41_0b1_RELEASE/mobile/android/locales/all-locales

^ most recent additions tsz and cak:
https://hg.mozilla.org/releases/mozilla-beta/rev/432e6a592cf9

I've filed https://bugzil.la/1193613 to get these locales put into bouncer

Comment 8

[Jordan Lund (:jlund)]

looks like all desktop firefox repacks are failing

e.g. 
make[4]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n/config/external/sqlite'
make[4]: *** No rule to make target `../../../db/sqlite3/src/libdb_sqlite3_src.a', needed by `libmozsqlite3.so'.  Stop.
make[3]: *** [config/external/sqlite/target] Error 2
make[3]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n'
make[2]: *** [compile] Error 2
make[2]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n'
make[1]: *** [compile] Error 2
make[1]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n/config'
make: *** [default] Error 2
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['make']' returned non-zero exit status 2
command: END (1.23s elapsed)

Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/scripts/l10n/create-release-repacks.py", line 377, in <module>
    mozillaSrcDir=mozillaSrcDir
  File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/scripts/l10n/create-release-repacks.py", line 104, in createRepacks
    makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
  File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/lib/python/build/l10n.py", line 96, in l10nRepackPrep
    env=env)
  File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make']' returned non-zero exit status 2

investigating

Comment 9

[Jordan Lund (:jlund)]

also of note,

xulrunner win64 failed: http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/release-mozilla-beta-xulrunner_win64_build/builds/10

Comment 10

[Jordan Lund (:jlund)]

repack 'make config' error in comment 7 was patched via 1193620

force builds of the repacks yielded a new error:

macosx64 example fail:

+ rsync -a /builds/slave/rel-m-beta-m64_rpk_2-000000000/mozilla-beta/obj-l10n/dist/unpack.tmp/Nightly.app firefox
rsync: link_stat "/builds/slave/rel-m-beta-m64_rpk_2-000000000/mozilla-beta/obj-l10n/dist/unpack.tmp/Nightly.app" failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-42/rsync/main.c(992) [sender=2.6.9]
make: *** [/builds/slave/rel-m-beta-m64_rpk_2-000000000/mozilla-beta/obj-l10n/dist/l10n-stage/firefox/Nightly.app/Contents/Resources] Error 23
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['make', 'installers-bn-BD']' returned non-zero exit status 2
command: END (32.21s elapsed)

Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/scripts/l10n/create-release-repacks.py", line 131, in createRepacks
    mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir)
  File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/lib/python/build/l10n.py", line 191, in repackLocale
    run_cmd(make + ["installers-%s" % locale], cwd=localeSrcDir, env=env)
  File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['make', 'installers-bn-BD']' returned non-zero exit status 2





linux64 example fail:
command: START
command: perl ../../../tools/update-packaging/unwrap_full_update.pl /builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar
command: cwd: /builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/current
command: env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'AB_CD': 'bg', 'MOZ_SIGN_CMD': 'python /builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/release/signing/signtool.py --cachedir /builds/slave/rel-m-beta-l64_rpk_1-000000000/signing_cache -t /builds/slave/rel-m-beta-l64_rpk_1-000000000/token -n /builds/slave/rel-m-beta-l64_rpk_1-000000000/nonce -c /builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmg:mac-signing2.srv.releng.scl3.mozilla.com:9120 -H dmg:mac-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120', 'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_rsa', 'LOCALE_MERGEDIR': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/browser/locales/merged', 'COMM_REV': 'FIREFOX_41_0b1_RELEASE', 'MOZ_OBJDIR': 'obj-l10n', 'MAR': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/host/bin/mar', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib64', 'ZIP_IN': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/firefox.tar.bz2', 'UPLOAD_USER': 'ffxbld', 'MBSDIFF': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/host/bin/mbsdiff', 'UPLOAD_HOST': 'stage.mozilla.org', 'POST_UPLOAD_CMD': 'post_upload.py -p firefox -n 1 -v 41.0b1 --release-to-candidates-dir --signed', 'MOZILLA_REV': 'FIREFOX_41_0b1_RELEASE', 'MOZ_PKG_PRETTYNAMES': '1', 'MOZ_PKG_VERSION': '41.0b1'}
command: output:
ERROR: could not open file in mar_open()
/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar: No such file or directory
Couldn't run "/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/host/bin/mar" -t at ../../../tools/update-packaging/unwrap_full_update.pl line 51.
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['perl', '../../../tools/update-packaging/unwrap_full_update.pl', '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar']' returned non-zero exit status 255
command: END (0.01s elapsed)

Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 131, in createRepacks
    mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir)
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/build/l10n.py", line 197, in repackLocale
    cwd=path.join(nativeDistDir, 'current'), env=env)
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['perl', '../../../tools/update-packaging/unwrap_full_update.pl', '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar']' returned non-zero exit status 255
The following tracebacks were detected during repacks:
ach:
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 131, in createRepacks
    mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir)
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/build/l10n.py", line 197, in repackLocale
    cwd=path.join(nativeDistDir, 'current'), env=env)
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['perl', '../../../tools/update-packaging/unwrap_full_update.pl', '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/ach/firefox-41.0b1.complete.mar']' returned non-zero exit status 255


I am going to have to pick these up tomorrow.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Fix for l10n branding:
https://hg.mozilla.org/releases/mozilla-beta/rev/41eefdf2aed4   (relbranch)
https://hg.mozilla.org/releases/mozilla-beta/rev/aabfbc74aa01   (relbranch move tag)
https://hg.mozilla.org/releases/mozilla-beta/rev/3e5094441819   (default)

This gets us back to the state we had in 40.0 betas - no branding config in l10n-mozconfig, and the repack script adds --enable-official-branding at runtime (!!).

Comment 12

[Jordan Lund (:jlund)]

(In reply to Nick Thomas [:nthomas] from comment #11)
> Fix for l10n branding:
> https://hg.mozilla.org/releases/mozilla-beta/rev/41eefdf2aed4   (relbranch)
> https://hg.mozilla.org/releases/mozilla-beta/rev/aabfbc74aa01   (relbranch
> move tag)
> https://hg.mozilla.org/releases/mozilla-beta/rev/3e5094441819   (default)
> 
> This gets us back to the state we had in 40.0 betas - no branding config in
> l10n-mozconfig, and the repack script adds --enable-official-branding at
> runtime (!!).

all repacks completed successfully. thank you for your help here!

Comment 13

[Jordan Lund (:jlund)]

firefox desktop has been pushed to beta channel, signedoff the beta channel and post release steps complete.

however, xulrunner post release steps failed:
ERROR: push to mirrors directory, /pub/mozilla.org/xulrunner/releases/41.0b1/, does not exist on stage.mozilla.org
ERROR: Did you push to mirrors before running post release?
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/scripts/release/stage-tasks.py", line 390, in <module>
    target=ftpSymlinkName)
  File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/scripts/release/stage-tasks.py", line 192, in updateSymlink
    sshKey=stageSshKey)
  File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/lib/python/util/commands.py", line 75, in run_remote_cmd
    return run_cmd(cmd_prefix + cmd, **kwargs)
  File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)


this is likely due to https://bugzilla.mozilla.org/show_bug.cgi?id=1188493#c9

Comment 14

[Jordan Lund (:jlund)]

fennec mobile has also been pushed to the play store, waiting for QE signoff before running post release steps

Comment 15

[Jordan Lund (:jlund)]

still planning bouncer solution for single locale fennec fix. tracking that effort in: https://bugzilla.mozilla.org/show_bug.cgi?id=1193613

Comment 16

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #14)
> fennec mobile has also been pushed to the play store, waiting for QE signoff
> before running post release steps

ritu helped track this down. the sign off fell through the cracks due to other stuff going on.

I triggered the post-release steps for fennec and it failed on the last step because of https://bugzilla.mozilla.org/show_bug.cgi?id=1193613 and comment 5

Like the factory in comment 5, I am not going to fix it. the real fix is fixing 1193613 for future betas

Comment 17

[Jordan Lund (:jlund)]

that's a wrap for beta1

starting beta2 tracking: https://wiki.mozilla.org/Releases/Firefox_41.0b2/BuildNotes

ship-it for beta2 started and failed due to l10n changesets missing. A new one is about to start

Comment 18

[Jordan Lund (:jlund)]

[release] Firefox 41.0b2 build1: failed at repack_9/10 on win64

was a slow slave and timed out in configure step. nick re-ran this builder

Comment 19

[Jordan Lund (:jlund)]

Attachment: 150818_release_41_update_patcher_configs_beta2.patch

r+ by bhearsum over ff hello

Firefox 41.0b2 build1: failed at firefox_beta_updates

this may be something we need to do on every beta2 as patcher configs mangled beta and release status-updates. this verify step or at least this assert will probably go away (https://bugzil.la/1191074)

I am going to land, bump the two beta2 tags and rebuild this builder

Comment 20

[Jordan Lund (:jlund)]

Comment on attachment 8649409 [details] [diff] [review]
150818_release_41_update_patcher_configs_beta2.patch

landed and bumped release tags:
remote:   https://hg.mozilla.org/build/tools/rev/e0a6768fcf2b
remote:   https://hg.mozilla.org/build/tools/rev/c047c7f595be

Comment 21

[Jordan Lund (:jlund)]

beta2 complete.

beta3 submitted to release-kickoff(ship-it):

* win32 build failed because of slow win slave trying to remove old buildirs (retriggered)

* win{32,64}_xul failed because of https://bugzil.la/1170212 (win32_xul didn't even make it that far because it too timed out trying to remove old builddirs)

I am going to ignore the xulrunner failures and file a dep bug to fix for beta4

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Pre-emptive strike so that the updates builder doesn't fail for 41.0b3 build1 - 2nd hunk of attachment 8649409 [details] [diff] [review] again:
https://hg.mozilla.org/build/tools/rev/ab2eb5392456

Comment 23

[Jordan Lund (:jlund)]

[release] Firefox 41.0b3 build1: failed at update_verify_beta_4/6 on win32

^ hung for 8 hours. cancelled and re-triggered because it was due to windows getting in a hung state of a tree removeal. this blocked "[release] Firefox 41.0b3 build1: ready to ship to the beta channel" email.

also of note, dozens of the ui_update_verify jobs were hung for over 8 hours too. I've canceled them as they are not blocking.

Comment 24

[Jordan Lund (:jlund)]

Attachment: 150916_ff_41_disable_doPartnerRepacks-bbot-cfgs.patch

14:01 <•nthomas> coop: did you want to turn off auotmated partner repacks for 41.0 build2 ?
14:01 <•coop|mtg> nthomas: yes, i think we should
14:04 <jlund> do we want to turn this off for both Fennec and Firefox?
14:04 <•nthomas> we were talking about it a meeting, and c.oop wants to switch over to the github based system
14:04 <•nthomas> it's on for Fennec ?
14:04 <•nthomas> we'll generate 41.0 bulid2 partner repacks manually
14:14 <•nthomas> but anyway, we are just looking at firefox release right now
14:14 <jlund> okay. I'll revert my fennec parts in the patch

Comment 25

[Jordan Lund (:jlund)]

Comment on attachment 8662049 [details] [diff] [review]
150916_ff_41_disable_doPartnerRepacks-bbot-cfgs.patch

http://hg.mozilla.org/build/buildbot-configs/rev/7cbf830b20f1

and live in prod

Comment 26

[Jordan Lund (:jlund)]

started RC build3 (https://wiki.mozilla.org/index.php?title=Releases/Firefox_41.0)

currently failing: beta_final_verification failed for Firefox 41.0 build3

logs:

first fail:  http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-release-beta_final_verification/builds/1

reason:
   It is getting 404s from download links from obtained balrog xml. however, if I take one of those failed verify urls, e.g. the complete mar in here: https://aus4.mozilla.org/update/3/Firefox/30.0/20140529161749/WINNT_x86-msvc/sv-SE/beta-cdntest/default/default/default/update.xml?force=1

it downloads fine. Because of that, I re-triggered a second build..


second fail(after rebuild):  http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-release-beta_final_verification/builds/2

this failed much faster. the logs suggest missing xml but again I can download the xml:
log snippet:
Thu Sep 17 23:49:55 PDT 2015:  FAILURE 1: Update xml file not available
Thu Sep 17 23:49:55 PDT 2015:  
Thu Sep 17 23:49:55 PDT 2015:      Download url: https://aus4.mozilla.org/update/3/Firefox/39.0/20150609130336/Linux_x86-gcc3/en-ZA/beta-cdntest/default/default/default/update.xml?force=1
Thu Sep 17 23:49:55 PDT 2015:      Curl returned exit code: 52

ben: would you be able to have a peek at this when you come online or 302 it to someone?

Comment 27

[[e-mail] (:bhearsum)]

(In reply to Jordan Lund (:jlund) from comment #26)
> started RC build3
> (https://wiki.mozilla.org/index.php?title=Releases/Firefox_41.0)
> 
> currently failing: beta_final_verification failed for Firefox 41.0 build3
> 
> logs:
> 
> first fail: 
> http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-
> mozilla-release-beta_final_verification/builds/1
> 
> reason:
>    It is getting 404s from download links from obtained balrog xml. however,
> if I take one of those failed verify urls, e.g. the complete mar in here:
> https://aus4.mozilla.org/update/3/Firefox/30.0/20140529161749/WINNT_x86-msvc/
> sv-SE/beta-cdntest/default/default/default/update.xml?force=1
> 
> it downloads fine. Because of that, I re-triggered a second build..
> 
> 
> second fail(after rebuild): 
> http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-
> mozilla-release-beta_final_verification/builds/2
> 
> this failed much faster. the logs suggest missing xml but again I can
> download the xml:
> log snippet:
> Thu Sep 17 23:49:55 PDT 2015:  FAILURE 1: Update xml file not available
> Thu Sep 17 23:49:55 PDT 2015:  
> Thu Sep 17 23:49:55 PDT 2015:      Download url:
> https://aus4.mozilla.org/update/3/Firefox/39.0/20150609130336/Linux_x86-gcc3/
> en-ZA/beta-cdntest/default/default/default/update.xml?force=1
> Thu Sep 17 23:49:55 PDT 2015:      Curl returned exit code: 52
> 
> ben: would you be able to have a peek at this when you come online or 302 it
> to someone?

I think these were both intermittent issues. The first one might be related to the move to Cloudfront, as Nick suggested in e-mail, and the second one is probably related to https://bugzilla.mozilla.org/show_bug.cgi?id=1199721. A third run fully passed.

Comment 28

[Jordan Lund (:jlund)]

Attachment: 150922_41_release-disable_partner_repack_post_release_steps-bbotcfgs.patch

bc we disabled partner repacks for 41 but still tried to do post release partner steps:

cp: cannot stat `/pub/mozilla.org/firefox/candidates/41.0-candidates/build3/partner-repacks/bing/mac/en-US/Firefox 41.0.dmg': No such file or directory
command: ERROR