Closed Bug 1188493 Opened 4 years ago Closed 4 years ago

tracking bug for build and release of Firefox and Fennec 41.0

Categories

(Release Engineering :: Release Requests, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rail, Assigned: jlund)

References

Details

Attachments

(4 files, 2 obsolete files)

No description provided.
Assignee: nobody → jlund
release-runner failed on first attempt of beta1 build1 post ship-it:
2015-08-11 14:30:20,302 - INFO - command: output:
2015-08-11 14:30:20,303 - INFO - e928b851d8d9
2015-08-11 14:30:20,304 - INFO - mark as failed Fennec-41.0b1-build1
2015-08-11 14:30:20,304 - DEBUG - Request to https://ship-it.mozilla.org/releases/Fennec-41.0b1-build1
2015-08-11 14:30:20,305 - DEBUG - Data sent: {'ready': False, 'status': "Failed: KeyError('nextVersion',)", 'csrf_token': '20150811220015##a6621c41e288d1dfd091141212e7a62fb439a0e6'}
2015-08-11 14:30:20,305 - DEBUG - attempt 1/5
2015-08-11 14:30:20,306 - DEBUG - retry: Calling <function _req at 0x1205578> with args: (), kwargs: {}, attempt #1
2015-08-11 14:30:20,356 - DEBUG - "POST /releases/Fennec-41.0b1-build1 HTTP/1.1" 200 0
2015-08-11 14:30:20,357 - INFO - mark as failed Firefox-41.0b1-build1
2015-08-11 14:30:20,357 - DEBUG - Request to https://ship-it.mozilla.org/releases/Firefox-41.0b1-build1
2015-08-11 14:30:20,357 - DEBUG - Data sent: {'ready': False, 'status': "Failed: KeyError('nextVersion',)", 'csrf_token': '20150811220015##a6621c41e288d1dfd091141212e7a62fb439a0e6'}
2015-08-11 14:30:20,357 - DEBUG - attempt 1/5
2015-08-11 14:30:20,358 - DEBUG - retry: Calling <function _req at 0x138d488> with args: (), kwargs: {}, attempt #1
2015-08-11 14:30:20,402 - DEBUG - "POST /releases/Firefox-41.0b1-build1 HTTP/1.1" 200 0
2015-08-11 14:30:20,403 - DEBUG - Releasing lock: /builds/releaserunner/tools/buildfarm/release/.release-runner.lock
Traceback (most recent call last):
  File "release-runner.py", line 482, in <module>
    main(options)
  File "release-runner.py", line 404, in main
    ssh_username=hg_username, ssh_key=hg_ssh_key)
  File "/builds/releaserunner/tools/lib/python/util/hg.py", line 634, in apply_and_push
    changer(localrepo, 1)
  File "release-runner.py", line 385, in process_configs
    sendMailRD(smtp_server, notify_from, "%s/mozilla/%s" % (configs_workdir, cfgFile), release)
  File "release-runner.py", line 215, in sendMailRD
    release_config = readReleaseConfig(cfgFile)
  File "/builds/releaserunner/tools/lib/python/release/info.py", line 69, in readReleaseConfig
    return readConfig(configfile, keys=['releaseConfig'], required=required)
  File "/builds/releaserunner/tools/lib/python/release/info.py", line 87, in readConfig
    execfile(configfile, c)
  File "buildbot-configs/mozilla/release-fennec-mozilla-beta.py", line 46, in <module>
    'nextVersion': releaseConfig['nextVersion']
KeyError: 'nextVersion'

The full log is available on buildbot-master81 in /builds/releaserunner/release-runner.log
I'll sleep for 60 seconds before retry

nthomas is landing: https://bugzilla.mozilla.org/attachment.cgi?id=8646602

and we are going to reconfig and start again
Depends on: 1175331
how do I test this?
Attachment #8646674 - Flags: review?(nthomas)
Attachment #8646674 - Attachment is obsolete: true
Attachment #8646674 - Flags: review?(nthomas)
Attachment #8646676 - Flags: review?(nthomas)
I tried the patch and it was still, which I think is due to the comparitor tracking which mozconfig each line is in. So here's a counter-offer which passes for me.
Attachment #8646676 - Attachment is obsolete: true
Attachment #8646676 - Flags: review?(nthomas)
Attachment #8646700 - Flags: review?(jlund)
Attachment #8646700 - Flags: review?(jlund) → review+
bouncer submitter errors for fennec:
18:53:12     INFO - Checking if Fennec-41.0b1 already exists
18:53:12     INFO - retry: Calling _api_call with args: ('product_show?product=Fennec-41.0b1', None), kwargs: {}, attempt #1
18:53:12     INFO - Submitting to https://bounceradmin.mozilla.com/api/product_show?product=Fennec-41.0b1
18:53:13     INFO - Server response
18:53:13     INFO - <?xml version="1.0" encoding="utf-8"?><products/>
18:53:13     INFO - Products found: 0
18:53:13     INFO - Adding installer...
18:53:13     INFO - retry: Calling _api_call with args: ('product_add/', {'languages': ['ach', 'af', 'ak', 'ar', 'an', 'as', 'az', 'be', 'bn-IN', 'br', 'ca', 'cak', 'cs', 'cy', 'da', 'de', 'dsb', 'el', 'en-GB', 'en-ZA', 'eo', 'es-AR', 'es-ES', 'es-MX', 'et', 'eu', 'fa', 'ff', 'fi', 'fr', 'fy-NL', 'ga-IE', 'gd', 'gl', 'gu-IN', 'he', 'hi-IN', 'hr', 'hsb', 'hu', 'hy-AM', 'id', 'is', 'it', 'ja-JP-mac', 'ja', 'kk', 'km', 'kn', 'ko', 'lg', 'lij', 'lt', 'lv', 'mai', 'ml', 'mr', 'ms', 'my', 'nb-NO', 'nl', 'nn-NO', 'nso', 'or', 'pa-IN', 'pl', 'pt-BR', 'pt-PT', 'ro', 'ru', 'sk', 'sl', 'son', 'sq', 'sr', 'sv-SE', 'sw', 'ta', 'te', 'th', 'tr', 'tsz', 'uk', 'uz', 'vi', 'wo', 'xh', 'zh-CN', 'zh-TW', 'zu'], 'product': 'Fennec-41.0b1'}), kwargs: {}, attempt #1
18:53:13     INFO - POST data: languages=ach&languages=af&languages=ak&languages=ar&languages=an&languages=as&languages=az&languages=be&languages=bn-IN&languages=br&languages=ca&languages=cak&languages=cs&languages=cy&languages=da&languages=de&languages=dsb&languages=el&languages=en-GB&languages=en-ZA&languages=eo&languages=es-AR&languages=es-ES&languages=es-MX&languages=et&languages=eu&languages=fa&languages=ff&languages=fi&languages=fr&languages=fy-NL&languages=ga-IE&languages=gd&languages=gl&languages=gu-IN&languages=he&languages=hi-IN&languages=hr&languages=hsb&languages=hu&languages=hy-AM&languages=id&languages=is&languages=it&languages=ja-JP-mac&languages=ja&languages=kk&languages=km&languages=kn&languages=ko&languages=lg&languages=lij&languages=lt&languages=lv&languages=mai&languages=ml&languages=mr&languages=ms&languages=my&languages=nb-NO&languages=nl&languages=nn-NO&languages=nso&languages=or&languages=pa-IN&languages=pl&languages=pt-BR&languages=pt-PT&languages=ro&languages=ru&languages=sk&languages=sl&languages=son&languages=sq&languages=sr&languages=sv-SE&languages=sw&languages=ta&languages=te&languages=th&languages=tr&languages=tsz&languages=uk&languages=uz&languages=vi&languages=wo&languages=xh&languages=zh-CN&languages=zh-TW&languages=zu&product=Fennec-41.0b1
18:53:13     INFO - Submitting to https://bounceradmin.mozilla.com/api/product_add/
18:53:13  WARNING - Cannot access https://bounceradmin.mozilla.com/api/product_add/
18:53:13  WARNING - Returned page source:
18:53:13  WARNING - <?xml version="1.0" encoding="utf-8"?><error number="103">invalid language code(s)</error>

this seems to be because we have new locales that are not in bouncer:
https://hg.mozilla.org/releases/mozilla-beta/file/FENNEC_41_0b1_RELEASE/mobile/android/locales/all-locales

^ most recent additions tsz and cak:
https://hg.mozilla.org/releases/mozilla-beta/rev/432e6a592cf9

I've filed https://bugzil.la/1193613 to get these locales put into bouncer
Depends on: 1193613
looks like all desktop firefox repacks are failing

e.g. 
make[4]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n/config/external/sqlite'
make[4]: *** No rule to make target `../../../db/sqlite3/src/libdb_sqlite3_src.a', needed by `libmozsqlite3.so'.  Stop.
make[3]: *** [config/external/sqlite/target] Error 2
make[3]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n'
make[2]: *** [compile] Error 2
make[2]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n'
make[1]: *** [compile] Error 2
make[1]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n/config'
make: *** [default] Error 2
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['make']' returned non-zero exit status 2
command: END (1.23s elapsed)

Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/scripts/l10n/create-release-repacks.py", line 377, in <module>
    mozillaSrcDir=mozillaSrcDir
  File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/scripts/l10n/create-release-repacks.py", line 104, in createRepacks
    makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
  File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/lib/python/build/l10n.py", line 96, in l10nRepackPrep
    env=env)
  File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make']' returned non-zero exit status 2

investigating
Depends on: 1193620
repack 'make config' error in comment 7 was patched via 1193620

force builds of the repacks yielded a new error:

macosx64 example fail:

+ rsync -a /builds/slave/rel-m-beta-m64_rpk_2-000000000/mozilla-beta/obj-l10n/dist/unpack.tmp/Nightly.app firefox
rsync: link_stat "/builds/slave/rel-m-beta-m64_rpk_2-000000000/mozilla-beta/obj-l10n/dist/unpack.tmp/Nightly.app" failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-42/rsync/main.c(992) [sender=2.6.9]
make: *** [/builds/slave/rel-m-beta-m64_rpk_2-000000000/mozilla-beta/obj-l10n/dist/l10n-stage/firefox/Nightly.app/Contents/Resources] Error 23
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['make', 'installers-bn-BD']' returned non-zero exit status 2
command: END (32.21s elapsed)

Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/scripts/l10n/create-release-repacks.py", line 131, in createRepacks
    mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir)
  File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/lib/python/build/l10n.py", line 191, in repackLocale
    run_cmd(make + ["installers-%s" % locale], cwd=localeSrcDir, env=env)
  File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['make', 'installers-bn-BD']' returned non-zero exit status 2





linux64 example fail:
command: START
command: perl ../../../tools/update-packaging/unwrap_full_update.pl /builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar
command: cwd: /builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/current
command: env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'AB_CD': 'bg', 'MOZ_SIGN_CMD': 'python /builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/release/signing/signtool.py --cachedir /builds/slave/rel-m-beta-l64_rpk_1-000000000/signing_cache -t /builds/slave/rel-m-beta-l64_rpk_1-000000000/token -n /builds/slave/rel-m-beta-l64_rpk_1-000000000/nonce -c /builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmg:mac-signing2.srv.releng.scl3.mozilla.com:9120 -H dmg:mac-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120', 'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_rsa', 'LOCALE_MERGEDIR': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/browser/locales/merged', 'COMM_REV': 'FIREFOX_41_0b1_RELEASE', 'MOZ_OBJDIR': 'obj-l10n', 'MAR': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/host/bin/mar', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib64', 'ZIP_IN': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/firefox.tar.bz2', 'UPLOAD_USER': 'ffxbld', 'MBSDIFF': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/host/bin/mbsdiff', 'UPLOAD_HOST': 'stage.mozilla.org', 'POST_UPLOAD_CMD': 'post_upload.py -p firefox -n 1 -v 41.0b1 --release-to-candidates-dir --signed', 'MOZILLA_REV': 'FIREFOX_41_0b1_RELEASE', 'MOZ_PKG_PRETTYNAMES': '1', 'MOZ_PKG_VERSION': '41.0b1'}
command: output:
ERROR: could not open file in mar_open()
/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar: No such file or directory
Couldn't run "/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/host/bin/mar" -t at ../../../tools/update-packaging/unwrap_full_update.pl line 51.
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['perl', '../../../tools/update-packaging/unwrap_full_update.pl', '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar']' returned non-zero exit status 255
command: END (0.01s elapsed)

Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 131, in createRepacks
    mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir)
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/build/l10n.py", line 197, in repackLocale
    cwd=path.join(nativeDistDir, 'current'), env=env)
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['perl', '../../../tools/update-packaging/unwrap_full_update.pl', '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar']' returned non-zero exit status 255
The following tracebacks were detected during repacks:
ach:
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 131, in createRepacks
    mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir)
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/build/l10n.py", line 197, in repackLocale
    cwd=path.join(nativeDistDir, 'current'), env=env)
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['perl', '../../../tools/update-packaging/unwrap_full_update.pl', '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/ach/firefox-41.0b1.complete.mar']' returned non-zero exit status 255


I am going to have to pick these up tomorrow.
Fix for l10n branding:
https://hg.mozilla.org/releases/mozilla-beta/rev/41eefdf2aed4   (relbranch)
https://hg.mozilla.org/releases/mozilla-beta/rev/aabfbc74aa01   (relbranch move tag)
https://hg.mozilla.org/releases/mozilla-beta/rev/3e5094441819   (default)

This gets us back to the state we had in 40.0 betas - no branding config in l10n-mozconfig, and the repack script adds --enable-official-branding at runtime (!!).
(In reply to Nick Thomas [:nthomas] from comment #11)
> Fix for l10n branding:
> https://hg.mozilla.org/releases/mozilla-beta/rev/41eefdf2aed4   (relbranch)
> https://hg.mozilla.org/releases/mozilla-beta/rev/aabfbc74aa01   (relbranch
> move tag)
> https://hg.mozilla.org/releases/mozilla-beta/rev/3e5094441819   (default)
> 
> This gets us back to the state we had in 40.0 betas - no branding config in
> l10n-mozconfig, and the repack script adds --enable-official-branding at
> runtime (!!).

all repacks completed successfully. thank you for your help here!
firefox desktop has been pushed to beta channel, signedoff the beta channel and post release steps complete.

however, xulrunner post release steps failed:
ERROR: push to mirrors directory, /pub/mozilla.org/xulrunner/releases/41.0b1/, does not exist on stage.mozilla.org
ERROR: Did you push to mirrors before running post release?
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/scripts/release/stage-tasks.py", line 390, in <module>
    target=ftpSymlinkName)
  File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/scripts/release/stage-tasks.py", line 192, in updateSymlink
    sshKey=stageSshKey)
  File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/lib/python/util/commands.py", line 75, in run_remote_cmd
    return run_cmd(cmd_prefix + cmd, **kwargs)
  File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)


this is likely due to https://bugzilla.mozilla.org/show_bug.cgi?id=1188493#c9
fennec mobile has also been pushed to the play store, waiting for QE signoff before running post release steps
still planning bouncer solution for single locale fennec fix. tracking that effort in: https://bugzilla.mozilla.org/show_bug.cgi?id=1193613
(In reply to Jordan Lund (:jlund) from comment #14)
> fennec mobile has also been pushed to the play store, waiting for QE signoff
> before running post release steps

ritu helped track this down. the sign off fell through the cracks due to other stuff going on.

I triggered the post-release steps for fennec and it failed on the last step because of https://bugzilla.mozilla.org/show_bug.cgi?id=1193613 and comment 5

Like the factory in comment 5, I am not going to fix it. the real fix is fixing 1193613 for future betas
that's a wrap for beta1

starting beta2 tracking: https://wiki.mozilla.org/Releases/Firefox_41.0b2/BuildNotes

ship-it for beta2 started and failed due to l10n changesets missing. A new one is about to start
[release] Firefox 41.0b2 build1: failed at repack_9/10 on win64

was a slow slave and timed out in configure step. nick re-ran this builder
r+ by bhearsum over ff hello

Firefox 41.0b2 build1: failed at firefox_beta_updates

this may be something we need to do on every beta2 as patcher configs mangled beta and release status-updates. this verify step or at least this assert will probably go away (https://bugzil.la/1191074)

I am going to land, bump the two beta2 tags and rebuild this builder
Attachment #8649409 - Flags: review+
Comment on attachment 8649409 [details] [diff] [review]
150818_release_41_update_patcher_configs_beta2.patch

landed and bumped release tags:
remote:   https://hg.mozilla.org/build/tools/rev/e0a6768fcf2b
remote:   https://hg.mozilla.org/build/tools/rev/c047c7f595be
Attachment #8649409 - Flags: checked-in+
beta2 complete.

beta3 submitted to release-kickoff(ship-it):

* win32 build failed because of slow win slave trying to remove old buildirs (retriggered)

* win{32,64}_xul failed because of https://bugzil.la/1170212 (win32_xul didn't even make it that far because it too timed out trying to remove old builddirs)

I am going to ignore the xulrunner failures and file a dep bug to fix for beta4
Depends on: 1197011
Pre-emptive strike so that the updates builder doesn't fail for 41.0b3 build1 - 2nd hunk of attachment 8649409 [details] [diff] [review] again:
https://hg.mozilla.org/build/tools/rev/ab2eb5392456
[release] Firefox 41.0b3 build1: failed at update_verify_beta_4/6 on win32

^ hung for 8 hours. cancelled and re-triggered because it was due to windows getting in a hung state of a tree removeal. this blocked "[release] Firefox 41.0b3 build1: ready to ship to the beta channel" email.

also of note, dozens of the ui_update_verify jobs were hung for over 8 hours too. I've canceled them as they are not blocking.
No longer depends on: 1180792
14:01 <•nthomas> coop: did you want to turn off auotmated partner repacks for 41.0 build2 ?
14:01 <•coop|mtg> nthomas: yes, i think we should
14:04 <jlund> do we want to turn this off for both Fennec and Firefox?
14:04 <•nthomas> we were talking about it a meeting, and c.oop wants to switch over to the github based system
14:04 <•nthomas> it's on for Fennec ?
14:04 <•nthomas> we'll generate 41.0 bulid2 partner repacks manually
14:14 <•nthomas> but anyway, we are just looking at firefox release right now
14:14 <jlund> okay. I'll revert my fennec parts in the patch
Attachment #8662049 - Flags: review?(nthomas)
Attachment #8662049 - Flags: review?(nthomas) → review+
Depends on: 1191633
Comment on attachment 8662049 [details] [diff] [review]
150916_ff_41_disable_doPartnerRepacks-bbot-cfgs.patch

http://hg.mozilla.org/build/buildbot-configs/rev/7cbf830b20f1

and live in prod
Attachment #8662049 - Flags: checked-in+
started RC build3 (https://wiki.mozilla.org/index.php?title=Releases/Firefox_41.0)

currently failing: beta_final_verification failed for Firefox 41.0 build3

logs:

first fail:  http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-release-beta_final_verification/builds/1

reason:
   It is getting 404s from download links from obtained balrog xml. however, if I take one of those failed verify urls, e.g. the complete mar in here: https://aus4.mozilla.org/update/3/Firefox/30.0/20140529161749/WINNT_x86-msvc/sv-SE/beta-cdntest/default/default/default/update.xml?force=1

it downloads fine. Because of that, I re-triggered a second build..


second fail(after rebuild):  http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-release-beta_final_verification/builds/2

this failed much faster. the logs suggest missing xml but again I can download the xml:
log snippet:
Thu Sep 17 23:49:55 PDT 2015:  FAILURE 1: Update xml file not available
Thu Sep 17 23:49:55 PDT 2015:  
Thu Sep 17 23:49:55 PDT 2015:      Download url: https://aus4.mozilla.org/update/3/Firefox/39.0/20150609130336/Linux_x86-gcc3/en-ZA/beta-cdntest/default/default/default/update.xml?force=1
Thu Sep 17 23:49:55 PDT 2015:      Curl returned exit code: 52

ben: would you be able to have a peek at this when you come online or 302 it to someone?
Flags: needinfo?(bhearsum)
(In reply to Jordan Lund (:jlund) from comment #26)
> started RC build3
> (https://wiki.mozilla.org/index.php?title=Releases/Firefox_41.0)
> 
> currently failing: beta_final_verification failed for Firefox 41.0 build3
> 
> logs:
> 
> first fail: 
> http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-
> mozilla-release-beta_final_verification/builds/1
> 
> reason:
>    It is getting 404s from download links from obtained balrog xml. however,
> if I take one of those failed verify urls, e.g. the complete mar in here:
> https://aus4.mozilla.org/update/3/Firefox/30.0/20140529161749/WINNT_x86-msvc/
> sv-SE/beta-cdntest/default/default/default/update.xml?force=1
> 
> it downloads fine. Because of that, I re-triggered a second build..
> 
> 
> second fail(after rebuild): 
> http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-
> mozilla-release-beta_final_verification/builds/2
> 
> this failed much faster. the logs suggest missing xml but again I can
> download the xml:
> log snippet:
> Thu Sep 17 23:49:55 PDT 2015:  FAILURE 1: Update xml file not available
> Thu Sep 17 23:49:55 PDT 2015:  
> Thu Sep 17 23:49:55 PDT 2015:      Download url:
> https://aus4.mozilla.org/update/3/Firefox/39.0/20150609130336/Linux_x86-gcc3/
> en-ZA/beta-cdntest/default/default/default/update.xml?force=1
> Thu Sep 17 23:49:55 PDT 2015:      Curl returned exit code: 52
> 
> ben: would you be able to have a peek at this when you come online or 302 it
> to someone?

I think these were both intermittent issues. The first one might be related to the move to Cloudfront, as Nick suggested in e-mail, and the second one is probably related to https://bugzilla.mozilla.org/show_bug.cgi?id=1199721. A third run fully passed.
Flags: needinfo?(bhearsum)
No longer depends on: 1175331
bc we disabled partner repacks for 41 but still tried to do post release partner steps:

cp: cannot stat `/pub/mozilla.org/firefox/candidates/41.0-candidates/build3/partner-repacks/bing/mac/en-US/Firefox 41.0.dmg': No such file or directory
command: ERROR
Attachment #8664352 - Flags: review?(bhearsum)
Attachment #8664352 - Flags: review?(bhearsum) → review+
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.