Created attachment 725432 [details] screenshot with error after update The Mozmill fallback update tests started failing on Nightly IT with today's build. It appears that the Italian locale is being offered the wrong fallback update: https://aus3.mozilla.org/update/3/Firefox/21.0a2/20130313042013/WINNT_x86-msvc/it/aurora/Windows_NT%206.1/default/default/update.xml?force=1 Mozmill report: http://mozmill-crowd.blargon7.com/#/update/report/2a6536e9db9f5f44ed48c58510b605d1 Jenkins link: http://mm-ci-master.qa.scl3.mozilla.com:8080/job/mozilla-central_update/1377/console The error can be reproduced locally by running the Mozmill tests and after update Firefox cannot be opened (I have attached the screenshot with the error received when trying to open Firefox after update).
Component: Application Update → Release Engineering: Releases
Product: Toolkit → mozilla.org
QA Contact: bhearsum
Version: 22 Branch → other
Confirmed manually, here is the update I'm offered on fallback (br instead of it): http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/03/2013-03-15-03-09-43-mozilla-central-l10n/firefox-22.0a1.br.win32.complete.mar
Looking at the update.xml snippet it seems like there's a typo: https://aus3.mozilla.org/update/3/Firefox/22.0a1/20130314030914/WINNT_x86-msvc/it/nightly/Windows_NT%220.127.116.11%20%28x64%29/default/default/update.xml URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/03/2013-03-15-03-09-43-mozilla-central-l10n/firefox-22.0a1.br.win32.complete.mar"
I'm disabling nightly updates while we investigate this.
Component: Release Engineering: Releases → Release Engineering: Automation (General)
QA Contact: bhearsum → catlee
So, looking at logs I see: ========= Started find filepath (results: 0, elapsed: 0 secs) (at 2013-03-15 08:24:59.857282) ========= 'bash' '-c' 'find build/mozilla-central/obj-firefox/dist/update -maxdepth 1 -type f -name *.complete.mar' in dir e:\builds\moz2_slave\m-cen-w32-l10n-ntly-0000000000\. (timeout 1200 secs) build/mozilla-central/obj-firefox/dist/update/firefox-22.0a1.br.win32.complete.mar build/mozilla-central/obj-firefox/dist/update/firefox-22.0a1.et.win32.complete.mar build/mozilla-central/obj-firefox/dist/update/firefox-22.0a1.he.win32.complete.mar build/mozilla-central/obj-firefox/dist/update/firefox-22.0a1.it.win32.complete.mar and we end up taking the first one as the complete MAR. So, either something isn't cleaning up like it did before, or we changed something in our buildbot code that's making us less choosey about the MAR.
Looks like this is related to bug 784848 13:14 < bhearsum> i see "'sh' '-c' 'if [ -d mozilla-central/dist/update ]; then rm -rf mozilla-central/dist/update; fi'" is yesterday's l10n nightly.... 13:14 < bhearsum> coop: ah, whic his why https://github.com/mozilla/build-buildbotcustom/commit/1115642ad5573710d40280c6c603d034af3e1a81#L1R2439 got added, i guess? 13:15 < bhearsum> i see this in today's nightly: "'sh' '-c' 'if [ -d obj-firefox/dist/update ]; then rm -rf obj-firefox/dist/update; fi'" 13:15 < bhearsum> which gets run in in dir e:\builds\moz2_slave\m-cen-w32-l10n-ntly-0000000000\build (timeout 1200 secs) 13:15 <@coop> yes 13:15 < bhearsum> so that dir probably doesn't exist...needs to be mozilla-central/obj-firefox 13:15 <@coop> that's not the correct dir, yeah 13:15 < bhearsum> boo
Assignee: nobody → coop
After this gets fixed and a new set of nightlies get run we need to chmod the snippet directory to enable nightlies again: chmod 775 /opt/aus2/incoming/2/Firefox/mozilla-central should do it.
Created attachment 725595 [details] [diff] [review] Specify the proper path to the dist dir for pre-build cleanup
Attachment #725595 - Flags: review?(bhearsum)
Comment on attachment 725595 [details] [diff] [review] Specify the proper path to the dist dir for pre-build cleanup https://hg.mozilla.org/build/buildbotcustom/rev/84f077434979
Attachment #725595 - Flags: checked-in+
In production now, although bug 851720 may prevent aurora updates from happening tonight.
Status: NEW → ASSIGNED
Priority: -- → P2
Updates for nightly still seem to be off. Can we turn them back on now?
(In reply to Reed Loden [:reed] from comment #10) > Updates for nightly still seem to be off. Can we turn them back on now? I just asked about this and was told that bug 851793 also need to be fixed first.
We'll track further issues via bug 852106.
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.