762714 - Android single locale repacks broken on aurora + central

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Aki Sasaki (not active)]

05:25:33     INFO -  /bin/sh: res/menu-v11/gecko_menu.xml: No such file or directory
05:25:33    ERROR -  make[2]: *** [res/menu-v11/gecko_menu.xml] Error 1
05:25:33     INFO -  make[2]: Leaving directory `/builds/slave/m-cen-andrd-l10n-2/build/mozilla-central/obj-l10n/mobile/android/base'
05:25:33    ERROR -  make[1]: *** [repackage-zip] Error 2
05:25:33     INFO -  make[1]: Leaving directory `/builds/slave/m-cen-andrd-l10n-2/build/mozilla-central/obj-l10n/mobile/android/locales'
05:25:33     INFO -  make: *** [repackage-zip-et] Error 2
05:25:33    ERROR - Return code: 2

This is affecting every locale.

Comment 1

[Aki Sasaki (not active)]

This appears to have started after May 25's nightlies on m-c.
Aurora started breaking after the merge.

On m-c, our 25-May-2012 04:25 repacks worked.
Our 25-May-2012 07:58 repacks didn't.
Getting a regression range.

Comment 2

[Aki Sasaki (not active)]

Good revision 1dd0c5c6d9fd
Bad revision 3871d6ca5fb2.

Comment 3

[Aki Sasaki (not active)]

I think this is the way to get this bug on the Aurora triage radar.

Comment 4

[Axel Hecht [:Pike]]

Just guessing.

hg diff -r 1dd0c5c6d9fd -r 3871d6ca5fb2 Makefile.in shows the removal of res/menu-v11 from RES_DIRS, http://hg.mozilla.org/mozilla-central/log/3871d6ca5fb2/mobile/android/base/Makefile.in shows sriram for the most part. Maybe bug 739355.

But that's really just guessing by the names.

Comment 5

[Sriram Ramasubramanian [:sriram]]

menu-v11 was once removed, as it was created with no value.
In a day or two, the folder was added back. As per current code, "menu-v11" is available.

Comment 6

[Aki Sasaki (not active)]

I suspected they needed a clobber, but these all clobber the tree before starting.
This is still broken as of this morning.

Comment 7

[Aki Sasaki (not active)]

Aurora is fixed as of last night:
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012-06-12-04-20-11-mozilla-aurora-android-l10n/

I'm not sure why that is.  Central is still busted.
I'm going to guess something landed in Aurora last night that fixed this.

Good revision: 776c453ae0ea
Monday's revision (bad): dd1621e5f855

Comment 8

[Aki Sasaki (not active)]

We backed out bug 748470 in Aurora (and Inbound) last night.

Comment 9

[Aki Sasaki (not active)]

Inbound merged to Central right after this morning's nightly (131961e5e0d1 was the nightly, 733994f12c53 was the merge). Tonight's m-c single locale repacks should work if that was the culprit.

Comment 10

[Aki Sasaki (not active)]

Now Aurora's fixed.
Central is dying with:

06:04:16    ERROR -  /builds/slave/m-cen-andrd-l10n-5/build/mozilla-central/js/src/configure: line 54: dn: command not found

Should be fixed by http://hg.mozilla.org/mozilla-central/rev/c7e76fae10e5 .

Comment 11

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #10)
> 06:04:16    ERROR - 
> /builds/slave/m-cen-andrd-l10n-5/build/mozilla-central/js/src/configure:
> line 54: dn: command not found
> 
> Should be fixed by http://hg.mozilla.org/mozilla-central/rev/c7e76fae10e5 .

This is still a problem.
I can:

a) file a bug on this
b) get the mozharness mobile l10n configure step to not error out on this.

Comment 12

[Aki Sasaki (not active)]

Attachment: fix mobile_l10n

(In reply to Aki Sasaki [:aki] from comment #11)
> b) get the mozharness mobile l10n configure step to not error out on this.

With this patch, we get past the error and get the following:

16:25:40    ERROR - fa failed in make installers-fa!
16:25:40    ERROR - gd failed in make installers-gd!
16:25:40    ERROR - he failed in make installers-he!
16:25:40    ERROR - Repacked 42 of 45 binaries successfully.
16:25:40     INFO - Uploaded 42 of 42 binaries successfully.
16:25:40    ERROR - Created 42 of 45 snippets successfully.
16:25:40     INFO - Setting buildbot property locales to {"el": "Success", "zh-TW": "Success", "ja-JP-mac": "Success", "vi": "Success", "ca": "Success", "it": "Success", "eu": "Success", "pa-IN": "Success", "gd": "Failed", "nb-NO": "Success", "cs": "Success", "et": "Success", "gl": "Success", "es-ES": "Success", "id": "Success", "ru": "Success", "sr": "Success", "nl": "Success", "tr": "Success", "lt": "Success", "th": "Success", "ro": "Success", "pl": "Success", "be": "Success", "fr": "Success", "sl": "Success", "de": "Success", "da": "Success", "fa": "Failed", "pt-BR": "Success", "fi": "Success", "hu": "Success", "ja": "Success", "he": "Failed", "pt-PT": "Success", "es-AR": "Success", "sq": "Success", "ko": "Success", "nn-NO": "Success", "ga-IE": "Success", "sk": "Success", "zh-CN": "Success", "ar": "Success", "uk": "Success", "fy-NL": "Success"}

We now have single locale repacks for Android on m-c, albeit from me running this script manually.  When we land this patch, it should fix the automation.

Comment 13

[Jordan Lund (:jlund)]

Comment on attachment 633727 [details] [diff] [review]
fix mobile_l10n

Look's good to me! Is this a temporary fix till we figure out what's going on? My only thought is if we leave this permanently, will it not increase unneeded build times for some machines that should have falled over sooner? You have probably thought about this and more but, since you asked me to review, I am trying to think about what this means.

Comment 14

[Aki Sasaki (not active)]

(In reply to Jordan Lund (:jlund) from comment #13)
> Comment on attachment 633727 [details] [diff] [review]
> fix mobile_l10n
> 
> Look's good to me! Is this a temporary fix till we figure out what's going
> on? My only thought is if we leave this permanently, will it not increase
> unneeded build times for some machines that should have falled over sooner?

This may be a permanent patch.
This patch changes the run_command call from exiting the script if it finds any ERROR lines to only exiting if the return status is non-zero; it's specifically to deal with scenarios like comment 10.

Comment 15

[Aki Sasaki (not active)]

Comment on attachment 633727 [details] [diff] [review]
fix mobile_l10n

http://hg.mozilla.org/build/mozharness/rev/f3cdddc46048

We now have android l10n repacks on m-c:
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-l10n/

Comment 16

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #11)
> (In reply to Aki Sasaki [:aki] from comment #10)
> > 06:04:16    ERROR - 
> > /builds/slave/m-cen-andrd-l10n-5/build/mozilla-central/js/src/configure:
> > line 54: dn: command not found
> > 
> > Should be fixed by http://hg.mozilla.org/mozilla-central/rev/c7e76fae10e5 .
> 
> This is still a problem.
> I can:
> 
> a) file a bug on this

Filed bug 765808 for js/src/configure.

Comment 17

[Jordan Lund (:jlund)]

(In reply to Aki Sasaki [:aki] from comment #14)
> (In reply to Jordan Lund (:jlund) from comment #13)
> > Comment on attachment 633727 [details] [diff] [review]
> > fix mobile_l10n
> > 
> > Look's good to me! Is this a temporary fix till we figure out what's going
> > on? My only thought is if we leave this permanently, will it not increase
> > unneeded build times for some machines that should have falled over sooner?
> 
> This may be a permanent patch.
> This patch changes the run_command call from exiting the script if it finds
> any ERROR lines to only exiting if the return status is non-zero; it's
> specifically to deal with scenarios like comment 10.

Right, this is what I thought. I was just saying letting the run_command run through entirely, check for its return status, and then fail if its anything other then 0, would it cause unneeded time versus stopping the command as soon as it finds an error. I am guessing what I am pointing out isn't an issue and stopping the command when an error is found is not an option since its causing comment 10.

Comment 18

[Aki Sasaki (not active)]

(In reply to Jordan Lund (:jlund) from comment #17)
> Right, this is what I thought. I was just saying letting the run_command run
> through entirely, check for its return status, and then fail if its anything
> other then 0, would it cause unneeded time versus stopping the command as
> soon as it finds an error. I am guessing what I am pointing out isn't an
> issue and stopping the command when an error is found is not an option since
> its causing comment 10.

Aha!

And it actually doesn't work that way with halt_on_failure; it runs all the way through, and then asks, "did I hit any errors while running?"

So no time savings/loss either way, just that halt_on_failure exits on an error we should ignore until bug 765808 is fixed, and checking return status doesn't.

Comment 19

[Jordan Lund (:jlund)]

ahh, I see. Thx, makes sense.

Comment 20

[Jordan Lund (:jlund)]

Woops, unintentionally reopened this twice. Please ignore