790637 - Updating a localized Nightly build results in an en-US build

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Maniac Vlad Florin (:vladmaniac)]

Update this build
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/09/2012-09-11-14-03-51-mozilla-central-l10n/firefox-18.0a1.fr.linux-x86_64.tar.bz2

It will give you an en-US build, not the right one. 
This triggered an error on our ci 
http://10.250.73.243:8080/job/mozilla-central_update/321/testReport/junit/testFallbackUpdate/test4/testFallbackUpdate_AppliedAndNoUpdatesFound/

Detailed log:
http://10.250.73.243:8080/job/mozilla-central_update/318/console

You probably won't this amount of data, its just necessary to install the build and update to reproduce.

Comment 1

[[e-mail] (:bhearsum)]

Looking into this.

Comment 2

[[e-mail] (:bhearsum)]

I just repro'ed. I shut off all Nightly and Aurora updates as a preacaution.

Comment 3

[Maniac Vlad Florin (:vladmaniac)]

(In reply to Ben Hearsum [:bhearsum] from comment #2)
> I just repro'ed. I shut off all Nightly and Aurora updates as a preacaution.

Nice to have this addressed so quickly. Thanks Ben

Comment 4

[[e-mail] (:bhearsum)]

The snippets look correct:
[ffxbld@dp-ausstage01 fr]$ pwd
/opt/aus2/incoming/2/Firefox/mozilla-central/Linux_x86_64-gcc3/20120911140351/fr
[ffxbld@dp-ausstage01 fr]$ cat complete.txt 
version=1
type=complete
url=http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/09/2012-09-12-03-06-13-mozilla-central-l10n/firefox-18.0a1.fr.linux-x86_64.complete.mar
hashFunction=sha512
hashValue=9a01068198b0e570c0a6f10385ea392fe8a78639db49a8ffd0ce78405b91c8b340d3c187688422868a652bb2cd5ddbbfa037f4dc8a3228a2744da050fac3d870
size=38123502
build=20120912030613
appv=18.0a1
extv=18.0a1
[ffxbld@dp-ausstage01 fr]$ cat partial.txt 
version=1
type=partial
url=http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/09/2012-09-12-03-06-13-mozilla-central-l10n/firefox-18.0a1.fr.linux-x86_64.partial.20120911140351-20120912030613.mar
hashFunction=sha512
hashValue=a18d1b3b4235c83f6b62d813557669314a0e19ff95d85bda8cc794b5cbe2c528c04552573214211531060a786a59c112ce753f368043c5a1036b240baf06e099
size=4088498
build=20120912030613
appv=18.0a1
extv=18.0a1


However, the build from comment #0 gets this AUS URL: https://aus3.mozilla.org/update/3/Firefox/18.0a1/20120911140351/Linux_x86_64-gcc3/en-US/nightly/Linux%203.2.0-24-generic%20(GTK%202.24.10)/default/default/update.xml?force=1, which is obviously wrong.

Not sure what's causing it yet, but it smells like client side updater or something else in-product. Could potentially be related to l10n build code, too.

Comment 5

[[e-mail] (:bhearsum)]

Did this test just start failing today? When was the last successful one? Bisecting to find the last working nightly would be really helpful here.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #2)
> I just repro'ed. I shut off all Nightly and Aurora updates as a preacaution.

Lukas & Alex - thought you'd want to be aware of this.

Comment 7

[[e-mail] (:bhearsum)]

Debugged this a bit further with Ehsan, Axel, and snorp on IRC. The update.locale file in omni.ja says "en-US", which seems to be why things are getting confused.

Comment 8

[(no longer active)]

So here's what I think is happening.  We create the update.locale file during an en-US build and put it in omni.ja.  Then, during the l10n repack, we repack en-US.jar and don't touch other things in omni.ja, which means that we would still use the file from an en-US build.

Comment 9

[[e-mail] (:bhearsum)]

OK, so Glandium confirmed that this was caused by https://bugzilla.mozilla.org/show_bug.cgi?id=787165#c2

And Ted even mentions there that it might break repacks =\. We're going to backout and respin nightlies. I turned Aurora updates back on, because it is clearly not affected.

Comment 10

[Lukas Blakk [:lsblakk] use ?needinfo]

Adjusting tracking accordingly then. Thanks for the speedy response to this.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As mentioned partly before we caught this failure with our Mozmill update testrun. The failure has been logged before as bug 790620. So marking it dependent on this fix.

I think if comment 0 would have given the content of the CI console data it would have saved you from debugging:

*** AUS:SVC getLocale - getting locale from file: resource://app/update.locale, locale: en-US

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Could this have also affected the default bookmarks for that build? We are also getting a failure here when checking for 'http://www.mozilla.com/fr/firefox/central/'. See bug 790579. At least we will see that when the new nightly builds have been made available.

Comment 13

[[e-mail] (:bhearsum)]

I retriggered nightlies after glandium backed out. It will take quite a few hours for everything to get rebuilt. Once we have new l10n builds on all platforms I'll turn Nightly updates back on.

Comment 14

[[e-mail] (:bhearsum)]

Catlee pointed out to me that users that got the broken builds won't be fixed by waiting for new ones, since they'll still report in as en-US and get updated to en-US. Unfortunately, there's nothing we can do for them =(. I turned Nightly updates back on.

Comment 15

[Axel Hecht [:Pike]]

Yeah. One thing you can do is retweet https://twitter.com/axelhecht/status/245951801491329025

Comment 16

[Alex Keybl [:akeybl]]

If this had happened on Aurora, it would have been more catastrophic. Let's figure out how to prevent this from happening again.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Alex Keybl [:akeybl] from comment #16)
> If this had happened on Aurora, it would have been more catastrophic. Let's
> figure out how to prevent this from happening again.

We have our Mozmill test but those are running for public Nightlies. If we could have a (nightly|aurora)-test channel so we could test before updates and builds put up on the FTP server we could avoid those issues. If our tests fail we stop everything. I think we filed a bug already but can't find right now.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Looks like this is bug 588398.

Comment 20

[Virgil Dicu [:virgil] [QA]]

I installed the available "fr" builds from 2012-11-29 for both Nightly and Aurora. I checked if they were in french and then updated them from Help > About. After restart, the new versions were, as expected, in french. Verified also for a "de" version.

Verified on Windows 7 x64, Mac OS X 10.8, Ubuntu 12.04.

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20121129 Firefox/20.0
Build ID: 20121129030820 >> 20121204030754

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20121129 Firefox/19.0
Build ID: 20121129042015 >> 20121204042015