Closed Bug 466807 Opened 16 years ago Closed 16 years ago

L10n builds are being incorrectly/mis-generated

Categories

(Mozilla Messaging Graveyard :: Release Engineering, defect)

Product:
Mozilla Messaging Graveyard
Component:
Release Engineering
Type:
defect
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: standard8, Assigned: gozer)

References

Details

We've been seeing various reports of problems with l10n nightly builds on the newsgroups. I've verified the de ones are definitely an issue. I'm just kicking off a local build to confirm, but compare-locales is saying green, and I'm inclined to trust that.

So I think something is going wrong build side here and pulling the wrong source code or the wrong l10n version.

Email that was sent by simon:

here are some excerpts from the recent TB3 beta1 l10n opt-in thread:

| From: tim_maks@planet.nl (Tim Maks van den Broek)
| Subject: Re: [ANNOUNCE] Thunderbird 3 Beta 1 string frozen, OPT IN HERE
| | I would really like to opt in the nl version with the revision you | used as a example but when I download the latest build from
| ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-central-l10n
| I don't get a folder view, start page and advanced preferences. I | have no clue what the problem is, the buildbot is green but the | tinderboxes are burning.
| | I used the linuxbuild for testing

| From: tim_maks@planet.nl (Tim Maks van den Broek)
| Subject: Re: [ANNOUNCE] Thunderbird 3 Beta 1 string frozen, OPT IN HERE
| | I see the same in the de linuxbuild , is there something wrong with | the l10n building or do i do something wrong.....

| From: tomerc@gmail.com (Tomer Cohen)
| Subject: Re: Thunderbird 3 Beta 1 string frozen, OPT IN HERE
| | Hebrew (he) is ready.
| http://hg.mozilla.org/l10n-central/he/file/beab2fe87a0e
| | As other people said in this thread, the recent builds are not very
| stable...

I just tested this myself in a German Windows ZIP-build and can
see the same things that Tim mentioned.

Here is the output from the error console:

| Error: uncaught exception: [Exception... "Component returned failure | code: 0x80004005 (NS_ERROR_FAILURE) | [nsIStringBundle.GetStringFromName]"  nsresult: "0x80004005 | (NS_ERROR_FAILURE)"  location: "JS frame :: XStringBundle :: getString | :: line 17"  data: no]

| Error: Undefined entity
| Source file: chrome://messenger/content/preferences/advanced.xul
| Line: 111, Column: 17
| Source: <checkbox id="enableGloda"

| Error: this._tree is null
| Source file: chrome://messenger/content/folderPane.js
| Line: 649
I don't know if this is a helpful information, but I doesn't have this problem with my own (selfmade) german Thunderbird build.

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; de; rv:1.9.1b3pre) Gecko/20081126 Thunderbird/3.0b1pre (de)
As mentioned in .l10n, compare-locales is only as good as your actual source is. If the source doesn't match the binary you're repackaging, doom is inevitable.

Depending on when you build, different string changes might have happened between the en-US build of the nightly and the current tip in comm-central and friends.

In Firefox, we added the mozilla-central source stamp to application.ini to work around that issue. See bug 452426. This is trickier here as there's more than one repo involved, of course.
(In reply to comment #2)
> Depending on when you build, different string changes might have happened
> between the en-US build of the nightly and the current tip in comm-central and
> friends.

I believe that is not the issue here for this problem at least:

The final string changes to en-US went into comm-central at Mon Nov 24 12:40:33 2008 -0800
The de changes to match that went into the l10n-central/de at Mon Nov 24 14:34:14 2008 -0800

The en-US nightly builds are generated at approx 03:00 with the l10n builds straight after that.

Therefore we've now had two sets of nightly builds that should have included both changes assuming we're pulling the latest code in both cases. I'm thinking that something may have gone wrong in the pulling/update process, though I couldn't see anything in the few logs I had a look at. Hopefull gozer can see something more.
I still see the problem in the nightly build of 26-11. Also the last 2 string changes made a few days ago are not present in the nl.jar
Ok, I had a look at the recent de l10n changes and from what I could see every l10n change that affects the German TB localization after http://hg.mozilla.org/l10n-central/de/rev/738b7fc3f508 (18 days ago - Sun Nov 09 01:17:07 2008 +0100) is not in de.jar of a current TB nightly downloaded today.

Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1b3pre) Gecko/20081126 Shredder/3.0b1pre

So either that change is the culprit, although Pike and KaiRo told me in #l10n that this would unlikely or something else broke around that time. The latter seems to be more likely IMO, since that push would not explain the problems that the Dutch l10n team and other teams are seeing in their localizations.
Looking at the comm-central changes around the time of the push mentioned in comment 6, only this push http://hg.mozilla.org/comm-central/rev/cc3ffc99b4bf for bug 463385 looks like a possible candidate for the misgeneration.

Is that possible or am I totally on the wrong track here?
I have a theory:

Looking at the logs, the pulls go to -r tip, and not default. So if while it's pulling the tip happens to be a relbranch, the local clone switches over to the relbranch. I think.

So it might be that some of the local clones on the build slaves are stuck being on a relbranch, and need to be freed.
Did we have a branch on Nov 9 as well, so we might have been stuck since then?
(In reply to comment #8)
> I have a theory:
> 
> Looking at the logs, the pulls go to -r tip, and not default. So if while it's
> pulling the tip happens to be a relbranch, the local clone switches over to the
> relbranch. I think.
> 
> So it might be that some of the local clones on the build slaves are stuck
> being on a relbranch, and need to be freed.

This could be true, polish nightly builds don't contain this change: http://hg.mozilla.org/l10n-central/pl/rev/ef217beddcf6 - the first after 191b2 l10n branching. (xpi seems to be ok)
(In reply to comment #10)
> This could be true, polish nightly builds don't contain this change:
> http://hg.mozilla.org/l10n-central/pl/rev/ef217beddcf6 - the first after 191b2
> l10n branching. (xpi seems to be ok)

Ah, sorry, please ignore my comment above - my scripts downloads only successful builds.

Orange (after push) and red (scheduled) tinderbox builds contain all up to the default tip (after l10n branching) changes (at least for pl).
(In reply to comment #8)
> I have a theory:
> 
> Looking at the logs, the pulls go to -r tip, and not default. So if while it's
> pulling the tip happens to be a relbranch, the local clone switches over to the
> relbranch. I think.

And that is exactly what happened. de is stuck on the GECKO191b2_20081125_RELBRANCH. for instance

> So it might be that some of the local clones on the build slaves are stuck
> being on a relbranch, and need to be freed.

would using -r default instead of -r tip fix the problem ?
The beta 2 relbranch is actually pretty recent, see http://hg.mozilla.org/l10n-central/de/pushloghtml.

But yes, -r default is way more resistant against this than -r tip is.
Not sure how best teach buildbot how to do that yet.

In the meantime, here is what it looks like on the linux builder:

[cltbld@tb-linux-tbox l10n]$ for d in *; do echo -n "$d: " ; hg identify -R $d; done
af: 65813b9c3941 (GECKO191b2_20081125_RELBRANCH) tip
ar: c2e7dfb9120f tip
be: 1024dffd45da tip
bg: a1d2df4e601b (GECKO191b2_20081125_RELBRANCH) tip
ca: 13f463e265ae (GECKO191b2_20081125_RELBRANCH) tip
cs: d834a4624bee tip
da: 7843384d63a6 tip
de: fc0d4a96634e (GECKO191b2_20081125_RELBRANCH) tip
el: 9ea6d8ef7483 (GECKO191b2_20081125_RELBRANCH) tip
en-GB: 80bb81c3276e (GECKO191b2_20081125_RELBRANCH) tip
es-AR: e9506d69bc18 (GECKO191b2_20081125_RELBRANCH) tip
es-ES: 9da5fdfbc0a0 (GECKO191b2_20081125_RELBRANCH) tip
et: 6326ce7c229f tip
eu: 315f4208b728 tip
fi: a534bff696d1 (GECKO191b2_20081125_RELBRANCH) tip
fr: dc07428312a5 tip
fy-NL: b0ab3c707ac2 (GECKO191b2_20081125_RELBRANCH) tip
ga-IE: e3a312ddc590 (GECKO191b2_20081125_RELBRANCH) tip
gu-IN: 16f7a85d2e58 (GECKO191b2_20081125_RELBRANCH) tip
he: beab2fe87a0e tip
hu: 5a22808c7ce7 tip
id: 2acfa6404378 (GECKO191b2_20081125_RELBRANCH) tip
it: 32935707080b tip
ja: e370c19bf8c6 (GECKO191b2_20081125_RELBRANCH) tip
ja-JP-mac: 59dd0f5cb354 (GECKO191b2_20081125_RELBRANCH) tip
ka: 4d5e15d0ee92 (GECKO191b2_20081125_RELBRANCH) tip
ko: 5c9e6f9c2348 tip
lt: e621b85d388c tip
mk: 9c6b2085c598 tip
mn: 233ee371e4e6 tip
nb-NO: bb6c8e2cd8d6 tip
nl: c332fb500ac8 (GECKO191b2_20081125_RELBRANCH) tip
nn-NO: dc1345e72f40 tip
pa-IN: f0f5d9761b77 (GECKO191b2_20081125_RELBRANCH) tip
pl: 18ea38d5bcee tip
pt-BR: d90114ae4b9d (GECKO191b2_20081125_RELBRANCH) tip
pt-PT: cabe4afa285a (GECKO191b2_20081125_RELBRANCH) tip
ro: 9c4f1c33796d (GECKO191b2_20081125_RELBRANCH) tip
ru: eba545b85bb4 tip
si: e348a155439e tip
sk: e76e0a2d155d tip
sl: b1e83abcbad5 (GECKO191b2_20081125_RELBRANCH) tip
sq: a962b4472294 tip
sr: 7a7b97dd35f5 tip
sv-SE: ff7390c6a162 tip
tr: 3e09b32742c3 (GECKO191b2_20081125_RELBRANCH) tip
uk: 0bf107b54279 (GECKO191b2_20081125_RELBRANCH) tip
zh-CN: 0c4daed57140 (GECKO191b2_20081125_RELBRANCH) tip
zh-TW: 88091baf80af tip

So, a whole bunch of locales stuck in the past
I've manually updated all the locales to the tip of the default branch on all 3 builders, for the time being.
Now, as for what can be done so it doesn't happen in the future, I am not certain. the L10n factory uses pretty much a good old plain Mercurial() checkout from buildbot, and can't find an easy way to pass -r default down there.
(In reply to comment #12)

> And that is exactly what happened. de is stuck on the
> GECKO191b2_20081125_RELBRANCH. for instance

I remember I've had a similar problem after GECKO191b1_20081007_RELBRANCH. But this disappears by itself.
http://forums.mozillazine.org/viewtopic.php?f=42&t=894215
Gozer, are you sure this is/was the cause at all? e.g for de, GECKO191b2_20081125_RELBRANCH is the current tip and is L10n-code-wise identical to the default tip, only the tags are different - so builds should be the same.
Well whatever has happened, the latest de build on mac seems to be working fine on start up at least.
i can conform that it is fixed for the nl locale (linux), thanks
Works for de as well now. Tested with Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1b3pre) Gecko/20081127 Shredder/3.0b1pre from yesterday 11PM.

Shall we mark this FIXED/WORKSFORME now?
Severity: critical → blocker
I'd rather wait a little, as I am still investigating into how we can prevent this from hapenning again.
BuildID=20081201030423
German nightly build has empty pane. The same error is back again now?

Fehler: Nicht definierte Entität
Quelldatei: jar:file:///usr/local/thunderbird-20081201de/chrome/messenger.jar!/content/messenger/mailWidgets.xml
Zeile: 241, Spalte: 11
Quelltext:
          <xul:label class="moreIndicator" value="&more.label;" anonid="more"
-----------^

Fehler: uncaught exception: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIStringBundle.GetStringFromName]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: XStringBundle :: getString :: line 17"  data: no]
What platform is this? 

l10n builder on win32 looks clean
l10n builder on linux looks clean
l10n builder on mac looks clean

Doesn't look like any of the l10n repack builders are stuck in a similar situation, broken locale ?
Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.9.1b3pre) Gecko/20081201 Shredder/3.0b1pre

It's a 64 bit Linux.
The examples I have tested, are broken.

thunderbird-3.0b1pre.de.linux-i686.tar.bz2 : Empty pane, menu usable.
thunderbird-3.0b1pre.en-GB.linux-i686.tar.bz2 : Empty pane, no menu.
thunderbird-3.0b1pre.lt.linux-i686.tar.bz2 : Empty pane, menu usable.
thunderbird-3.0b1pre.nl.linux-i686.tar.bz2 : Empty pane, menu usable.

The error console views the same file and line number.

Interesting. The file http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk-l10n/thunderbird-3.0b1pre.en-US.linux-i686.tar.bz2 is out of date, it's from 08-Sep-2008, and runs. The file http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/thunderbird-3.0b1pre.en-US.linux-i686.tar.bz2  from 01-Dec-2008 04:14 also runs nice.
The 3.0b1pre builds are leftovers from when we bumped the version to 3.0b2pre, can you try these ?
Version=3.0b2pre
BuildID=20081204030419
Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.9.1b3pre) Gecko/20081204 Shredder/3.0b2pre
- Is OK.

Version=3.0b2pre
BuildID=20081204030419
Mozilla/5.0 (X11; U; Linux i686 (x86_64); nl; rv:1.9.1b3pre) Gecko/20081204 Shredder/3.0b2pre
- Is OK.

Version=3.0b2pre
BuildID=20081204030419
Mozilla/5.0 (X11; U; Linux i686 (x86_64); lt; rv:1.9.1b3pre) Gecko/20081204 Shredder/3.0b2pre
- Is OK.

-------------

Version=3.0b2pre
BuildID=20081204030419
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk-l10n/thunderbird-3.0b2pre.af.linux-i686.tar.bz2
- Empty pane and no menu.

Version=3.0b1
BuildID=20081204114258
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/3.0b1-candidates/build2/thunderbird-3.0b1.af.linux-i686.tar.bz2
- Empty pane and no menu.

Think, this "af" is a localization problem for this language only.

All Beta versions have spot checked so far, see https://wiki.mozilla.org/Thunderbird3/TestResults/Beta1/L10N
Can anybody please have a look at the german Mac build. At Datei --> Offline, I have a interrupted line there "Gekennzeichnete Na..richten herrunterladen". 
I'am a bit confused, I'am the only one with this, because its correct in the l10n file: http://mxr.mozilla.org/l10n-central/source/de/mail/chrome/messenger/messenger.dtd#62

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; de; rv:1.9.1b3pre) Gecko/20081204 Thunderbird/3.0b1
I guess you're hitting the maximum width of menu items, http://mxr.mozilla.org/mozilla-central/source/toolkit/themes/pmstripe/global/menu.css#154.

File a bug on the German localization?
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; de; rv:1.9.1b3pre)
Gecko/20081204 Thunderbird/3.0b1

Yes, string is to long, there are 3 dots in the middle.
Nomis, please create a bug for localizations.
(In reply to comment #30)
> Yes, string is to long, there are 3 dots in the middle.
> Nomis, please create a bug for localizations.
Bug 468128
Can this be closed? I am assuming that this is now really only bug 468128?
(In reply to comment #32)
> Can this be closed?

Yes. Have checked the German build every 2 days.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.