Closed Bug 674830 Opened 9 years ago Closed 8 years ago

XML parsing error: Undefined entity on startup, Fennec broken; busts profile

Categories

(Release Engineering :: General, defect, P1, blocker)

ARM
Android
defect

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: aryx, Assigned: aki)

References

Details

(Keywords: regression, reproducible)

Attachments

(2 files)

Fennec Nightly (after upgrade from 20110725 or 26 to 20110727), Android 2.3.4, Nexus S

After upgrading the Nightly from 20110725 or 26 to 20110727, all I get is a undefined entity error on startup (in German, as the device is set to use German):

XML-Verarbeitungsfehler: Nicht definierte Entität
Adresse: chrome://browser/content/browser.xul
Zeile Nr. 60, Spalte 1:

<window id="main-window"
^
Component: General → de / German
Product: Fennec → Mozilla Localizations
QA Contact: general → german.de
Version: Trunk → unspecified
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 668807
I'm getting the same for the dutch localization.
This worked in the 2011-07-26 build, but started to occur in the 2011-07-27 03 build.
I guess something was checked in that caused all the locale builds to be bust?
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Duplicate of this bug: 668807
Component: de / German → General
Product: Mozilla Localizations → Fennec
QA Contact: german.de → general
Version: unspecified → Trunk
Portuguese, Dansk, Japanese, Suomi, Polski, Norsk, Nederlands all busted
Severity: major → critical
Keywords: regression
Priority: -- → P1
Summary: Undefined entity error on startup, Fennec broken → Undefined entity error on startup, Fennec broken; busts profile
Last known good: 58c04967ac5b (07/26)
First bad: a627b24e684e (07/27)

http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=58c04967ac5b&tochange=a627b24e684e
Aaron mentioned on IRC, could this be a regression from bug 652771?
bug 672320 looks suspicious again, as part of that range
Status: REOPENED → NEW
Axel Hecht mentioned or irc this could be the cause of releng issues.
tracking-fennec: --- → ?
Depends on: 675223
(In reply to comment #7)
> bug 672320 looks suspicious again, as part of that range

I don't understand how that could have any effect on fennec startup. (I'm not ruling out the possibility of a connection, I just don't see one.)

Can we somehow determine *what* entity is undefined when this error occurs?
I looked at the build in ftp://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2011-07-28-03-08-19-mozilla-central-android/, and that's busted.

Without going into it deeply, it looks like l10n-merged files don't show up in the apk at all, even if the there's a incomplete file in the l10n repo.

Checked with danish headUpDisplay.properties.

This is a build or releng issue.
Yeah, I just noticed the same thing. For instance, da doesn't have browser.dtd packaged at all.
Moving to RelEng
Component: General → Release Engineering
Product: Fennec → mozilla.org
QA Contact: general → release
Version: Trunk → other
Keywords: reproducible
This is hurting any Nightly users who don't use en-US.
We need someone from RelEng to look at this bug
Severity: critical → blocker
Duplicate of this bug: 676216
Assignee: nobody → aki
Blocks: 675223
No longer depends on: 675223
Blocks: 674516
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-macosx-l10n/fennec-8.0a1.de.mac.dmg launches fine (from this morning).

That means that the de locale is repacking for single-locale repacks on mac fine.

This narrows it down to:

* multilocale builds in general
** mozharness multilocale configs/script
* android specifically

Still working on this.
(In reply to comment #4)
> Portuguese, Dansk, Japanese, Suomi, Polski, Norsk, Nederlands all busted

Aaron, does that mean that the other locales work, or that you've only tested these locales?

I'm looking at http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2011-08-03-03-07-53-mozilla-central-android/mozilla-central-linux-android-nightly-build4.txt.gz and searching for browser.dtd; it seems like some locales are broken and others aren't.
(In reply to comment #17)
> (In reply to comment #4)
> > Portuguese, Dansk, Japanese, Suomi, Polski, Norsk, Nederlands all busted
> 
> Aaron, does that mean that the other locales work, or that you've only
> tested these locales?
> 
> I'm looking at
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2011-08-03-03-07-53-
> mozilla-central-android/mozilla-central-linux-android-nightly-build4.txt.gz
> and searching for browser.dtd; it seems like some locales are broken and
> others aren't.

Those are the locales with the issues I ran into. Tested all the supplied locales in the multi-locale build.
Can you check if any of the files that are mentioned as l10n-merged show up in the omni.jar?
As mentioned in IRC, I looked for the l10n-merged files in the unzipped multi apk. All the ones I've looked for are missing.

This appears to be the issue that needs solving; we're trying to determine what's causing it.
Axel wanted me to look at the merge directory, so I tried but it was removed along with the entire build directory.

I kicked off another nightly and it was still gone, so I dug deeper and found that we remove the entire build directory at the end of every nightly, which helps with getting builds out earlier but kills our ability to debug issues.

Filed bug 676391 and am now working on getting a build going either manually or in staging with the rm -rf build step disabled.
Attached patch fix merge dirSplinter Review
Found it =P
I don't know how this worked for so long, really.

* Previously, I removed but didn't mkdir the merge dir. compare-locales and make chrome-LOCALE didn't complain, and things worked somehow without this since at least January.  Added the mkdir_p() call.
* We never turn off merge_locales, so we never hit this, but I'm fixing the bug where we don't actually call 'make chrome-LOCALE' if merge_locales is off.

Armen's out and Ben's busy with 3.6.20, so flagging Coop for review. Let me know if you'd like me to ask someone else.
Attachment #550581 - Flags: review?(coop)
Attachment #550581 - Flags: review?(coop) → review+
tracking-fennec: ? → ---
Status: NEW → RESOLVED
Closed: 9 years ago9 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
New nightly finished.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
With http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1312475356.1312477801.18512.gz (http://hg.mozilla.org/mozilla-central/rev/c7ea065539d2f84a2d92c98aae474123a6c704f8) -- the mentioned Nightly above, I am still seeing this bugs behaviour
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Duplicate of this bug: 676667
Summary: Undefined entity error on startup, Fennec broken; busts profile → XML parsing error: Undefined entity on startup, Fennec broken; busts profile
Ok.
Now with a merge dir, I'm seeing some of the merged files in the apk.
Still don't see a browser.dtd on various locales.
Not entirely sure where to go from here; suggestions welcome.
I'll take stabs at it, though.
Duplicate of this bug: 677319
Status: REOPENED → ASSIGNED
This is not an issue in Aurora multilocale builds; there are all 13 browser.dtd files in the latest 7.0a2 apk.

If I'm not mistaken, this narrows it down to:

1. mozilla-central changes
2. l10n-central changes, or lack thereof
3. mozilla-central specific configuration on the buildbot/mozharness side.
Status: ASSIGNED → UNCONFIRMED
Ever confirmed: false
Status: UNCONFIRMED → NEW
Ever confirmed: true
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Using m-c revision 0a936ddb70e9, I did a

make -f client.mk build
cd ..
python mozharness/scripts/multil10n.py --config-file multi_locale/mozilla-central_linux-android.json --merge-locales --only-pull-locale-source --only-add-locales --only-package-multi

And was able to replicate the problem (no browser.dtd for specific locales).

I'm going to try backing out bug 664907 locally to see if that helps.
If it doesn't, I'll verify building 4f38df646524 works, and bisect.
(In reply to Aki Sasaki [:aki] from comment #31)
> Using m-c revision 0a936ddb70e9, I did a
> 
> make -f client.mk build
> cd ..
> python mozharness/scripts/multil10n.py --config-file
> multi_locale/mozilla-central_linux-android.json --merge-locales
> --only-pull-locale-source --only-add-locales --only-package-multi
> 
> And was able to replicate the problem (no browser.dtd for specific locales).
> 
> I'm going to try backing out bug 664907 locally to see if that helps.

Still using http://hg.mozilla.org/mozilla-central/rev/0a936ddb70e9, I backed out 

http://hg.mozilla.org/mozilla-central/rev/d6fe12d237b0
http://hg.mozilla.org/mozilla-central/rev/dd550792ca90
http://hg.mozilla.org/mozilla-central/rev/95eda84ea6ba

per bug 664907 comment 10.

I blew away the objdir, rebuilt, repackaged the multi apk.
Afterwards, we had all 14 browser.dtd files.

Reopening bug 664907 for backout.
Blocks: 664907
bug 664907 disabled on Android:
https://hg.mozilla.org/mozilla-central/rev/1d071e2fa07f
Status: ASSIGNED → RESOLVED
Closed: 9 years ago8 years ago
Resolution: --- → FIXED
Verified Fixed
Mozilla/5.0 (Android; Linux armv7l; rv:8.0a1) Gecko/20110813 Firefox/8.0a1 Fennec/8.0a1
Status: RESOLVED → VERIFIED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.