903716 - Firefox 24 beta 1 xml parse error on startup: browser.xul, devtoolsMenuBroadcaster_ChromeDebugger

Status: VERIFIED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Alex]

User Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.49 Safari/537.36

Steps to reproduce:

Upgraded from Firefox 23 Beta 10 to Firefox 24 Beta 1 (Local: de-DE)
Start Firefox.
Start Firefox in Safe-Mode.
Start Firefox with fresh profile.

All same start-up error.


Actual results:

XML-Verarbeitungsfehler: Nicht definierte Entität Adresse: chrome://browser/content/browser.xul Zeile Nr. 238, Spalte 5: <broadcaster id="devtoolsMenuBroadcaster_ChromeDebugger" ----^


Expected results:

Firefox should start normally.

Comment 1

[Alex]

Bug does not exist in en-US Version. It affects only localized versions like German.

Comment 2

[Scoobidiver (away)]

It works for me when upgrading from 23.0 Beta 10 to 24.0 Beta 1 (en-US and fr).
Can you install again Firefox 23.0 Beta 10 de (ftp://ftp.mozilla.org/pub/firefox/candidates/23.0b10-candidates/build1/win32/de/Firefox%20Setup%2023.0b10.exe) and try again?

Comment 3

[Alex]

Installed 23.0 Beta 10 de, upgraded to 24.0 Beta 1 de. 
Still crashing at start-up.

Upgraded to 24.0 Beta 1 en-US. No crash. 
Something is wrong with the german version.

Comment 4

[Scoobidiver (away)]

(In reply to Scoobidiver from comment #2)
> and fr).
I lied about that. I downloaded 24.0b1 fr because About Firefox in 23.0b10 fr says it's up-to-date but I don't think the issue here is about updating.

(In reply to Alex from comment #0)
> XML-Verarbeitungsfehler: Nicht definierte Entität Adresse:
> chrome://browser/content/browser.xul Zeile Nr. 238, Spalte 5: <broadcaster
> id="devtoolsMenuBroadcaster_ChromeDebugger" ----^
In English, it's:
XML Parsing Error: undefined entity Location: chrome://browser/content/browser.xul Line Number 238, Column 5 : <broadcaster id="devtoolsMenuBroadcaster_ChromeDebugger" ----^

Here is the faulty line:
http://mxr.mozilla.org/mozilla-beta/source/browser/base/content/browser-sets.inc#185
The string translation (a new one in 24.0) in German is: http://transvision.mozfr.org/?repo=beta&sourcelocale=en-US&locale=de&search_type=entities&recherche=browser%2Fchrome%2Fbrowser%2Fbrowser.dtd%3AchromeDebuggerMenu.label
I think the XML parser doesn't like minus but it's the same in Aurora and Nightly and no German people reported that.

I am triaging it as a German localization bug for now but maybe there's also an underlying issue about minus to fix somewhere else.

Comment 5

[André Ziegler]

I have the same issue when installing the German firefox 24 Beta.

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

I can confirm the issue with the beta manually downloaded from ftp://ftp.mozilla.org/pub/firefox/releases/24.0b1/win32/de/

The German Firefox Desktop 24 translation is not in this build, I guess the signoff has been accepted after the build tag had been set. Comparing the browser.dtd from the build and beta repository only shows the obsolete errorConsoleCmd.commandkey entity as not being in both files.

I had verified the German Firefox Desktop 24 translation while still on Aurora.

Comment 7

[Axel Hecht [:Pike]]

Over to release engineering for investigation, l10n-merge should cover issues like this.

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Archaeopteryx [:aryx] from comment #6)
> The German Firefox Desktop 24 translation is not in this build, I guess the
> signoff has been accepted after the build tag had been set. Comparing the
> browser.dtd from the build and beta repository only shows the obsolete
> errorConsoleCmd.commandkey entity as not being in both files.

I don't know what I have done, but looking now into <install-dir\browser\omni.ja\chrome\de\locale\browser\ shows that the missing strings are missing (and comparing the files shows that now).

Comment 10

[Axel Hecht [:Pike]]

My suspicion is that during the back and forth to get window l10n repacks to begin with, the setup for l10n-merge during the builds regressed. pymake path specs?

Comment 11

[Andre Klapper]

Also see bug 903742

Comment 14

[Aris]

Wouldn't it make sense to remove all locales from Mozilla servers or at least from release folders until a fixed version is up?

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

So far we know this affects German (this bug), Czech (cs, bug 903742), Japanese (ja, bug 903744), and probably Slovenian (sl, bug 903847).

I've tested French (fr), Italian (it) and Simplified Chinese (zh-CN) and they launch OK for me, so this is affecting some subset of locales rather than all.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Test all the installers

50 of the 91 locales are busted: ach, af, ak, be, bg, bn-BD, bn-IN, bs, cs, da, de, el, en-ZA, es-AR, es-CL, es-MX, et, eu, fa, fi, gl, he, hr, id, ja, kk, ko, ku, lg, lt, mai, mk, mn, mr, nb-NO, nl, nn-NO, pa-IN, pt-BR, rm, si, sl, eq, sr, sq, ta-LK, th, uk, vi, zu

Several locales are generated sequentially on a single machine, but it's not that all locales in a set are busted. eg bn-IN,br,bs,ca,cs,csb,cy,da,de were generated as a set, but only bs,cs,da,de are busted.

It might be relevant that chromeDebuggerMenu.label moved from debugger.dtd to browser.dtd in https://hg.mozilla.org/releases/mozilla-beta/rev/9e0781b97e51. In the logs there is output like this for broken installers
      browser.dtd
          +chromeDebuggerMenu.label
        debugger.dtd
            -chromeDebuggerMenu.label
Locales that worked don't appear to have this output.

The compare-locales used was http://hg.mozilla.org/build/compare-locales/rev/1fc4e9bc8287, v0.9.5.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] from comment #16)
> Several locales are generated sequentially on a single machine, but it's not
> that all locales in a set are busted. eg bn-IN,br,bs,ca,cs,csb,cy,da,de were
> generated as a set, but only bs,cs,da,de are busted.

--> 'bn-IN,bs,cs,da,de' are busted.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

For convenience I collected all the actual win32 repack logs at 
 http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/24.0b1-candidates/build2/logs/release-mozilla-beta-win32_repack_all.tar

Comment 19

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Alex from comment #3)
> Installed 23.0 Beta 10 de, upgraded to 24.0 Beta 1 de. 
> Still crashing at start-up.

...

(In reply to Scoobidiver from comment #4)
> (In reply to Scoobidiver from comment #2)
> > and fr).
> I lied about that. I downloaded 24.0b1 fr because About Firefox in 23.0b10
> fr says it's up-to-date but I don't think the issue here is about updating.


alex, scoobidiver: ftr, on win32, it is correct and expected that non-en-US beta users can only update to FF23.0b10 using "check for update". Non-en-US beta win32 users should should not be able to "check for updates" to FF24.0b1. Note: all linux and osx beta users *can* update to FF24.0b1. Sorry for the mess here, we've had a long week on this specific FF24.0b1 release, and hope to be back to our usual "all locales can update on all OS on same day" asap.

Please confirm that you "updated" by downloading and running installer for FF24.0b1.

Comment 20

[Alex]

(In reply to John O'Duinn [:joduinn] from comment #19)
> Please confirm that you "updated" by downloading and running installer for
> FF24.0b1.

Yes i have downloaded 24.0b1 direct from http://www.mozilla.org/de/firefox/beta/

Comment 21

[Axel Hecht [:Pike]]

I looked at de and csb.

The german browser.dtd has missing strings in the tagged changeset, but the file in the zip doesn't contain any merged English entities.

Cashubian (csb) on the other is fully translated, and thus doesn't mind l10n-merge not working.

This is definitely a regression in either generating or picking up merged files.

Comment 22

[Armen [:armenzg]]

I will take this and hopefully figure out what is different.

Comment 23

[Armen [:armenzg]]

I have verified that we still have the same issue for "de" on the staging run of beta2:
$ wget http://dev-stage01.build.mozilla.org/pub/mozilla.org/firefox/candidates/24.0b2-candidates/build1/win32/de/Firefox%20Setup%2024.0b2.exe
$ 7za x -so Firefox\ Setup\ 24.0b2.exe core/browser/omni.ja > foo 2> /dev/null && unzip -c foo 'chrome/*/locale/browser/browser.dtd' 2>/dev/null | grep 'ENTITY chromeDebuggerMenu.label'



$ wget http://dev-stage01.build.mozilla.org/pub/mozilla.org/firefox/candidates/24.0b2-candidates/build1/win32/fr/Firefox%20Setup%2024.0b2.exe
$ 7za x -so Firefox\ Setup\ 24.0b2.exe.1 core/browser/omni.ja > foo 2> /dev/null && unzip -c foo 'chrome/*/locale/browser/browser.dtd' 2>/dev/null | grep 'ENTITY chromeDebuggerMenu.label'
<!ENTITY chromeDebuggerMenu.label       "Débogueur du navigateur">

Comment 24

[Armen [:armenzg]]

Comparing the FF23.0b10 log:
https://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/23.0b10-candidates/build1/logs/release-mozilla-beta-win32_repack_2-bm61-build1-build1.txt.gz
VS FF24.0b1 log:
https://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/24.0b1-candidates/build2/logs/release-mozilla-beta-win32_repack_2-bm64-build1-build12.txt.gz

I think we started using "mozilla-beta" instead of "l10n" as a checkout dir but I don't clearly see the connection. I will try to reproduce after lunch.

#######
'https://hg.mozilla.org/releases/l10n/mozilla-beta/de', 'mozilla-beta\\de'
'https://hg.mozilla.org/releases/l10n/mozilla-beta/de', 'l10n\\de'
#######
python compare-locales\scripts\compare-locales -m mozilla-beta\obj-l10n\browser\locales\merged mozilla-beta/browser/locales/l10n.ini l10n de
python compare-locales\scripts\compare-locales -m mozilla-beta\obj-l10n\browser\locales\merged mozilla-beta/browser/locales/l10n.ini mozilla-beta de
#######
make installers-de
cwd: mozilla-beta\obj-l10n\browser\locales
env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'AB_CD': 'de', 'MOZ_SIGN_CMD': 'python /e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/scripts/release/signing/signtool.py --cachedir /e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/signing_cache -t /e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/token -n /e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/nonce -c /e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/scripts/release/signing/host.cert -H signing4.srv.releng.scl3.mozilla.com:9120 -H signing5.srv.releng.scl3.mozilla.com:9120 -H signing6.srv.releng.scl3.mozilla.com:9120', 'WIN32_INSTALLER_IN': '/e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/firefox.exe', 'UPLOAD_SSH_KEY': '~\\.ssh\\ffxbld_dsa', 'LOCALE_MERGEDIR': '/e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/mozilla-beta/obj-l10n/browser/locales/merged', 'COMM_REV': 'FIREFOX_23_0b10_RELEASE', 'MOZ_OBJDIR': 'obj-l10n', 'MAR': '/e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/mozilla-beta/obj-l10n/dist/host/bin/mar', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '', 'ZIP_IN': '/e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/firefox.zip', 'UPLOAD_USER': 'ffxbld', 'MBSDIFF': '/e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/mozilla-beta/obj-l10n/dist/host/bin/mbsdiff', 'UPLOAD_EXTRA_FILES': 'update/win32/da/firefox-23.0b9-23.0b10.partial.mar update/win32/da/firefox-23.0b9-23.0b10.partial.mar.asc update/win32/da/firefox-23.0b8-23.0b10.partial.mar update/win32/da/firefox-23.0b8-23.0b10.partial.mar.asc', 'UPLOAD_HOST': 'stage.mozilla.org', 'POST_UPLOAD_CMD': 'post_upload.py -p firefox -n 1 -v 23.0b10 --release-to-candidates-dir --signed', 'MOZILLA_REV': 'FIREFOX_23_0b10_RELEASE', 'MOZ_PKG_PRETTYNAMES': '1', 'MOZ_PKG_VERSION': '23.0b10'}

c:\mozilla-build\python27\python.exe e:\builds\moz2_slave\rel-m-beta-w32_rpk_2-000000000\mozilla-beta/build/pymake/make.py installers-de
cwd: mozilla-beta\obj-l10n\browser\locales
 env: {'AB_CD': 'de', 'LIB': 'd:\\msvs10\\vc\\lib;d:\\msvs10\\vc\\atlmfc\\lib;d:\\sdks\\v7.0\\lib;d:\\msvs8\\VC\\PlatformSDK\\lib;d:\\msvs8\\SDK\\v2.0\\lib;d:\\mozilla-build\\atlthunk_compat;d:\\sdks\\dx10\\lib\\x86', 'MOZ_SIGN_CMD': 'python e:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\rel-m-beta-w32_rpk_2-000000000\\\\\\\\scripts\\\\\\\\release\\\\\\\\signing\\\\\\\\signtool.py --cachedir e:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\rel-m-beta-w32_rpk_2-000000000\\\\\\\\signing_cache -t e:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\rel-m-beta-w32_rpk_2-000000000\\\\\\\\token -n e:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\rel-m-beta-w32_rpk_2-000000000\\\\\\\\nonce -c e:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\rel-m-beta-w32_rpk_2-000000000\\\\\\\\scripts\\\\\\\\release\\\\\\\\signing\\\\\\\\host.cert -H signing6.srv.releng.scl3.mozilla.com:9120', 'LIBPATH': 'd:\\msvs10\\vc\\lib;d:\\msvs10\\vc\\atlmfc\\lib;\\c\\WINDOWS\\Microsoft.NET\\Framework\\v2.0.50727', 'WIN32_INSTALLER_IN': 'e:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\rel-m-beta-w32_rpk_2-000000000\\\\\\\\firefox.exe', 'UPLOAD_SSH_KEY': '~\\.ssh\\ffxbld_dsa', 'LOCALE_MERGEDIR': '/e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/mozilla-beta/obj-l10n/browser/locales/merged', 'MOZILLA_OFFICIAL': '1', 'COMM_REV': 'FIREFOX_24_0b1_RELEASE', 'MOZ_OBJDIR': 'obj-l10n', 'MAR': '/e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/mozilla-beta/obj-l10n/dist/host/bin/mar.exe', 'PATH': 'd:\\msvs10\\VSTSDB\\Deploy;d:\\msvs10\\Common7\\IDE\\;d:\\msvs10\\VC\\BIN;d:\\msvs10\\Common7\\Tools;d:\\msvs10\\VC\\VCPackages;c:\\mozilla-build\\nsis-2.46u;c:\\mozilla-build\\python27;c:\\mozilla-build\\buildbotve\\scripts;C:\\mozilla-build\\msys\\local\\bin;c:\\mozilla-build\\wget;c:\\mozilla-build\\7zip;c:\\mozilla-build\\blat261\\full;c:\\mozilla-build\\python;c:\\mozilla-build\\svn-win32-1.6.3\\bin;c:\\mozilla-build\\upx203w;c:\\mozilla-build\\emacs-22.3\\bin;c:\\mozilla-build\\info-zip;c:\\mozilla-build\\nsis-2.22;c:\\mozilla-build\\nsis-2.33u;c:\\mozilla-build\\nsis-2.46u;c:\\mozilla-build\\wix-351728;c:\\mozilla-build\\hg;c:\\mozilla-build\\python\\Scripts;c:\\mozilla-build\\kdiff3;c:\\mozilla-build\\yasm;.;C:\\mozilla-build\\msys\\local\\bin;C:\\mozilla-build\\msys\\mingw\\bin;C:\\mozilla-build\\msys\\bin;c:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\BIN\\amd64;c:\\Windows\\Microsoft.NET\\Framework64\\v3.5;c:\\Windows\\Microsoft.NET\\Framework64\\v3.5\\Microsoft .NET Framework 3.5 (Pre-Release Version);c:\\Windows\\Microsoft.NET\\Framework64\\v2.0.50727;c:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\VCPackages;c:\\Tools\\msvs9\\Common7\\IDE;c:\\Tools\\msvs9\\Common7\\Tools;c:\\Tools\\msvs9\\Common7\\Tools\\bin;c:\\Tools\\sdks\\v7.0\\bin\\x64;c:\\Tools\\sdks\\v7.0\\bin\\win64\\x64;c:\\Tools\\sdks\\v7.0\\bin;c:\\windows\\System32;c:\\windows;c:\\windows\\System32\\Wbem;c:\\mozilla-build\\moztools-x64\\bin;c:\\mozilla-build\\vim\\vim72', 'INCLUDE': 'd:\\msvs10\\vc\\include;d:\\msvs10\\vc\\atlmfc\\include;d:\\sdks\\v7.0\\include;d:\\sdks\\v7.0\\include\\atl;d:\\msvs8\\VC\\PlatformSDK\\include;d:\\sdks\\dx10\\include', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '', 'WIN32_REDIST_DIR': 'd:\\msvs10\\VC\\redist\\x86\\Microsoft.VC100.CRT', 'ZIP_IN': 'e:\\\\\\\\builds\\\\\\\\moz2_slave\\\\\\\\rel-m-beta-w32_rpk_2-000000000\\\\\\\\firefox.zip', 'UPLOAD_USER': 'ffxbld', 'USE_PYMAKE': '1', 'MBSDIFF': '/e/builds/moz2_slave/rel-m-beta-w32_rpk_2-000000000/mozilla-beta/obj-l10n/dist/host/bin/mbsdiff.exe', 'UPLOAD_EXTRA_FILES': '', 'UPLOAD_HOST': 'stage.mozilla.org', 'POST_UPLOAD_CMD': 'post_upload.py -p firefox -n 2 -v 24.0b1 --release-to-candidates-dir --signed', 'MOZILLA_REV': 'FIREFOX_24_0b1_RELEASE', 'MOZ_PKG_PRETTYNAMES': '1', 'MOZ_PKG_VERSION': '24.0b1'}

Comment 25

[Alex Keybl [:akeybl]]

Tracking for resolution in 24.0b2 (going to build EOD today). If it remains unresolved in 24.0b2, we will still go out with updates/stubs and consider fixing this elsewhere (bouncer?).

Comment 26

[Rail Aliiev [:rail]]

Attachment: LOCALE_MERGEDIR.diff

It turns out that python installed on w64 machines doesn't understand msys paths (msys /e/builds vs e:/builds or e:\builds).

Comment 27

[Aki Sasaki (not active)]

Comment on attachment 789241 [details] [diff] [review]
LOCALE_MERGEDIR.diff

\o/

Comment 28

[Rail Aliiev [:rail]]

Comment on attachment 789241 [details] [diff] [review]
LOCALE_MERGEDIR.diff

https://hg.mozilla.org/build/tools/rev/34eb63ff8684

Comment 31

[Aris]

The problem seems to be resolved on German beta 2 of Fx 24.

Comment 32

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The German builds should work because they picked up the signed-off, complete German localization. For verifying the fix, a locale incomplete on fx_beta should be used. You can these here: https://l10n.mozilla.org/shipping/dashboard

Czech (cs) and Japanese (ja) are still incomplete.

Comment 33

[Michal Stanke (Mozilla.cz) [:mstanke][:MikkCZ]]

I've tested Czech (cs) 2nd beta from FTP candidates folder and it works (despite the own localization is incomplete and we are working on it).

Comment 34

[Rail Aliiev [:rail]]

Thank you for the verification. I also tested a couple of builds and they look fine.

Comment 36

[u279076]

Henrik, is this something Mozmill can/should have caught?

Comment 37

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #36)
> Henrik, is this something Mozmill can/should have caught?

We catch those problems yes, but there haven't been run any update tests for other locales than en-US this time on Windows. Instead those were only executed on Linux and OS X, and those platforms are not affected by this bug AFAICS.

Comment 38

[Tracy Walker [:tracy]]

The Windows l10n builds were intentionally left off the mozmill testing configs for the 24beta1 build 2 testing cycle because we knew those builds were broken. Release Management was fine going ahead with broken Windows l10n builds for this beta.

Comment 39

[Aris]

Knowing that broken localized beta 1 builds shouldn't have been released at all. Many users experienced issues only because broken builds were 'intentionally' released. That is bad policy and not a good way to keep Firefox users happy, even if they are beta users.

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

Tracy is talking about update testing in comment #38, as we knew we had an issue with creating the mar files for doing updates. We didn't know the installers were broken when we made them available on the website. There are discussions going on about how this was missed and we will avoid it in future.

Comment 41

[Aris]

OK, thanks for clarification.

Comment 42

[Paul Silaghi, QA [:pauly]]

(In reply to Nick Thomas [:nthomas] from comment #15)
> So far we know this affects German (this bug), Czech (cs, bug 903742),
> Japanese (ja, bug 903744), and probably Slovenian (sl, bug 903847).
The above locales of FF 24b5 start with no problems.