1442145 - Nightly localized builds fail to start (XML Parsing Error: undefined entity)

Status: VERIFIED FIXED

(L20n :: Python Library, defect)

Description

[Alice0775 White]

[Tracking Requested - why for this release]: Start up error

XML パースエラー: 定義されていない実体が使用されています。
URL: chrome://browser/content/browser.xul
行番号: 327, 列番号: 5:    <key id="key_toggleReaderMode" keycode="&toggleReaderMode.win.keycode;" command="View:ReaderView" disabled="true"/>
----^

Comment 1

[Francesco Lodolo [:flod]]

Are you using a full localized build in Japanese? Which OS?

That key has been introduced recently, so it's missing in Japanese. 
A full build is expected to fall back to English at build time.

Comment 2

[Alice0775 White]

Windows10 1709 build 16299.248 x64.

Steps To Reproduce:
1. unzip the following 16th-Feb-2018 ja nightly[1] and launch nightly.
2. Auto/manual update from About Nightly and restart

[1]
http://archive.mozilla.org/pub/firefox/nightly/2018/02/2018-02-16-10-40-33-mozilla-central-l10n/firefox-60.0a1.ja.win64.zip

Actual Results:
Start up error


FYI,
Directly download latest ja Nightly[2], and launch it. it works as expected.

[2]
http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/firefox-60.0a1.ja.win64.zip

Comment 3

[Alice0775 White]

(In reply to Francesco Lodolo [:flod] from comment #1)
> Are you using a full localized build in Japanese? Which OS?
> 
> That key has been introduced recently, so it's missing in Japanese. 
> A full build is expected to fall back to English at build time.




Windows10 1709 build 16299.248 x64.

Steps To Reproduce:
1. unzip the following 16th-Feb-2018 ja nightly[1] and launch nightly.
2. Auto/manual update from About Nightly and restart

[1]
http://archive.mozilla.org/pub/firefox/nightly/2018/02/2018-02-16-10-40-33-mozilla-central-l10n/firefox-60.0a1.ja.win64.zip

Actual Results:
Start up error


FYI,
Directly download latest ja Nightly[2], and launch it. it also does not work.

[2]
http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/firefox-60.0a1.ja.win64.zip

Comment 4

[Francesco Lodolo [:flod]]

So, based on [2] the build does the right thing, it looks like something goes wrong with the update.

This bug doesn't really belong to Japanese: it's not a bug in the localization, it's a bug in either the update system or the build system.

Trying to reproduce on Mac in the meantime.

Comment 5

[Alice0775 White]

(In reply to Francesco Lodolo [:flod] from comment #4)
> So, based on [2] the build does the right thing, it looks like something
> goes wrong with the update.
> 

Sorry
Please see comment #3, [2] the build also fails to start.

Comment 6

[Axel Hecht [:Pike]]

Looking at https://ftp.mozilla.org/pub/firefox/nightly/2018/03/2018-03-01-02-47-24-mozilla-central-l10n/firefox-60.0a1.ja.win64.zip (which should be latest right now), browser.dtd in browser/omni.ja has no trailing english strings.

Indicates that l10n-merge doesn't work for that build right now.

Comment 7

[Axel Hecht [:Pike]]

This is a python-fluent bug, it seems, triggering a compare-locales error, which doesn't break the build.

04:55:34     INFO -  Error running mach:
04:55:34     INFO -      ['compare-locales', '--merge', 'z:/task_1519879020/build/src/obj-firefox/browser/locales/merge-dir', 'z:/task_1519879020/build/src/browser/locales/l10n.toml', 'z:/task_1519879020/build/l10n', 'ja']
04:55:34     INFO -  The error occurred in code that was called by the mach command. This is either
04:55:34     INFO -  a bug in the called code itself or in the way that mach is calling it.
04:55:34     INFO -  You should consider filing a bug for this issue.
04:55:34     INFO -  If filing a bug, please include the full output of mach, including this error
04:55:34     INFO -  message.
04:55:34     INFO -  The details of the failure are as follows:
04:55:34     INFO -  AttributeError: 'NumberExpression' object has no attribute 'name'
04:55:34     INFO -    File "z:/task_1519879020/build/src\tools/compare-locales/mach_commands.py", line 55, in compare
04:55:34     INFO -      return cmd.handle(**kwargs)
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/compare-locales\compare_locales\commands.py", line 149, in handle
04:55:34     INFO -      merge_stage=merge, clobber_merge=clobber)
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/compare-locales\compare_locales\compare.py", line 654, in compareProjects
04:55:34     INFO -      comparer.compare(reffile, l10n, mergepath, extra_tests)
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/compare-locales\compare_locales\compare.py", line 522, in compare
04:55:34     INFO -      if refent.equals(l10nent):
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/compare-locales\compare_locales\parser.py", line 685, in equals
04:55:34     INFO -      other.entry, ignored_fields=self.ignored_fields)
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/fluent\fluent\syntax\ast.py", line 120, in equals
04:55:34     INFO -      if not scalars_equal(elem1, elem2, ignored_fields):
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/fluent\fluent\syntax\ast.py", line 39, in scalars_equal
04:55:34     INFO -      return node1.equals(node2, ignored_fields)
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/fluent\fluent\syntax\ast.py", line 123, in equals
04:55:34     INFO -      elif not scalars_equal(field1, field2, ignored_fields):
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/fluent\fluent\syntax\ast.py", line 39, in scalars_equal
04:55:34     INFO -      return node1.equals(node2, ignored_fields)
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/fluent\fluent\syntax\ast.py", line 120, in equals
04:55:34     INFO -      if not scalars_equal(elem1, elem2, ignored_fields):
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/fluent\fluent\syntax\ast.py", line 39, in scalars_equal
04:55:34     INFO -      return node1.equals(node2, ignored_fields)
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/fluent\fluent\syntax\ast.py", line 123, in equals
04:55:34     INFO -      elif not scalars_equal(field1, field2, ignored_fields):
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/fluent\fluent\syntax\ast.py", line 39, in scalars_equal
04:55:34     INFO -      return node1.equals(node2, ignored_fields)
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/fluent\fluent\syntax\ast.py", line 116, in equals
04:55:34     INFO -      field1 = sorted(field1, key=sorting)
04:55:34     INFO -    File "z:/task_1519879020/build/src\third_party/python/fluent\fluent\syntax\ast.py", line 111, in <lambda>
04:55:34     INFO -      'variants': lambda elem: elem.key.name,
04:55:34     INFO -  z:/task_1519879020/build/src/toolkit/locales/l10n.mk:189: recipe for target 'merge-ja' failed

This falls between the sick and the dead guy :-(

Comment 8

[Francesco Lodolo [:flod]]

AttributeError: 'NumberExpression' object has no attribute 'name'

I believe it's the bug that was fixed yesterday for migration
https://github.com/projectfluent/python-fluent/pull/53

Comment 9

[Francesco Lodolo [:flod]]

Should we ask releng to stop updates for Nightly? I'm not sure you can recover from this without downloading a new build.

Comment 10

[Axel Hecht [:Pike]]

Yes.

also, the l10n dashboard is affected, https://l10n.mozilla.org/shipping/signoffs/ja/fx60. Seems c-l errors don't show up in the build status at all anymore, too.

Comment 13

[Axel Hecht [:Pike]]

Taking. Aryx, I'll try to get a path for you in the next half hour, would you mind landing that right on central so that we get the fix into the next nightly?

This is an existing fix, we just haven't cut a release of the library to uplift into m-c yet. Which is what I'd just do real quick.

Comment 16

[Staś Małolepszy :stas]

Axel, are you going to publish a new python-fluent release for this? Or do you want me to do it?

Comment 17

[Francesco Lodolo [:flod]]

Asked Releng to stop updates to Nightly in the meantime

Comment 18

[Francesco Lodolo [:flod]]

(In reply to Staś Małolepszy :stas from comment #16)
> Axel, are you going to publish a new python-fluent release for this? Or do
> you want me to do it?

Clearing the NI: https://github.com/projectfluent/python-fluent/releases/tag/0.6.4

Comment 19

[Axel Hecht [:Pike]]

Attachment: bug 1442145, update fluent to 0.6.4,

This picks up an existing fix for variant keys, which we started
using in gecko. The lack of this fix broke l10n-merge, and thus
l10n nightlies.

Review commit: https://reviewboard.mozilla.org/r/224246/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/224246/

Comment 20

[Francesco Lodolo [:flod]]

Comment on attachment 8955059 [details]
bug 1442145, update fluent to 0.6.4,

https://reviewboard.mozilla.org/r/224246/#review230172

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/cd9545935eb5
update fluent to 0.6.4, r=flod a=nightly-fix

Comment 22

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/cd9545935eb5

Comment 25

[Axel Hecht [:Pike]]

OK, some locales are affected, some are not. The list of affected locales is 

as, bn-BD, bn-IN, en-GB, en-ZA, es-ES, gn, gu-IN, hi-IN, ia, ja, kn, ko, mai, mr, my, ne-NP, or, pa-IN, si, te, zh-CN, zh-TW

The other locales happen to have a differing accesskey, which keeps the code from even seeing the [1] part of the string, and they exit early. Good for them, as they're not affected by a broken nightly.

Once we have all updates, we should publish them through update service again.

Comment 28

[Francesco Lodolo [:flod]]

I've downloaded Japanese for macOS and it starts correctly
https://www.mozilla.org/firefox/nightly/all/

Releng is re-enabling updates on Nightly.

For those affected by this bug: downloading a fresh Nightly build from the link above would work.

I'm trying to understand if Firefox is able to recover by itself from this kind of problem (using the broken Japanese on macOS, I still have access to About and force update search), but I'm getting confusing results.

Comment 29

[Francesco Lodolo [:flod]]

One thing that finally worked for me on macOS is to open the About dialog from command line

./firefox -chrome chrome://browser/content/aboutDialog.xul

That should trigger the download of the latest version, and let you update correctly

Comment 30

[emil_prager]

(In reply to Francesco Lodolo [:flod] from comment #29)
> One thing that finally worked for me on macOS is to open the About dialog
> from command line
> 
> ./firefox -chrome chrome://browser/content/aboutDialog.xul
> 
> That should trigger the download of the latest version, and let you update
> correctly

Thanks for the tip with starting with the aboutDialog.xul argument! 
However, using this trick on Windows (10, 64-bit) on the affected 32-bit build didn't work. I could see indeed the "About Firefox Nightly" dialog window, where the displayed version was "60.0a1 (2018-02-28) (32-bit)" (firefox.exe file version is 60.0.0.6633). Apparently the new version was already downloaded and ready to be installed, since the button had the label "Restart to update Nightly". However, after each restart attempt, the same error was displayed as when starting with other (or no) arguments:

XML Parsing Error: undefined entity
Location: chrome://browser/content/browser.xul
Line Number 327, Column 5:    <key id="key_toggleReaderMode" keycode="&toggleReaderMode.win.keycode;" command="View:ReaderView" disabled="true"/>
----^

Note: Only the 32-bit build was affected on Windows, the 64-but one (published on the same day) had no problem.
The locale was en-GB, if I recall correctly.
System info: Windows 10 Version 1709 build 17074.1002, 64-bit, Home Insider Preview Ed.

PS: Downloading and installing a fresh new build solved the problem. 
PPS: If the affected installation was an alternative one (for your user only) and you want to upgrade that installation, and not perform a brand new system wide installation, you can do that by choosing "No" on the Windows UAC dialog window, then choosing Custom installation and specifying the old installation path (in my case it was %LocalAppData%\Mozilla Firefox Nightly\).

Comment 31

[Francesco Lodolo [:flod]]

(In reply to emil_prager from comment #30)
> Thanks for the tip with starting with the aboutDialog.xul argument! 
> However, using this trick on Windows (10, 64-bit) on the affected 32-bit
> build didn't work. I could see indeed the "About Firefox Nightly" dialog
> window, where the displayed version was "60.0a1 (2018-02-28) (32-bit)"
> (firefox.exe file version is 60.0.0.6633). Apparently the new version was
> already downloaded and ready to be installed, since the button had the label
> "Restart to update Nightly". However, after each restart attempt, the same
> error was displayed as when starting with other (or no) arguments:

Ugh, that's far from great. I wonder if someone familiar with the update system could help understanding why this happens.

Comment 32

[:Gijs (he/him)]

(In reply to Francesco Lodolo [:flod] from comment #31)
> (In reply to emil_prager from comment #30)
> > Thanks for the tip with starting with the aboutDialog.xul argument! 
> > However, using this trick on Windows (10, 64-bit) on the affected 32-bit
> > build didn't work. I could see indeed the "About Firefox Nightly" dialog
> > window, where the displayed version was "60.0a1 (2018-02-28) (32-bit)"
> > (firefox.exe file version is 60.0.0.6633). Apparently the new version was
> > already downloaded and ready to be installed, since the button had the label
> > "Restart to update Nightly". However, after each restart attempt, the same
> > error was displayed as when starting with other (or no) arguments:
> 
> Ugh, that's far from great. I wonder if someone familiar with the update
> system could help understanding why this happens.

:mhowell might know?

Comment 34

[Molly Howell (she/her) (mostly inactive)]

I'm guessing, but it could be that when this bug happens, startup doesn't get far enough to clean up the old update files, so maybe the same update to the bad build is getting applied over and over. If that's what's happening, then deleting the directory %LOCALAPPDATA%\Mozilla\updates should be enough to allow the new update to come through.

Comment 35

[emil_prager]

(In reply to Matt Howell [:mhowell] from comment #34)
> If that's what's happening, then deleting the directory %LOCALAPPDATA%\Mozilla\updates should be enough
> to allow the new update to come through.
Thanks! I'll remember that and try it with the next occasion.