1570563 - MinGW x64 builds seem to be missing some sort of l10n data

Status: RESOLVED FIXED

(Firefox Build System :: General: Unsupported Platforms, defect)

Description

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

The x64 debug and opt firefox-ui-functional-remote-e10s tests are failing. Comparing the x86 passing and x64 failing builds I found error messages of the form:

console.warn: "[fluent] Request for keys failed because no resource bundles got generated.\n keys: [{"args":null,"id":"safeb-blocked-unwanted-page-title"},{"args":null,"id":"safeb-blocked-unwanted-page-title"},{"args":null,"id":"safeb-blocked-unwanted-page-short-desc"},{"args":null,"id":"safeb-palm-accept-label"},{"args":null,"id":"safeb-palm-see-details-label"},{"args":{"sitename":"www.itisatrap.org"},"id":"safeb-blocked-unwanted-page-error-desc-override"},{"args":null,"id":"safeb-blocked-unwanted-page-learn-more"}].\n resourceIds: ["branding/brand.ftl","browser/safebrowsing/blockedSite.ftl"]."

This is coming from here.

Examples: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f597adec7adce690fdd2172dc3aeb67497110f2b&selectedJob=259079163

Comment 1

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

flod, is there any chance you can short-circuit a bunch of searching and provide any insight or information about where to look for these bundles in a correctly created build; and how they're made?

Comment 2

[Francesco Lodolo [:flod]]

Tentatively redirecting to pike, since I know very little about build system.

Comment 3

[Axel Hecht [:Pike]]

So, the culprit in the logs is

console.warn: "[fluent] Request for keys failed because no resource bundles got generated...

That indicates that no Fluent works for you.

Here's what I checked with the target.zip from https://tools.taskcluster.net/groups/S2XkAlX9SW6-_ajPDUwoxg/tasks/czQxfVEsS-yz2m7n_Zjqsw/runs/0/artifacts:

The ftl files are packaged in omni.ja and browser/omni.ja. The update.locale and res/multilocale.txt both refer to en-US. The components/components.manifest shows the l10n registry items.

That's all the files I know that go in to the runtime behavior, and they're all looking OK.

Means, there's some runtime inspection to do, and I don't have Windows. Tom, can you do those?

First up, check Services.locale.appLocalesAsBCP47. It should return ["en-US"]. Also check the files in resource://app/localization/ and resource://gre/localization/.

Next, debug https://searchfox.org/mozilla-central/source/intl/l10n/L10nRegistry.jsm#131. That's the generator that doesn't return any data. And I just don't know why it wouldn't, from looking at the build.

CCing Zibi to be in the loop.

Comment 4

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

resource://gre/localization/en-US/ has three folders: crashreporter, security, toolkit, with subfolders and .ftl files.

resource://app/localization (and localtion/en-US) gives File Not Found Firefox can’t find the file at jar:file:///C:/.../browser/omni.ja!/localization. However if I extract omni.ja then I do see localization/en-US present. Maybe something with resource:// URIs is broken...?

I placed a debugger; command at the top of generateBundles and ran firefox with --jsdebugger. The Browser toolbox opens up; but doesn't break there. How do I induce this code to be called?

Comment 5

[Axel Hecht [:Pike]]

The trailing slash matters in the resource protocol, resource://app/localization doesn't work, but resource://app/localization/ should.

If you open preferences, you should trigger fluent code paths. They're not yet part of the startup.

Comment 6

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

It seems like this has to do with the MinGW Content Processes due sync localization and non-MinGW content processes doing async.

Can't figure out why though. This might be a weird MinGW bug...

Comment 7

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

Okay yeah this is really strange. In this commit (this try run using x64 debug) I added logging outputting mIsSync.

Triggering itisatrap.org I get the following logs:

[Child 16412: Main Thread]: E/TomLoc Localization::Localization mIsSync=0
[Child 16412: Main Thread]: E/TomLoc DOMLocalization::DOMLocalization mIsSync=0
[Child 16412: Main Thread]: E/TomLoc DocumentL10n::DocumentL10n before mIsSync=0
[Child 16412: Main Thread]: E/TomLoc DocumentL10n::DocumentL10n after mIsSync=0
[Child 16412: Main Thread]: E/TomLoc DocumentL10n::Init 1 mIsSync=0
[Child 16412: Main Thread]: E/TomLoc Localization::Init mIsSync=0
[Child 16412: Main Thread]: E/TomLoc DocumentL10n::Init 2 mIsSync=0

And yet (same run) I break into the debugger in Localization.jsm::GetLocalization (called from Localization::Init and in the jsdebugger... sync is true!

Specific STR:

1. Set up logging by creating a .bat file containing:

set MOZ_LOG=TomLoc:5
set MOZ_LOG_FILE=log
firefox.exe

2. Run browser Go to preferences. Change home page and new tab page to about:blank
3. Open Browser Content Toolbox, open L10nRegistry.jsm, place a breakpoing at the top of generateBundles and generateBundlesSync.
4. Close browser, delete logs
5. Open browser. See blank page
6. Open Browser Content Toolbox
7. Paste or type http://www.itisatrap.org/firefox/its-a-trap.html into the address bar
8. The jsdebugger should break. Observe the stack and the lowest stack frame should be getLocalization. Click it and observe that 'sync' is true
9. Hit run to continue past breakpoint
10. Observe fluent error messages in the browser toolbox console
11. Observe the deceptive site warning has no strings
12. Close the browser
13. Only one of the log files for a child process should have messages. mIsSync=0 in all of the messages.

Comment 8

[Axel Hecht [:Pike]]

I don't have a good idea. AFAIK, only https://searchfox.org/mozilla-central/rev/0ffa9e372df56c95547fed9c3433ddec4fbf6f11/toolkit/components/mozintl/mozIntl.jsm#735 instantiates sync, or documents with a data-l10n-sync attribute on the document element.

Maybe something's broken in the c++/js interfacing logic that negates the bool? But that's me stabbing in a really dark place, not an educated guess. "compiler bug" would be a similarly educated stab. Or "sandbox". I'm really just name dropping at this point.

If you can tweak your logging such that it shows up on treeherder logs, I could look there. Also, good to log on both the c++ and the js side, in the hopes that one can inspect if values travel back and forth as expected.

Comment 9

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

Nathan, the bool change was suggested to me by dmajor; and I confirmed it fixed this problem.

The double/float change was found from doing a diff, and finding https://hg.mozilla.org/mozilla-central/rev/72856ca8f55cd558c29a396732a956cee1b69cb8

I did two try runs, one without the change: https://treeherder.mozilla.org/#/jobs?repo=try&revision=71fb65c143fdb543eb5c36de8a5b578487e1dee6&selectedJob=259980765
And one with: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a01b21ee8623a9f39fcb752d11d5aceb163b082e&selectedJob=259986024
(Note that I haven't gotten xpcshell tests green on MinGW, so there's a lot of orange there.)

But the failures for test_attributes and test_params did go away so I believe this change is needed too.

Comment 10

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

Attachment: Bug 1570563 - Sync in xptcstubs code from x86_64 to x86_64_gnu r?froydnj

This contains two changes. One is to zero out the top 56 bits for a bool
conversion so we can pass bools correctly from cpp -> js. This was also
needed in Bug 821628, 1513725

The other change copies the fix in Bug 689288 where we cast a float value
as a double.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e113011696a9
Sync in xptcstubs code from x86_64 to x86_64_gnu r=froydnj

Comment 12

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/e113011696a9

Comment 13

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

Comment on attachment 9083101 [details]
Bug 1570563 - Sync in xptcstubs code from x86_64 to x86_64_gnu r?froydnj

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Tor will need this patch
• User impact if declined: Tor will have to apply it manually
• Fix Landed on Version: 70
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is a super subtle change. However it's backed up with some tests I ran (On try, I haven't enabled the tests on treeherder generally) and most importantly it only applies to the mingw-clang build.
• String or UUID changes made by this patch:

Comment 14

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

[Tracking Requested - why for this release]: Tor needs this patch.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9083101 [details]
Bug 1570563 - Sync in xptcstubs code from x86_64 to x86_64_gnu r?froydnj

Fix for mingw-clang builds which Tor uses. Approved for 68.1esr. Calling this wontfix for 69 since those aren't builds we ship.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/b70b48361761