1527463 - Automation work to enable Widevine on Windows aarch64

Status: VERIFIED FIXED

(Firefox Build System :: Task Configuration, task)

Description

From cpearce

I'm working on getting Encrypted Media Extensions (EME) running in Firefox on Windows on ARM64. EME is the mechanism that sites like Netflix and Hulu use to play DRM protected video on the web platform.

The video is decrypted by a Content Decryption Module (CDM) plugin, which in this case is the Widevine CDM DLL provided by Google, which is loaded by and runs in a sandboxed child process inside the plugin-container.exe binary.

We think the shortest path to making Netflix work in Firefox on Windows on ARM is to run the x86 Widevine DLL inside an x86 plugin-container.exe as a child process of an aarch64 Firefox parent process. I have gotten this working in a local build, playing Netflix even.

To make this work, among other things, I had to build Firefox twice; once in aarch64, and a second time for x86, and copy the relevant x86 binaries into a subdirectory of the aarch64 build. This is where your team comes in. I don't know how to make the build system do this by itself. Can someone on your team help us?

Given that we expect/hope this solution to be temporary, Anthony suggested that the easiest and quickest path forward may be to just download the x86 binaries (and a couple of other needed files) which matches the version of mozilla-central being built for aarch64 via the artifact build mechanism at build time, thereby avoiding building everything for a second time altogether. This isn't entirely satisfying, but since we hope that this is a temporary solution, and we'd like to ship ASAP, this might be acceptable. You or the people on your team may be able to find a better way.

Timeframe is FF 68

---

We discussed in our team meeting today

Difficult to build both as part of the build
Taskcluster delay if you depend on x86 build for aarch64
First approach would be to to build as a host program.
If this doesn't work, build w/o x86 plugin container, combine plugin container at a later time in build, need releng support for that
Do we need to test on m-c? Or not? (My understanding that testing for aarch64 builds was still under development)

Comment 1

[Mike Hommey [:glandium]]

First approach would be to to build as a host program.

host for aarch64 is x86_64, not x86. And host cannot be switched to x86 because that means using a x86 rust, which would exhaust its address space while compiling. And going back to separate host for rust and the build system would bring us back to a lot of pain.

If this doesn't work, build w/o x86 plugin container, combine plugin container at a later time in build, need releng support for that

The simplest approach is to create a task that depends on both the x86 and aarch64 builds, and combines them.

Comment 2

[Mike Hommey [:glandium]]

download the x86 binaries (and a couple of other needed files)

This is under-specified to be actionable.

Comment 3

[Nathan Froyd [:froydnj]]

(In reply to Mike Hommey [:glandium] from comment #2)

download the x86 binaries (and a couple of other needed files)

This is under-specified to be actionable.

ajones claims that you "just" need plugin-container.exe and xul.dll. I think we at least need everything xul.dll depends on, and I think we also need all the redist files we pull in from msvc. There are probably a couple of other things as well.

The easiest thing to do would be to drop an x86 directory in the aarch64 package, which contains everything that a normal x86 firefox install would contain. I suspect that would be too much (omni.ja, various browser resources, etc.), but it would be the easiest thing to do to make the combined build glandium talks about in comment 1 to work. We can then incrementally pare things back.

Comment 4

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Mike Hommey [:glandium] from comment #2)

download the x86 binaries (and a couple of other needed files)

This is under-specified to be actionable.

Based on testing on my C630 the set of required x86 files is:

plugin-container.exe
plugin-container.exe.sig
xul.dll
xul.dll.sig
gmp-clearkey/0.1/clearkey.dll
gmp-clearkey/0.1/manifest.json
nss3.dll
mozglue.dll
lgpllibs.dll
msvcp140.dll
vcruntime140.dll

The .sig files are generated by the Bs taskcluster job, so it may be easiest to download the complete packaged zip, after the signing step, and extract what we need from that.

They need to be placed in a known location; my patches (which you can see in a bundle in Bug 1523765) assumed they were in the "i686" subdir of the package.

The directory structure of the clearkey files needs to be preserved, i.e. if the x86 files are in the i686 subdir, then clearkey.dll needs to be located at i686/gmp-clearkey/0.1/clearkey.dll.

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1527463 - Get full artifact build symbols for aarch64 EME builds.

We need to have full symbols uploaded for the upcoming EME-enabled
win64-aarch64 nightlies, and the tasks to do that are derived from the
nightly itself, which is going to be an artifact build. Bug 1527463 took
care of adding the option to enable that, and we turn it on for
EME-enabled builds.

MOZ_ARTIFACT_TASK_WIN32_OPT is not exactly the right thing, but we're
already using it to enable EME in
browser/config/mozconfigs/win64-aarch64/common-opt and is only set on
those builds.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1527463 - Enable EME on win64-aarch64 nightlies.

Bug 1534522 added win64-aarch64-eme/opt builds, which are artifact builds
that glue together a win64-aarch64/opt build and a win32/opt build.

This enables EME on the corresponding nightlies in a slightly different
way:

• this adds a no-eme build that corresponds to win64-aarch64/opt.
• this turns the existing nightly into an artifact build that glues
  together that no-eme build and the win32 nightly.

The no-eme build cannot have the nightly attribute set, first because
the beetmover transform fails in that case, and because that would imply
shipping those builds, but they're not meant to be shipped this way.

It also has run-on-projects set to an empty list so that it doesn't
appear by default in mach try fuzzy, while still being triggered when
needed due to being a dependency of the nightly build.

It is preferable to keep the win64-aarch64{,-eme}/opt builds untouched
to make things easier for try (the win64-aarch64 ones being the main
ones to try; also, the -eme builds currently fail with --artifacts).

Ideally, like in bug 1534522, we'd add a diffoscope build to ensure
the variations between the nightly and its base no-eme build are within
control, but currently, that would trigger nightlies on every push,
which is not desirable. Ideally, they'd trigger whenever both their
dependencies are in the target task graph. We live that to a followup.

Depends on D23639

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/235af3330b73
Get full artifact build symbols for aarch64 EME builds. r=mshal
https://hg.mozilla.org/mozilla-central/rev/74b3bf36f5e8
Enable EME on win64-aarch64 nightlies. r=tomprince, a=CristianB

Comment 8

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/cc2984b1f9f2
Backed out 2 changesets for bustages a=backout

Comment 9

[Andreea Pavel [:apavel]]

This got backed out.

Comment 10

[Mike Hommey [:glandium]]

Interestingly, despite the beetmover bustage, we still did get something published. I got nightly 20190322012300 just now, and it was before the backout (and after the landing). Chris, could you test EME on that nightly?

Comment 11

[Mike Hommey [:glandium]]

Tom, what do you think about adding the same kind of hack as for target.generated-files.tar.xz, but for target.langpack.xpi? Blanket r+ to reland with that?

Comment 12

[Mike Hommey [:glandium]]

Attachment: Bug 1527463 - Apply same changes as win64-aarch64 nightlies to the shippable builds.

Depends on D23640

Comment 13

[Tom Prince [:tomprince]]

(In reply to Mike Hommey [:glandium] from comment #11)

Tom, what do you think about adding the same kind of hack as for target.generated-files.tar.xz, but for target.langpack.xpi? Blanket r+ to reland with that?

On further reflection, this seems reasonable.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b7297e93ee98
Get full artifact build symbols for aarch64 EME builds. r=mshal
https://hg.mozilla.org/integration/mozilla-inbound/rev/4983a9d119c7
Enable EME on win64-aarch64 nightlies. r=tomprince
https://hg.mozilla.org/integration/mozilla-inbound/rev/e1e990731d52
Apply same changes as win64-aarch64 nightlies to the shippable builds. r=tomprince

Comment 15

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/b7297e93ee98
https://hg.mozilla.org/mozilla-central/rev/4983a9d119c7
https://hg.mozilla.org/mozilla-central/rev/e1e990731d52

Comment 16

[Chris Pearce [:cpearce (Not reading bugmail)]]

My ARM64 laptop has died, so I can't test anything until I get a new one, but ajones tells me that EME is working on his ARM64 laptop.

Comment 17

[Mike Hommey [:glandium]]

Comment on attachment 9051219 [details]
Bug 1527463 - Get full artifact build symbols for aarch64 EME builds.

Beta/Release Uplift Approval Request

• Feature/Bug causing the regression: None
• User impact if declined: Not possible to play videos on e.g. Netflix on arm64 Windows
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Open Netflix and watch videos on an arm64 Windows machine. QE may not have some, I don't know, but several engineers and managers have. Asa Dotzler checked nightlies in bug 1515208.
• List of other uplifts needed: Bug 1537707
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): I'd normally say risk is low, but you never know what sort of differences between central and beta can trigger interesting challenges on automation. Technically speaking, this adds a new task that builds the same as current "nightly" builds (what is presumably released as beta), and changes the "nightly" build to build an EME-enabled build. This is an all-or-nothing sort of deal, in that, if things go through the full release pipeline with no error, then everything is fine, and if not, then things are busted. In that sense, there is some risk that this might prevent from releasing the beta following the landing for arm64 windows (but wouldn't affect other platforms).
• String changes made/needed: N/A
• Do you want to request approval of these patches as well?: on

Comment 18

[Mike Hommey [:glandium]]

Comment on attachment 9051220 [details]
Bug 1527463 - Enable EME on win64-aarch64 nightlies.

Beta/Release Uplift Approval Request

• Feature/Bug causing the regression: None
• User impact if declined: See comment 17. I'm confused by the "Do you want to request approval of these patches as well?" thing, because I checked this one, and it didn't appear to do anything. The third patch must not be uplifted.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):
• String changes made/needed:
• Do you want to request approval of these patches as well?: on

Comment 19

[Pascal Chevrel:pascalc (PTO until April 26)]

Comment on attachment 9051219 [details]
Bug 1527463 - Get full artifact build symbols for aarch64 EME builds.

Uplift approved for 67 beta 6, thanks.

Comment 20

[Pascal Chevrel:pascalc (PTO until April 26)]

Comment on attachment 9051220 [details]
Bug 1527463 - Enable EME on win64-aarch64 nightlies.

ARM64 Uplift approved for 67 beta 6, thanks.

Comment 21

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/releases/mozilla-beta/rev/7c3c3ffea0a4
https://hg.mozilla.org/releases/mozilla-beta/rev/216a721a0b8c

Comment 22

[Timea Cernea [:tbabos][inactive]]

Verified the fix on latest Nightly (2019-03-27)(64-bit) and Beta build 67.0b6 on aarch64-windows build on a Lenovo Yoga laptop with Snapdragon processor.
Sites that use Widewine such as Netflix, Prime video and test page https://shaka-player-demo.appspot.com/demo/#asset=https://storage.googleapis.com/shaka-demo-assets/angel-one/dash.mpd;lang=en-US;build=uncompiled now work as expected.

Comment 23

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

I'm looking at bug 1562884 which is about drm not working in dev edition on aarch64-win. For those familiar with these changes, could any of the changes have overlooked the dev edition, resulting in it not getting the Widevine/drm changes? It's not clear what the exact issue is yet, but I see my dev edition install have widevine related prefs toggled off by default (as if MOZ_WIDEVINE_EME was not defined), as well as clearkey not being installed.

Comment 24

[Mike Hommey [:glandium]]

dev edition builds weren't changed accordingly. One more reason for https://bugzilla.mozilla.org/show_bug.cgi?id=1547395#c1

Comment 25

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

(In reply to Mike Hommey [:glandium] from comment #24)

dev edition builds weren't changed accordingly. One more reason for https://bugzilla.mozilla.org/show_bug.cgi?id=1547395#c1

Thanks for confirming that. Could you advise what you think would be the best way to address this? Would having the dev ed derive from shippables be something we could do in the short term?

Comment 26

[Mike Hommey [:glandium]]

I'll defer to Tom on that.

Comment 27

[Tom Prince [:tomprince]]

(In reply to Bryce Seager van Dyk (:bryce) from comment #25)

Would having the dev ed derive from shippables be something we could do in the short term?

I'm not familiar enough with branding or the build process to do this work, or even scope it out. It looks (at a quick glance) like
the differences are:

• branding
• MOZ_REQUIRE_SIGNING
• MOZ_ALLOW_LEGACY_EXTENSIONS
  Grepping suggests that the later two configs are only looked at in js code (they end up in AppConstants.jsm, so in theory we could generate devedition builds by repacking the non-deved builds). However, all this is novel work, so I don't think it is a good short term solution.

Probably the best short-term solution is to duplicate the work of this bug for devedition.