1932482 - Support alternative URLs for downloading GMP addons

Status: RESOLVED FIXED

(Core :: Audio/Video: GMP, enhancement, P3)

Description

[Andrew Osmond [:aosmond] (he/him)]

We should teach the GMP installation code to try multiple URLs if provided to allow for alternatives for users who cannot access the preferred URL.

Comment 1

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1932482 - Add support for downloading GMP plugins from mirror URLs.

We have found that some users are unable to download the GMP plugin from
the default URL because of their specific network
configuration/circumstances. For example, some users cannot access the
*.gvt1.com domain. This patch adds the ability for balrog to supply
alternate URLs to attempt to download the plugin, where we first try the
primary, and if supplied any secondaries, in the given order.

Comment 2

[Andrew Osmond [:aosmond] (he/him)]

Updated the tracking flags to signal my intent to release management on future uplift requests. With regards to the 133.0.1 dot release, the stars may not align for us to land this, and test it, on nightly before the deadline but I consider that something we should aspire to.

Comment 3

[BMO Github Automation]

Attachment: [mozilla-releng/balrog] Bug 1932482 - Update domain allowlist for Widevine (#3252)

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fe2ff0beaea7
Add support for downloading GMP plugins from mirror URLs. r=media-playback-reviewers,padenot,robwu

Comment 5

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/fe2ff0beaea7

Comment 6

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1932482 - Add support for downloading GMP plugins from mirror URLs.

We have found that some users are unable to download the GMP plugin from
the default URL because of their specific network
configuration/circumstances. For example, some users cannot access the
*.gvt1.com domain. This patch adds the ability for balrog to supply
alternative mirror URLs to attempt to download the plugin, where we first
try the primary, and if supplied any secondaries, in the given order.

Original Revision: https://phabricator.services.mozilla.com/D229690

Comment 7

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: Users who are unable to download the Widevine plugin via the current URLs will not be able to take advantage of mirror URLs supplied by balrog
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: N/A
• Risk associated with taking this patch: Low
• Explanation of risk level: Very good code coverage via xpcshell tests, verified manually with and without new balrog configuration
• String changes made/needed: N/A
• Is Android affected?: no

Comment 8

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1932482 - Add support for downloading GMP plugins from mirror URLs.

We have found that some users are unable to download the GMP plugin from
the default URL because of their specific network
configuration/circumstances. For example, some users cannot access the
*.gvt1.com domain. This patch adds the ability for balrog to supply
alternative mirror URLs to attempt to download the plugin, where we first
try the primary, and if supplied any secondaries, in the given order.

Original Revision: https://phabricator.services.mozilla.com/D229690

Comment 9

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1932482 - Add support for downloading GMP plugins from mirror URLs.

We have found that some users are unable to download the GMP plugin from
the default URL because of their specific network
configuration/circumstances. For example, some users cannot access the
*.gvt1.com domain. This patch adds the ability for balrog to supply
alternative mirror URLs to attempt to download the plugin, where we first
try the primary, and if supplied any secondaries, in the given order.

Original Revision: https://phabricator.services.mozilla.com/D229690

Comment 10

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1932482 - Add support for downloading GMP plugins from mirror URLs.

We have found that some users are unable to download the GMP plugin from
the default URL because of their specific network
configuration/circumstances. For example, some users cannot access the
*.gvt1.com domain. This patch adds the ability for balrog to supply
alternative mirror URLs to attempt to download the plugin, where we first
try the primary, and if supplied any secondaries, in the given order.

Original Revision: https://phabricator.services.mozilla.com/D229690

Comment 11

[Phabricator Automation]

release Uplift Approval Request

• User impact if declined: Users who are unable to download the Widevine plugin via the current URLs will not be able to take advantage of mirror URLs supplied by balrog
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: Do typical Widevine installation manual testing. Should work as usual.
• Risk associated with taking this patch: Low
• Explanation of risk level: Very good code coverage via xpcshell tests, verified manually with and without new balrog configuration
• String changes made/needed: N/A
• Is Android affected?: no

Comment 12

[Phabricator Automation]

esr128 Uplift Approval Request

• User impact if declined: Users who are unable to download the Widevine plugin via the current URLs will not be able to take advantage of mirror URLs supplied by balrog
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: Do typical Widevine installation manual testing. Should work as usual.
• Risk associated with taking this patch: Low
• Explanation of risk level: Very good code coverage via xpcshell tests, verified manually with and without new balrog configuration
• String changes made/needed: N/A
• Is Android affected?: no

Comment 13

[Phabricator Automation]

esr115 Uplift Approval Request

• User impact if declined: Users who are unable to download the Widevine plugin via the current URLs will not be able to take advantage of mirror URLs supplied by balrog
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: Do typical Widevine installation manual testing. Should work as usual.
• Risk associated with taking this patch: Low
• Explanation of risk level: Very good code coverage via xpcshell tests, verified manually with and without new balrog configuration
• String changes made/needed: N/A
• Is Android affected?: no

Comment 14

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/bd2314c7a52b

Comment 15

[Timea Zsoldos [:tzsoldos], Desktop QA]

Tested on latest nightly Firefox build from 2024.11.28 and on Firefox 134.0b2, with Windows 10, macOS 10.15 and with Ubuntu 22, accessing edgedl.me.gvt1.com in the URL is redirected to https://www.google.com/chrome/ address, then accessing netflix.com to install Widevine plugin, works as usual.
I hope I tested correctly, if not please guide me with the correct steps and I will resume the testing.

Comment 16

[Andrew Osmond [:aosmond] (he/him)]

(In reply to Timea Zsoldos [:zstimi/tzsoldos], Desktop QA from comment #15)

Tested on latest nightly Firefox build from 2024.11.28 and on Firefox 134.0b2, with Windows 10, macOS 10.15 and with Ubuntu 22, accessing edgedl.me.gvt1.com in the URL is redirected to https://www.google.com/chrome/ address, then accessing netflix.com to install Widevine plugin, works as usual.
I hope I tested correctly, if not please guide me with the correct steps and I will resume the testing.

Here was my test plan for verifying staging balrog/production balrog, for builds with and without my patch.

Success cases:

1. Run mozregression --launch 134.0b3 --pref media.gmp-manager.url.override:"https://stage.balrog.nonprod.cloudops.mozgcp.net/update/3/GMP/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml" -a https://profiler.firefox.com/
2. Start the profiler with the Networking configuration.
3. Enable DRM in the preferences, if not enabled by default.
4. Go to about:addons, click on the gear / Check For Updates and wait for the installation.
5. Stop profiling and ensure under the Parent Process / Network tab, that we downloaded the plugin from edgedl.me.gvt1.com.
6. Try the Shaka player demo to confirm the plugin works.
7. Go to your hosts file (/etc/hosts on Linux/OSX, C:\Windows\System32\Drivers\etc\hosts on Windows) and add two new lines 0.0.0.0 edgedl.me.gvt1.com and :: edgedl.me.gvt1.com.
8. Redo steps 1-6, but now ensure in step 5 we downloaded the plugin from www.google.com

Failure cases:

1. Go to your hosts file (/etc/hosts on Linux/OSX, C:\Windows\System32\Drivers\etc\hosts on Windows) and add four new lines 0.0.0.0 edgedl.me.gvt1.com, :: edgedl.me.gvt1.com, 0.0.0.0 www.google.com and :: www.google.com.
2. Run mozregression --launch 134.0b3 --pref media.gmp-manager.url.override:"https://stage.balrog.nonprod.cloudops.mozgcp.net/update/3/GMP/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml" -a https://profiler.firefox.com/
3. Start the profiler with the Networking configuration.
4. Enable DRM in the preferences, if not enabled by default.
5. Go to about:addons, click on the gear / Check For Updates and wait for the installation. Widevine should not get installed.
6. Stop profiling and ensure under the Parent Process / Network tab, that we tried to download the plugin from edgedl.me.gvt1.com and www.google.com and failed both times.
7. Confirm the Widevine failure prefs were set, media.gmp-widevinecdm.lastInstallFailReason and media.gmp-widevinecdm.lastInstallFailed.

You can do similar tests with the production balrog server (remove --pref media.gmp-manager.url.override:... from mozregression). It should be similar but you should never see us try www.google.com to download the CRX3 file containing the plugin. You can also do similar tests with builds without my patch to ensure that it ignores the new configuration supplied by the staging balrog server.

Comment 17

[Timea Zsoldos [:tzsoldos], Desktop QA]

Tested on Ubuntu 22, with latest nightly Firefox 135.0a1 and with Firefox 134.0b3, everything works as you describe in comment 16.
Note that at the failure cases the Widevine failure prefs which have been set are: media.gmp-widevinecdm.lastDownloadFailReason = Error: Failed downloading via ServiceRequest, status: 0, channelStatus: 2152398861, errorCode: 2, reason: error
media.gmp-widevinecdm.lastDownloadFailed = 1733226152
Continue with testing on Windows and macOS and report back here.

Comment 18

[Timea Zsoldos [:tzsoldos], Desktop QA]

Finished the testing on Windows 10 and macOS 10.15, verified on latest nightly Firefox 135.0a1 and Firefox 134.0b3 the results are the same as on Ubuntu, all good. I have to mention that at the failure cases the prefs which have been set are: media.gmp-widevinecdm.lastDownloadFailReason and media.gmp-widevinecdm.lastDownloadFailed.

Comment 19

[Donal Meehan [:dmeehan]]

Comment on attachment 9440632 [details]
Bug 1932482 - Add support for downloading GMP plugins from mirror URLs.

Rejecting release uplift request.

• This is not a new issue in Fx133 and the change involves balrog rules.
• The timing is just before the holidays. This should bake in beta with Fx134 and ride the train to release with Fx134

Comment 20

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr128/rev/1de76ef2e146

Comment 21

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/7b461b7d80ac

Comment 22

[Andrew Osmond [:aosmond] (he/him)]

I should note that we appear to have reduced users without 4.10.2830.0 on beta from ~1.8% of the total population to ~1.2% in the week and a half this has been deployed.

Comment 23

[Timea Zsoldos [:tzsoldos], Desktop QA]

I tried to test on Windows and Ubuntu using Firefox 128.6.0esr and Firefox 115.19.0esr builds, but without success it gives me Widevine version 4.10.2710.0, adding pref media.gmp-manager.url.override=https://stage.balrog.nonprod.cloudops.mozgcp.net/update/3/GMP/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml. Shaka video not works.
Please guide me with the valid link for media.gmp-manager.url.override pref, I guess this is not valid for esr builds.
Thanks!

Comment 24

[Timea Zsoldos [:tzsoldos], Desktop QA]

Tested on Windows 10, Ubuntu 22 and on macOS 10.15 using Firefox 128.6.0esr and Firefox 115.19.0esr builds, testing performed with the production balrog server (without media.gmp-manager.url.override pref). I never see www.google.com to download the CRX3 file containing the plugin, it was downloaded from edgedl.me.gvt1.com (Widevine plugin version 4.10.2830.0) then try the Shaka player demo to confirm the plugin works.
Please confirm if it's ok, or I still have to test other scenarios.

Comment 25

[Andrew Osmond [:aosmond] (he/him)]

(In reply to Timea Zsoldos [:zstimi/tzsoldos], Desktop QA from comment #24)

Tested on Windows 10, Ubuntu 22 and on macOS 10.15 using Firefox 128.6.0esr and Firefox 115.19.0esr builds, testing performed with the production balrog server (without media.gmp-manager.url.override pref). I never see www.google.com to download the CRX3 file containing the plugin, it was downloaded from edgedl.me.gvt1.com (Widevine plugin version 4.10.2830.0) then try the Shaka player demo to confirm the plugin works.
Please confirm if it's ok, or I still have to test other scenarios.

That is sufficient, thank you!

Comment 26

[Andrew Osmond [:aosmond] (he/him)]

(In reply to Timea Zsoldos [:zstimi/tzsoldos], Desktop QA from comment #23)

I tried to test on Windows and Ubuntu using Firefox 128.6.0esr and Firefox 115.19.0esr builds, but without success it gives me Widevine version 4.10.2710.0, adding pref media.gmp-manager.url.override=https://stage.balrog.nonprod.cloudops.mozgcp.net/update/3/GMP/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml. Shaka video not works.
Please guide me with the valid link for media.gmp-manager.url.override pref, I guess this is not valid for esr builds.
Thanks!

You can just use the default, no need to override anymore since the balrog changes have deployed to production.

Comment 27

[Andrew Osmond [:aosmond] (he/him)]

We haven't deployed the release configuration to release or esr channels yet. It should be fine, but out of an abundance of caution and respect for vacations over the holidays, we've decided to deploy the config changes to the remaining channels around January 6th, 2025.

Comment 28

[Timea Zsoldos [:tzsoldos], Desktop QA]

Based on comment 25, mark verified on esr channels.