1912274 - Update rules to serve Widevine 4.10.2830.0

Status: RESOLVED FIXED

(Release Engineering :: General, task)

Description

[Andrew Osmond [:aosmond] (he/him)]

For the convenience of QA to do testing, let's setup the nightlytest channel with the latest Widevine update.

Comment 1

[Andrew Osmond [:aosmond] (he/him)]

Attachment: WidevineCDM 4.10.2830.0 JSON

Attached is the output of the script updated in bug 1908139 to support the Chrome component service using the command: python3 dom/media/tools/generateGmpJson.py --testrequest widevine_component

Note that this will only work with nightlies that contain bug 1908144. That means build 20240808093537 and later. Ideally we would only push this rule to newer nightlies than that.

Comment 2

[Geoff Brown [:gbrown]]

I notice this new json contains urls like https://edgedl.me.gvt1.com/edgedl/...; the 4.10.2710.0 release has urls like https://redirector.gvt1.com/edgedl/.... Is that change intentional?

Comment 3

[Geoff Brown [:gbrown]]

We use a domain allowlist; a small update + balrog deployment would be needed to proceed with the new urls.

Comment 4

[Andrew Osmond [:aosmond] (he/him)]

(In reply to Geoff Brown [:gbrown] (last day Aug 23) from comment #3)

We use a domain allowlist; a small update + balrog deployment would be needed to proceed with the new urls.

It is intentional. Google requested we use that domain as part of the process of generating these URLs.

Comment 5

[BMO Github Automation]

Attachment: [mozilla-releng/balrog] Bug 1912274 - Update domain allowlist for Widevine (#3130)

Comment 6

[Geoff Brown [:gbrown]]

We need the balrog update (comment 5) deployed before proceeding. Because of mozweek, that deployment is on pause for the moment.

If the nightlytest update is urgent or blocking anyone, please let me know.

Comment 7

[Andrew Osmond [:aosmond] (he/him)]

Attachment: WidevineCDM 4.10.2830.0 JSON -- initial nightlytest only

Comment 8

[Andrew Osmond [:aosmond] (he/him)]

Attachment: WidevineCDM 4.10.2830.0 JSON -- initial nightlytest only

Comment 9

[Andrew Osmond [:aosmond] (he/him)]

To workaround the permitted domain list, for now let's just use WidevineCDM 4.10.2830.0 JSON -- initial nightlytest only instead.

Comment 10

[Geoff Brown [:gbrown]]

Release created with blob from comment 8; now available on nightlytest only.

Comment 11

[Geoff Brown [:gbrown]]

The balrog update (comment 3, etc) with updated domain allowlist has been deployed: We can use the new edgedl.me.gvt1.com urls going forward.

Comment 12

[Geoff Brown [:gbrown]]

:jlorenzo - This bug will need releng attention once Andrew is ready to move forward.

Comment 13

[Andrew Osmond [:aosmond] (he/him)]

Can we now please switch nightlytest to the original rules from attachment 9418329 [details]?

Comment 14

[Julien Cristau [:jcristau]]

Done. I've also updated the rule to only apply to buildid >= 20240808093537, per comment 1.

Comment 15

[Andrew Osmond [:aosmond] (he/him)]

Could we serve attachment 9418329 [details] to nightly >= 20240814213714, and beta >= 130.0b6? Thanks!

Comment 16

[Julien Cristau [:jcristau]]

Updated rules for nightly*, aurora* and beta*.

Comment 17

[Andrew Osmond [:aosmond] (he/him)]

Could we please serve attachment 9418329 [details] to release >= 129.0.2? It is very important not to serve this to earlier releases because they cannot handle the unpacking and sandbox requirements without patches we uplifted. Thanks!

There will be a follow up on special rules for OSX < 129.0.2 after we finish our rollout, since we did not have to update the sandbox.

Comment 18

[Julien Cristau [:jcristau]]

Widevine-4.10.2830.0 is now served to all channels for buildids >= 20240819150008, except for esr which is still at 4.10.2710.0.

Comment 19

[jilvinjacob]

I was having issues with 4.10.2830.0, had to go back to 4.10.2710.0.

See: https://bugzilla.mozilla.org/show_bug.cgi?id=1830431
https://crash-stats.mozilla.org/report/index/ea5c9426-be74-4200-a869-b183e0240903

Might not be an issue that's happening for everyone though. Can't confirm.

Comment 20

[Andrew Osmond [:aosmond] (he/him)]

(In reply to jilvinjacob from comment #19)

I was having issues with 4.10.2830.0, had to go back to 4.10.2710.0.

See: https://bugzilla.mozilla.org/show_bug.cgi?id=1830431
https://crash-stats.mozilla.org/report/index/ea5c9426-be74-4200-a869-b183e0240903

Might not be an issue that's happening for everyone though. Can't confirm.

I will see if I can add additional diagnostic information to the crash reports. Unfortunately the old plugin will only work for another month or so (outside our control).

Comment 21

[Andrew Osmond [:aosmond] (he/him)]

Could we please serve attachment 9418329 [details] to ESR128 >= 128.2.0? Again, it is very important not to serve this to earlier releases because they cannot handle the unpacking and sandbox requirements without patches we uplifted. Thanks!

Could we also please serve it to ESR115 >= 115.15.0 for Windows 10+ users? We cannot ship to Windows 7 users at this time, and to be safe, we would like to limit Windows 8 and 8.1 as well. Thanks!

Comment 22

[Andrew Osmond [:aosmond] (he/him)]

Attachment: WidevineCDM 4.10.2830.0 JSON -- Older Firefox for OSX only

Comment 23

[Andrew Osmond [:aosmond] (he/him)]

Could we please serve attachment 9424492 [details] to OSX users for Firefox users < ESR115.15.0, < ESR128.2.0, < 129.0.4 -- anyone on OSX not covered by the rules requested in comment 15, comment 17, and comment 21. Thanks! I realize this might be a more complicated request, so do not hesitate to reach out to me on slack or element about it.

Comment 24

[Julien Cristau [:jcristau]]

These changes are live now.

Comment 25

[Andrew Osmond [:aosmond] (he/him)]

Now that we have verified the necessary changes for Windows 7 and uplifted, could we please serve attachment 9418329 [details] to ESR115 >= 115.15.0 for Windows 8+ users, and ESR115 >= 115.16.0 for Windows 7 users? Once this is done, we can close the bug. Thanks!

Comment 26

[Julien Cristau [:jcristau]]

Attachment: widevine esr update rules

I believe we're done here.

Comment 27

[Andrew Osmond [:aosmond] (he/him)]

We seem to be serving the wrong URLs to older OSX builds. This is causing a spike in crashes on Firefox < 129.0.2 on OSX. It should be using the URLs from attachment 9424492 [details]. Specifically I would have expected to see one of the following:
https://edgedl.me.gvt1.com/edgedl/widevine-cdm/4.10.2830.0-mac-arm64.zip
https://edgedl.me.gvt1.com/edgedl/widevine-cdm/4.10.2830.0-mac-x64.zip

Instead it seems to be using the new values intended for 129.0.2, e.g.:
https://edgedl.me.gvt1.com/edgedl/release2/chrome_component/dhuvyipbjqknanqk7f3zqnv4ku_4.10.2830.0/oimompecagnajdejgnnjijobebaeigek_4.10.2830.0_mac_arm64_lt3covpsgi3bctnfic74bafcxa.crx3"
https://edgedl.me.gvt1.com/edgedl/release2/chrome_component/huwtmpvskwwuqnqp6gacinr2pu_4.10.2830.0/oimompecagnajdejgnnjijobebaeigek_4.10.2830.0_mac64_ldaompc5meaw6wtyokr33tsf2u.crx3

This causes us to fail the extraction because the layouts are different, and the older Firefox releases aren't compatible (they needed bug 1908144).

Can we get this fixed ASAP?

Comment 28

[Julien Cristau [:jcristau]]

We've rolled back mac <129.0.2 / <115.15.0 / <128.2.0 to the previous version of the plugin.

Comment 29

[Julien Cristau [:jcristau]]

I've uploaded attachment 9424492 [details] to balrog as https://aus-api.mozilla.org/api/v1/releases/Widevine-4.10.2830.0-old-mac, so we should be ready to serve that when wanted (either across all channels or one at a time?).

Comment 30

[Andrew Osmond [:aosmond] (he/him)]

(In reply to Julien Cristau [:jcristau] from comment #29)

I've uploaded attachment 9424492 [details] to balrog as https://aus-api.mozilla.org/api/v1/releases/Widevine-4.10.2830.0-old-mac, so we should be ready to serve that when wanted (either across all channels or one at a time?).

Perfect, let's do it then. All channels. I will verify as soon as it is live.

Comment 31

[Julien Cristau [:jcristau]]

This is now live. Hopefully for good this time.