Closed Bug 1238140 Opened 8 years ago Closed 8 years ago

Netflix tests should verify the GMP Plugin is installed/updated before playing videos.

Categories

(Testing Graveyard :: external-media-tests, defect)

Product:
Testing Graveyard
Component:
external-media-tests
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox48 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla48
Tracking Flags:
Tracking Status
firefox48 --- fixed

People

(Reporter: sydpolk, Assigned: sydpolk)

Details

Attachments

(1 file)

MozReview Request: Bug 1238140 - Always download the GMP plugin before playing Netflix - r?maja_zf, r?ksteuber
58 bytes, text/x-review-board-request
bytesized
: review+
impossibus
: review+
Details 
When the Adobe Plugin is out of date, the Netflix page for a video will present a page about installing Silverlight. When Firefox is launched by hand with the same profile, a ribbon informing the user that something will be installed and to please wait will be presented.

The tests should at the very least sleep for a couple of minutes. At the best they should try to figure out when this is happening and wait for the update to complete.
In email, I asked cpearce:

Is there a way with javascript or preferecnes to thell that the Adobe GMP is out of date?

And Chris answered:

You can reset the pref "media.gmp-eme-adobe.version", which will cause us to delete the GMP from disk if it is present, and then call navigator.requestMediaKeySystemAccess() and the promise that returns will only resolve when the latest GMP has been installed, or it will be rejected after 60 seconds if there's a download failure/timeout.

i.e.:

navigator.requestMediaKeySystemAccess('com.adobe.primetime', [{initDataType: 'cenc'}]).then(
  function (access) {
    // Latest CDM should be installed now.
  },
  function (ex) {
    // Network or other error. Check ex.message for details
  }));
Summary: Netflix tests should wait until the GMP Plugin is installed/updated before plauing videos → Netflix tests should verify the GMP Plugin is installed/updated before playing videos.
Comment on attachment 8727946 [details]
MozReview Request: Bug 1238140 - Always download the GMP plugin before playing Netflix - r?maja_zf, r?ksteuber

https://reviewboard.mozilla.org/r/38703/#review35607

I think that waiting or polling for plugin installation is necessary here. Otherwise, the test could fail just because it was scheduled ahead of plugin installation. Also, it would be nice if there were an error message like "Plugin was not properly installed" rather than just "Preference has the wrong value".
Attachment #8727946 - Flags: review?(ksteuber)
I will take a look at this tomorrow.
Assignee: nobody → spolk
Comment on attachment 8727946 [details]
MozReview Request: Bug 1238140 - Always download the GMP plugin before playing Netflix - r?maja_zf, r?ksteuber

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/38703/diff/1-2/
Attachment #8727946 - Flags: review?(ksteuber)
Comment on attachment 8727946 [details]
MozReview Request: Bug 1238140 - Always download the GMP plugin before playing Netflix - r?maja_zf, r?ksteuber

https://reviewboard.mozilla.org/r/38703/#review35729

I think that this looks good!
Attachment #8727946 - Flags: review?(ksteuber) → review+
Comment on attachment 8727946 [details]
MozReview Request: Bug 1238140 - Always download the GMP plugin before playing Netflix - r?maja_zf, r?ksteuber

https://reviewboard.mozilla.org/r/38703/#review35887

::: dom/media/test/external/external_media_tests/playback/test_eme_playback.py:23
(Diff revision 2)
>      def setUp(self):
>          super(TestEMEPlayback, self).setUp()
>          self.set_eme_prefs()
> +        self.reset_GMP_version()
>          assert(self.check_eme_prefs())

Note that setUp is called between each test method. You have the option of putting the GMP-reset call in `setUpClass`, for example.

::: dom/media/test/external/external_media_tests/playback/test_eme_playback.py:41
(Diff revision 2)
> +            self.marionette.set_script_timeout(60000)

This modifies the default timeout for all scripts. If that's not what you want then the `script_timeout` keyword arg in `execute_async_script` should be used instead.

::: dom/media/test/external/external_media_tests/playback/test_eme_playback.py:42
(Diff revision 2)
> +            result = self.marionette.execute_async_script(reset_adobe_gmp_script)

`ScriptTimeoutException` should be handled here as well.

::: dom/media/test/external/external_media_tests/playback/test_eme_playback.py:44
(Diff revision 2)
> +                raise BaseException('ERROR: Resetting Adobe GMP failed % s' % result)

Raise `Exception` or `VideoException` here. Note that `BaseException` is not caught by `except Exception`. https://docs.python.org/3/library/exceptions.html#exception-hierarchy

These changes can't be checked on try, right? In that case, could you post some results of your testing on pf-jenkins, say? Thanks.
Attachment #8727946 - Flags: review?(mjzffr)
https://reviewboard.mozilla.org/r/38703/#review35887

> `ScriptTimeoutException` should be handled here as well.

And what should I do exactly here? I am just going to re-raise...
Attachment #8727946 - Flags: review?(mjzffr)
Comment on attachment 8727946 [details]
MozReview Request: Bug 1238140 - Always download the GMP plugin before playing Netflix - r?maja_zf, r?ksteuber

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/38703/diff/2-3/
Running this on pf-jenkins is difficult. However, here is the log from my local run:

mozilla@mozilla-05290 /c/dev/mozilla/mozilla-central
$ ./mach external-media-tests --profile /c/testing/eme_profile --urls dom/media/test/external/external_media_tests/urls/netflix/default.ini dom/media/test/external/external_media_tests/playback/eme.ini
 0:00.00 LOG: MainThread INFO Using workspace for temporary data: "c:\dev\mozilla\mozilla-central"
 0:04.88 LOG: MainThread INFO Initial Profile Destination is "c:\users\mozilla\appdata\local\temp\tmp6xgy9w"
 0:04.88 LOG: MainThread INFO starting httpd
 0:04.88 LOG: MainThread INFO running httpd on http://127.0.0.1:2734/
 0:04.88 LOG: MainThread mozversion INFO application_buildid: 20160310141949
 0:04.88 LOG: MainThread mozversion INFO application_changeset: 2466c13bd1716e091e876582c0b8e517f6430120
 0:04.88 LOG: MainThread mozversion INFO application_display_name: Nightly
 0:04.88 LOG: MainThread mozversion INFO application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
 0:04.88 LOG: MainThread mozversion INFO application_name: Firefox
 0:04.88 LOG: MainThread mozversion INFO application_remotingname: firefox
 0:04.88 LOG: MainThread mozversion INFO application_vendor: Mozilla
 0:04.88 LOG: MainThread mozversion INFO application_version: 48.0a1
 0:04.88 LOG: MainThread mozversion INFO platform_buildid: 20160310141949
 0:04.88 LOG: MainThread mozversion INFO platform_changeset: 2466c13bd1716e091e876582c0b8e517f6430120
 0:04.88 LOG: MainThread mozversion INFO platform_version: 48.0a1
 0:04.88 SUITE_START: MainThread 1
 0:04.88 TEST_START: MainThread test_eme_playback.py TestEMEPlayback.test_playback_starts
 0:05.09 LOG: MainThread INFO Pref media.mediasource.enabled = True
 0:05.10 LOG: MainThread INFO Pref media.eme.enabled = True
 0:05.11 LOG: MainThread INFO Pref media.mediasource.mp4.enabled = True
 0:05.12 LOG: MainThread INFO Pref media.gmp-eme-adobe.enabled = True
 0:05.13 LOG: MainThread INFO Pref media.gmp-eme-adobe.version = 17
 0:13.38 LOG: MainThread INFO http://www.netflix.com/watch/70136810
 0:17.36 LOG: MainThread INFO http://www.netflix.com/watch/70304192
 0:23.07 LOG: MainThread INFO http://www.netflix.com/watch/80015538
 0:23.92 TEST_END: MainThread PASS
 0:23.92 TEST_START: MainThread test_eme_playback.py TestEMEPlayback.test_video_playback_partial
 0:24.11 LOG: MainThread INFO Pref media.mediasource.enabled = True
 0:24.12 LOG: MainThread INFO Pref media.eme.enabled = True
 0:24.13 LOG: MainThread INFO Pref media.mediasource.mp4.enabled = True
 0:24.14 LOG: MainThread INFO Pref media.gmp-eme-adobe.enabled = True
 0:24.15 LOG: MainThread INFO Pref media.gmp-eme-adobe.version = 17
 0:30.34 LOG: MainThread INFO http://www.netflix.com/watch/70136810
 1:33.99 LOG: MainThread INFO http://www.netflix.com/watch/70304192
 2:40.29 LOG: MainThread INFO http://www.netflix.com/watch/80015538
 3:40.27 TEST_END: MainThread PASS
 3:40.41 LOG: MainThread INFO START LOG:
 3:40.41 LOG: MainThread INFO INFO TEST-START: c:\dev\mozilla\mozilla-central\dom\media\test\external\external_media_tests\playback\test_eme_playback.py:test_playback_starts Thu Mar 10 2016 16:19:22 GMT-0600 (Central Standard Time)
 3:40.41 LOG: MainThread INFO INFO URL: http://www.netflix.com/watch/70136810 Thu Mar 10 2016 16:19:26 GMT-0600 (Central Standard Time)
 3:40.41 LOG: MainThread INFO INFO video element obtained Thu Mar 10 2016 16:19:28 GMT-0600 (Central Standard Time)
 3:40.41 LOG: MainThread INFO INFO URL: http://www.netflix.com/watch/70304192 Thu Mar 10 2016 16:19:32 GMT-0600 (Central Standard Time)
 3:40.41 LOG: MainThread INFO INFO video element obtained Thu Mar 10 2016 16:19:34 GMT-0600 (Central Standard Time)
 3:40.41 LOG: MainThread INFO INFO URL: http://www.netflix.com/watch/80015538 Thu Mar 10 2016 16:19:37 GMT-0600 (Central Standard Time)
 3:40.41 LOG: MainThread INFO INFO video element obtained Thu Mar 10 2016 16:19:39 GMT-0600 (Central Standard Time)
 3:40.41 LOG: MainThread INFO INFO TEST-END: c:\dev\mozilla\mozilla-central\dom\media\test\external\external_media_tests\playback\test_eme_playback.py:test_playback_starts Thu Mar 10 2016 16:19:41 GMT-0600 (Central Standard Time)
 3:40.41 LOG: MainThread INFO END LOG:
 3:40.41 LOG: MainThread INFO START LOG:
 3:40.41 LOG: MainThread INFO INFO TEST-START: c:\dev\mozilla\mozilla-central\dom\media\test\external\external_media_tests\playback\test_eme_playback.py:test_video_playback_partial Thu Mar 10 2016 16:19:41 GMT-0600 (Central Standard Time)
 3:40.41 LOG: MainThread INFO INFO URL: http://www.netflix.com/watch/70136810 Thu Mar 10 2016 16:19:43 GMT-0600 (Central Standard Time)
 3:40.41 LOG: MainThread INFO INFO video element obtained Thu Mar 10 2016 16:19:46 GMT-0600 (Central Standard Time)
 3:40.42 LOG: MainThread INFO INFO URL: http://www.netflix.com/watch/70304192 Thu Mar 10 2016 16:20:49 GMT-0600 (Central Standard Time)
 3:40.42 LOG: MainThread INFO INFO video element obtained Thu Mar 10 2016 16:20:50 GMT-0600 (Central Standard Time)
 3:40.42 LOG: MainThread INFO INFO URL: http://www.netflix.com/watch/80015538 Thu Mar 10 2016 16:21:54 GMT-0600 (Central Standard Time)
 3:40.42 LOG: MainThread INFO INFO video element obtained Thu Mar 10 2016 16:21:56 GMT-0600 (Central Standard Time)
 3:40.42 LOG: MainThread INFO INFO TEST-END: c:\dev\mozilla\mozilla-central\dom\media\test\external\external_media_tests\playback\test_eme_playback.py:test_video_playback_partial Thu Mar 10 2016 16:22:57 GMT-0600 (Central Standard Time)
 3:40.42 LOG: MainThread INFO END LOG:
 3:40.42 LOG: MainThread INFO
SUMMARY
-------
 3:40.42 LOG: MainThread INFO passed: 2
 3:40.42 LOG: MainThread INFO failed: 0
 3:40.42 LOG: MainThread INFO todo: 0
 3:41.05 SUITE_END: MainThread
Summary
=======

Ran 2 tests
Expected results: 2
Unexpected results: 0

OK

mozilla@mozilla-05290 /c/dev/mozilla/mozilla-central
$
Comment on attachment 8727946 [details]
MozReview Request: Bug 1238140 - Always download the GMP plugin before playing Netflix - r?maja_zf, r?ksteuber

https://reviewboard.mozilla.org/r/38703/#review36145

::: dom/media/test/external/external_media_tests/playback/test_eme_playback.py:42
(Diff revision 2)
> +            result = self.marionette.execute_async_script(reset_adobe_gmp_script)

Yeah, you're right. I could have worded that as a question -- just trying to bring options to your attention. It makes sense that you just want the test to fail at that point anyway; no point is re-raising as a different exception.
Attachment #8727946 - Flags: review?(mjzffr)
Comment on attachment 8727946 [details]
MozReview Request: Bug 1238140 - Always download the GMP plugin before playing Netflix - r?maja_zf, r?ksteuber

https://reviewboard.mozilla.org/r/38703/#review36147

::: dom/media/test/external/external_media_tests/playback/test_eme_playback.py:51
(Diff revision 3)
> +                    raise VideoException('ERROR: Resetting Adobe GMP failed % s' % ex)

As I said earlier, feel free to remove this.
Attachment #8727946 - Flags: review+
Comment on attachment 8727946 [details]
MozReview Request: Bug 1238140 - Always download the GMP plugin before playing Netflix - r?maja_zf, r?ksteuber

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/38703/diff/3-4/
https://hg.mozilla.org/mozilla-central/rev/302353ddf84f
Status: NEW → RESOLVED
Closed: 8 years ago
status-firefox48: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: