1042843 - How to verify whether video playback is going on from video playback gaiatest script?

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[vasanth]

I see this way [1] of checking whether "video playback is going on" doesn't work reliably. We are using same script and tried alternative ways to do the same. Nothing helps. Raising this bug to track that issue.

Not sure if there is already a bug for this?

[1] https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/endurance/test_endurance_video_playback.py#L44

Comment 1

[Michael Schwartz [:m4]]

Adding the author for ni to comment.

Comment 2

[Zac C (:zac)]

Nobody is working on this.

In the functional tests we check the timestamp from the video element. You can copy over this code if you find it satisfactory.

Why don't you submit a pull request and we'll review it?

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Michael Schwartz [:m4] from comment #1)
> Adding the author for ni to comment.

I'm not the original author of this test, I believe that would be Rob Wood [:rwood].

Comment 4

[vasanth]

(In reply to Zac C (:zac) from comment #2)
> In the functional tests we check the timestamp from the video element. You
> can copy over this code if you find it satisfactory.

Yes. That works well for 1 iteration.
For more iterations, during some iterations, controls doesn't become visible and timestamp check fails.

Adding ni on author to find whether he has additional info.

Comment 5

[Robert Wood [:rwood]]

Hi Vasanth, the test_endurance_video_playback test is currently disabled in the endurance suite manifest (https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/endurance/manifest.ini) as it needs to be updated. Sounds like a gaia bug to me though, if the controls don't become visible during some iterations, unless it is a script timing issue.

Comment 6

[Zac C (:zac)]

This would be a good task for a contributor to take as it can be run on desktopb2g.

Comment 7

[Martijn Wargers (dead)]

Attachment: endurance_video

I had to change quite a bit to get this endurance test working again. Basically using almost everything that is used in current gaia-ui video tests.

According to: https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Endurance
"
Please note: The Gaia-UI endurance tests are currently not being run by Mozilla and are receiving only critical maintenance updates. While you may find them of use, please understand you may have compatibility issues and need to update tests to match the Gaia tip.
"

I guess that's the reason why so many are disabled:
http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/endurance/manifest.ini

Comment 8

[Robert Wood [:rwood]]

Comment on attachment 8541919 [details] [review]
endurance_video

As noted in comment 7, the gaia-ui endurance tests are no longer running. R- for that reason; no point in updating the gaiatest GaiaEnduranceTestCase class to fix this when the endurance tests are now obsolete. I would suggest you update the corresponding (or create a new) functional test instead.

Comment 9

[Martijn Wargers (dead)]

Shouldn't we remove the gaia-ui endurance tests then?
I guess the endurance tests are made obsolete by the mtbf tests?

Comment 10

[Robert Wood [:rwood]]

My understanding is that some of our b2g partners are using the GaiaEnduranceTestCase class, so that functionality should remain in gaiatest. We shouldn't remove endurance test functionality from the gaiatest python package in case other gaiatest consumers have their own endurance test implementations.

As for the endurance tests themselves, I have no problem with them being removed from gaia, although someone else who has partner knowledge should probably make that decision (?) to ensure that our b2g partners are not dependent upon the tests themselves.

More info on why the endurance tests are not currently running/supported:

In my opinion, to have real value, the endurance tests would need to be overhauled drastically. i.e. Besides updating to work with the latest gaia changes, the test framework should use the get_about_memory tool to grab memory info for each app (instead of just b2g-ps and using the RSS value for the main b2g process), re-evaluate the usefulness of each test, force garbage collection first, etc. So this would be potentially be a large project. Also, there may be other tests already running that provide more value, even if the existing gaia-ui endurance tests were overhauled (i.e. the *_memory tests that are posting to DataZilla).

Note that most of the endurance tests were developed from existing functional gaia-ui tests, which are still running. The MTBF tests may cover some of the same gaia functionality also.

Comment 11

[Martijn Wargers (dead)]

Thanks for the explanation.
Yes, most of the endurance tests look like they were made of existing gaia-ui tests. I get the impression that most of the endurance tests already don't work anymore, they seem out of date.

(In reply to Robert Wood [:rwood] from comment #10)
> project. Also, there may be other tests already running that provide more
> value, even if the existing gaia-ui endurance tests were overhauled (i.e.
> the *_memory tests that are posting to DataZilla).

Where can those tests be found, if I may ask? Are those one of the repos from https://github.com/mozilla-b2g ?

Comment 12

[Robert Wood [:rwood]]

(In reply to Martijn Wargers [:mwargers] (QA) from comment #11)
> Thanks for the explanation.
> Yes, most of the endurance tests look like they were made of existing
> gaia-ui tests. I get the impression that most of the endurance tests already
> don't work anymore, they seem out of date.
> 
> (In reply to Robert Wood [:rwood] from comment #10)
> > project. Also, there may be other tests already running that provide more
> > value, even if the existing gaia-ui endurance tests were overhauled (i.e.
> > the *_memory tests that are posting to DataZilla).
> 
> Where can those tests be found, if I may ask? Are those one of the repos
> from https://github.com/mozilla-b2g ?

I am guessing they are Gaia Performance Tests [1] and located with the gaia apps source, but I don't know for certain. Dave, do you know the answer?

[1] https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Gaia_performance_tests

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Robert Wood [:rwood] from comment #12)
> (In reply to Martijn Wargers [:mwargers] (QA) from comment #11)
> > Thanks for the explanation.
> > Yes, most of the endurance tests look like they were made of existing
> > gaia-ui tests. I get the impression that most of the endurance tests already
> > don't work anymore, they seem out of date.
> > 
> > (In reply to Robert Wood [:rwood] from comment #10)
> > > project. Also, there may be other tests already running that provide more
> > > value, even if the existing gaia-ui endurance tests were overhauled (i.e.
> > > the *_memory tests that are posting to DataZilla).
> > 
> > Where can those tests be found, if I may ask? Are those one of the repos
> > from https://github.com/mozilla-b2g ?
> 
> I am guessing they are Gaia Performance Tests [1] and located with the gaia
> apps source, but I don't know for certain. Dave, do you know the answer?
> 
> [1]
> https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/
> Gaia_performance_tests

That's correct. If you'd like more information on the Gaia performance tests I would suggest speaking to Eli Perelman, who's actively working on them.

Comment 14

[Martijn Wargers (dead)]

I filed bug 1117144 on removing the endurance tests.

Comment 15

[BMO Automation]

Firefox OS is not being worked on