1095080 - gaiatest endurance test test_endurance_browser_wifi.py failing

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Stability, defect, P1)

Description

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

We are seeing that browser app is failing to URL if we run following gaiatest



gaiatest --type=b2g --address=localhost:$MARIONETTE_PORT --testvars=b2gtestvars.json --iterations=3000 --checkpoint=1 gaia/tests/python/gaia-ui-tests/gaiatest/tests/endurance/test_endurance_browser_wifi.py



You can find exact python test code in :

https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gaia/tree/tests/python/gaia-ui-tests/gaiatest/tests/endurance/test_endurance_browser_wifi.py?h=mozilla/v2.1

It seems to be marionette failure as we are able to open browser app on device when marionette fails.

Issue can be reproducible only v2.1 branch.

Comment 1

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Hi Bhavna,

We knew that you have some MTBF scripts and it seems that browser app changed a lot (see bug 1088886 comment 8) in v2.1 .

Could you please let us know if you have any working browser test code in v2.1 which can do following things : 

1) Open browser app
2) Navigate to URL
3) Closing browser app .

Comment 2

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

[Blocking Requested - why for this release]:

Comment 3

[bhavana bajaj [:bajaj]]

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #1)
> Hi Bhavna,
> 
> We knew that you have some MTBF scripts and it seems that browser app
> changed a lot (see bug 1088886 comment 8) in v2.1 .
> 
> Could you please let us know if you have any working browser test code in
> v2.1 which can do following things : 
> 
> 1) Open browser app
> 2) Navigate to URL
> 3) Closing browser app .

NI :zac to comment on the gaia ui test piece.

Brian, can you share the in for on coverage of this test in our stability testing?

Comment 4

[Malini Das [:mdas] - Away, not checking bugmail]

rwood may be of some help here

Comment 5

[Robert Wood [:rwood]]

Hi Tapas,

Yes the browser endurance tests are out of date, and need to be updated to use the search app instead of the browser app. I will assign this bug to myself, for the updating of the endurance test accordingly.

In the mean time *possibly* the following code might work on 2.1, however :zac would need to verify as he is familiar with the status of the functional tests.

https://github.com/mozilla-b2g/gaia/blob/v2.1/tests/python/gaia-ui-tests/gaiatest/tests/functional/browser/test_browser_lan.py

Comment 6

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

please also look into bug 1095154

Comment 7

[Brian Huang [:brianhuang][:brhuang]]

(In reply to bhavana bajaj [:bajaj] from comment #3)
> (In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from
> comment #1)
> > Hi Bhavna,
> > 
> > We knew that you have some MTBF scripts and it seems that browser app
> > changed a lot (see bug 1088886 comment 8) in v2.1 .
> > 
> > Could you please let us know if you have any working browser test code in
> > v2.1 which can do following things : 
> > 
> > 1) Open browser app
> > 2) Navigate to URL
> > 3) Closing browser app .
> 
> NI :zac to comment on the gaia ui test piece.
> 
> Brian, can you share the in for on coverage of this test in our stability
> testing?

Hi Bhavana, Tapas,

Here are some information that about browser, 

Browser - 1. browse a website 2. add a bookmark 3. browse from bookmark 

I ni? Walter to have the detail explanation for test cases.

Comment 8

[Walter Chen[:ypwalter][:wachen]]

We have navigation, search, and browsing test cases. However, we temporarily stopped those testing because we met way too many issues in past month. We will gradually get those tests back.

Comment 9

[Zac C (:zac)]

We don't run this test so we probably won't investigate/fix it, but if you file a pull request for it we'll review and merge it. (Please file the pull request against master and we'll uplift it).

Comment 10

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

(In reply to Walter Chen[:ypwalter][:wachen] from comment #8)
> We have navigation, search, and browsing test cases. However, we temporarily
> stopped those testing because we met way too many issues in past month. We
> will gradually get those tests back.

Could you please point me to your test scripts :) thanks a lot .

Comment 11

[Andrew Overholt [:overholt]]

The triage group talked about this and decided that given comment 5 and comment 6 we can't block the release on this bug.

Comment 12

[James Lal [:lightsofapollo]]

to answer Tapas I found these tests from MTBF https://github.com/Mozilla-TWQA/MTBF-Driver/blob/master/mtbf_driver/tests/mtbf/browser/test_mtbf_browser_navigation.py (see that entire browser folder)... I am pretty sure there is nothing else that needs to be done here regarding the endurance test suite.

If there is something preventing these tests from being run let me know we can help.

Comment 13

[Robert Wood [:rwood]]

Attachment: https://github.com/mozilla-b2g/gaia/pull/26416

Comment 14

[Zac C (:zac)]

Comment on attachment 8527756 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/26416

r+

Comment 15

[Robert Wood [:rwood]]

Thanks Zac. Squashed and merged.
https://github.com/mozilla-b2g/gaia/commit/9ec3033b2f83983f778c85143182d51b35387eee