1030459 - gaiatest test_endurance_camera_photo.py broken in FFOS 2.0

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect, P1)

Description

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Attachment: failure logs

We are seeing following error for gaiatest failure : 

raise JavascriptException(message=message, status=status, stacktrace=stacktrace)
JavascriptException: JavascriptException: TypeError: apps is undefined
stacktrace:
	execute_async_script @gaia_test.py, line 44
	inline javascript, line 109
	src: "    let installedApps = apps.installedApps;"
TEST-UNEXPECTED-FAIL | test_endurance_camera_photo.py 


It seems like that this testcase is broken in FFOS 2.0. It is working fine v1.5

Comment 1

[Hema Koka [:hema]]

Justindarc or diego, please check this test failure.

Thanks
Hema

Comment 2

[Hema Koka [:hema]]

Blocking because of test failure (works fine in previous version)

Comment 3

[Diego Marcos [:dmarcos]]

We don't maintain the python integration tests and the message is not very descriptive. We now have integration tests using marionette js that run on device. 

https://github.com/mozilla-b2g/gaia/tree/master/apps/camera#integration-tests

The failing tests can be easily replicated in JS. I'm happy to help migrate it so we can help maintain it. We have no bandwidth to keep two identical test suites in sync.

Comment 4

[Jason Smith [:jsmith]]

(In reply to Diego Marcos [:dmarcos] from comment #3)
> We don't maintain the python integration tests and the message is not very
> descriptive. We now have integration tests using marionette js that run on
> device. 

If they do, they aren't running in a CI environment. The earliest plan to have MarionetteJS on device in a CI environment is planned in Q3 for only the most critical tests on the phone to be ran on a per commit basis. There's no plans to support a CI-based solution here for MarionetteJS for all tests that currently exist in the tree as of right now.

Comment 5

[Jason Smith [:jsmith]]

Anyways - the problem here is in Gaia UI Tests, not developer-driven integration tests, so I'm moving this over to UI Tests & adding needinfo on Rob to take a look.

Comment 6

[Zac C (:zac)]

The problem here is that the get_permission/set_permission need to be executed from the system frame.

The user is running without --restart flag (which the tests are designed to run with) which causes the test to execute set_permission from the homescreen frame.

Comment 7

[Zac C (:zac)]

As per comment #6

Comment 8

[Jason Smith [:jsmith]]

From Moz's perspective, we can't block on this. CAF is welcome to take advantage of the tests & framework written, but Moz is not liable for fixing issues on CAF's behalf. CAF needs to assign their own engineers to resolve the problem.

Comment 9

[Justin D'Arcangelo [:justindarc]]

(In reply to Jason Smith [:jsmith] from comment #4)
> If they do, they aren't running in a CI environment. The earliest plan to
> have MarionetteJS on device in a CI environment is planned in Q3 for only
> the most critical tests on the phone to be ran on a per commit basis.
> There's no plans to support a CI-based solution here for MarionetteJS for
> all tests that currently exist in the tree as of right now.

It would be nice to get a timeline in place to get MarionetteJS integration tests running in CI. The Python-based tests are cumbersome to work with because they rely on a library that is fetched from the PIP Python package manager repo. Any time the application changes, the Python library in the PIP repo also needs to be updated to reflect it and we are not aware of policy for how/when/who this gets updated.

Comment 10

[Zac C (:zac)]

That's not correct about gaiatest/pypi because you can run it straight from the tree, but in any case QA are also working to get the JS tests on device too.

Comment 11

[Justin D'Arcangelo [:justindarc]]

(In reply to Zac C (:zac) from comment #10)
> That's not correct about gaiatest/pypi because you can run it straight from
> the tree, but in any case QA are also working to get the JS tests on device
> too.

But, are our CI environments set up to pull gaiatest from the tree? I'm fairly certain at some point, CI was pulling it from PIP, but I might be wrong.

Comment 12

[Zac C (:zac)]

(In reply to Justin D'Arcangelo [:justindarc] from comment #11)

> 
> But, are our CI environments set up to pull gaiatest from the tree? I'm
> fairly certain at some point, CI was pulling it from PIP, but I might be
> wrong.

Yes they are using in-tree! 
but we package and push it to pypi so that some outside-tree test suites can take advantage of the test framework.

Comment 13

[Justin D'Arcangelo [:justindarc]]

(In reply to Zac C (:zac) from comment #12)
> Yes they are using in-tree! 
> but we package and push it to pypi so that some outside-tree test suites can
> take advantage of the test framework.

Ok, good to know! Us Camera devs are Python n00bs, so its also good news to know that QA is working on getting the JS integration tests working in CI :-)

Comment 14

[Jason Smith [:jsmith]]

(In reply to Justin D'Arcangelo [:justindarc] from comment #13)
> (In reply to Zac C (:zac) from comment #12)
> > Yes they are using in-tree! 
> > but we package and push it to pypi so that some outside-tree test suites can
> > take advantage of the test framework.
> 
> Ok, good to know! Us Camera devs are Python n00bs, so its also good news to
> know that QA is working on getting the JS integration tests working in CI :-)

FTW - you can see the definition of the goal around this here - https://wiki.mozilla.org/QA/Goals/2014q3#Firefox_OS

Comment 15

[Jason Smith [:jsmith]]

(In reply to Jason Smith [:jsmith] from comment #14)
> (In reply to Justin D'Arcangelo [:justindarc] from comment #13)
> > (In reply to Zac C (:zac) from comment #12)
> > > Yes they are using in-tree! 
> > > but we package and push it to pypi so that some outside-tree test suites can
> > > take advantage of the test framework.
> > 
> > Ok, good to know! Us Camera devs are Python n00bs, so its also good news to
> > know that QA is working on getting the JS integration tests working in CI :-)
> 
> FTW - you can see the definition of the goal around this here -
> https://wiki.mozilla.org/QA/Goals/2014q3#Firefox_OS

Meant to say FWIW, not FTW

Comment 16

[Jonathan Griffin (:jgriffin)]

Rob, can you take a look at this when you get back?

Comment 17

[bhavana bajaj [:bajaj]]

Indra, it will be good to know if things changed based on our offline discussion about CAF running the tests per recommendation in comment #6.

In parallel, we have someone from our A-team investigating this as well.

Comment 18

[Zac C (:zac)]

(In reply to Jonathan Griffin (:jgriffin) from comment #16)
> Rob, can you take a look at this when you get back?

We fixed this on master inside the framework, no need for Rob to investigate.

Comment 19

[bhavana bajaj [:bajaj]]

(In reply to Zac C (:zac) from comment #18)
> (In reply to Jonathan Griffin (:jgriffin) from comment #16)
> > Rob, can you take a look at this when you get back?
> 
> We fixed this on master inside the framework, no need for Rob to investigate.

Will that also be patched on the gaia 2.0v branch which is where CAF pulls from and is hitting this issue ?

Comment 20

[Inder]

> 
> We fixed this on master inside the framework, no need for Rob to investigate.
:zac -- Can you please also point us to the bug with the fix.

Comment 21

[Zac C (:zac)]

The patch was unrelated to this bug but it was put on v2.0 about a month ago.

Comment 22

[Diego Marcos [:dmarcos]]

Can we close this then?

Comment 23

[Zac C (:zac)]

Yes please.

Comment 24

[[:RJ]]

No STR is present to create test case to address bug.