Closed Bug 1030459 Opened 6 years ago Closed 6 years ago

gaiatest test_endurance_camera_photo.py broken in FFOS 2.0

Categories

(Firefox OS Graveyard :: Gaia::UI Tests, defect, P1)

ARM
Gonk (Firefox OS)
defect

Tracking

(blocking-b2g:2.0+, b2g-v2.0 fixed, b2g-v2.1 fixed)

RESOLVED FIXED
blocking-b2g 2.0+
Tracking Status
b2g-v2.0 --- fixed
b2g-v2.1 --- fixed

People

(Reporter: tkundu, Unassigned)

References

Details

(Keywords: regression)

Attachments

(1 file)

Attached file failure logs
We are seeing following error for gaiatest failure : 

raise JavascriptException(message=message, status=status, stacktrace=stacktrace)
JavascriptException: JavascriptException: TypeError: apps is undefined
stacktrace:
	execute_async_script @gaia_test.py, line 44
	inline javascript, line 109
	src: "    let installedApps = apps.installedApps;"
TEST-UNEXPECTED-FAIL | test_endurance_camera_photo.py 


It seems like that this testcase is broken in FFOS 2.0. It is working fine v1.5
blocking-b2g: --- → 2.0?

Justindarc or diego, please check this test failure.

Thanks
Hema
Flags: needinfo?(jdarcangelo)
Flags: needinfo?(dmarcos)
Blocking because of test failure (works fine in previous version)
blocking-b2g: 2.0? → 2.0+
We don't maintain the python integration tests and the message is not very descriptive. We now have integration tests using marionette js that run on device. 

https://github.com/mozilla-b2g/gaia/tree/master/apps/camera#integration-tests

The failing tests can be easily replicated in JS. I'm happy to help migrate it so we can help maintain it. We have no bandwidth to keep two identical test suites in sync.
Flags: needinfo?(dmarcos)
(In reply to Diego Marcos [:dmarcos] from comment #3)
> We don't maintain the python integration tests and the message is not very
> descriptive. We now have integration tests using marionette js that run on
> device. 

If they do, they aren't running in a CI environment. The earliest plan to have MarionetteJS on device in a CI environment is planned in Q3 for only the most critical tests on the phone to be ran on a per commit basis. There's no plans to support a CI-based solution here for MarionetteJS for all tests that currently exist in the tree as of right now.
Anyways - the problem here is in Gaia UI Tests, not developer-driven integration tests, so I'm moving this over to UI Tests & adding needinfo on Rob to take a look.
Component: Gaia::Camera → Gaia::UI Tests
Flags: needinfo?(rwood)
The problem here is that the get_permission/set_permission need to be executed from the system frame.

The user is running without --restart flag (which the tests are designed to run with) which causes the test to execute set_permission from the homescreen frame.
As per comment #6
Flags: needinfo?(rwood)
Flags: needinfo?(jdarcangelo)
From Moz's perspective, we can't block on this. CAF is welcome to take advantage of the tests & framework written, but Moz is not liable for fixing issues on CAF's behalf. CAF needs to assign their own engineers to resolve the problem.
blocking-b2g: 2.0+ → 2.0?
(In reply to Jason Smith [:jsmith] from comment #4)
> If they do, they aren't running in a CI environment. The earliest plan to
> have MarionetteJS on device in a CI environment is planned in Q3 for only
> the most critical tests on the phone to be ran on a per commit basis.
> There's no plans to support a CI-based solution here for MarionetteJS for
> all tests that currently exist in the tree as of right now.

It would be nice to get a timeline in place to get MarionetteJS integration tests running in CI. The Python-based tests are cumbersome to work with because they rely on a library that is fetched from the PIP Python package manager repo. Any time the application changes, the Python library in the PIP repo also needs to be updated to reflect it and we are not aware of policy for how/when/who this gets updated.
That's not correct about gaiatest/pypi because you can run it straight from the tree, but in any case QA are also working to get the JS tests on device too.
(In reply to Zac C (:zac) from comment #10)
> That's not correct about gaiatest/pypi because you can run it straight from
> the tree, but in any case QA are also working to get the JS tests on device
> too.

But, are our CI environments set up to pull gaiatest from the tree? I'm fairly certain at some point, CI was pulling it from PIP, but I might be wrong.
(In reply to Justin D'Arcangelo [:justindarc] from comment #11)

> 
> But, are our CI environments set up to pull gaiatest from the tree? I'm
> fairly certain at some point, CI was pulling it from PIP, but I might be
> wrong.

Yes they are using in-tree! 
but we package and push it to pypi so that some outside-tree test suites can take advantage of the test framework.
(In reply to Zac C (:zac) from comment #12)
> Yes they are using in-tree! 
> but we package and push it to pypi so that some outside-tree test suites can
> take advantage of the test framework.

Ok, good to know! Us Camera devs are Python n00bs, so its also good news to know that QA is working on getting the JS integration tests working in CI :-)
(In reply to Justin D'Arcangelo [:justindarc] from comment #13)
> (In reply to Zac C (:zac) from comment #12)
> > Yes they are using in-tree! 
> > but we package and push it to pypi so that some outside-tree test suites can
> > take advantage of the test framework.
> 
> Ok, good to know! Us Camera devs are Python n00bs, so its also good news to
> know that QA is working on getting the JS integration tests working in CI :-)

FTW - you can see the definition of the goal around this here - https://wiki.mozilla.org/QA/Goals/2014q3#Firefox_OS
(In reply to Jason Smith [:jsmith] from comment #14)
> (In reply to Justin D'Arcangelo [:justindarc] from comment #13)
> > (In reply to Zac C (:zac) from comment #12)
> > > Yes they are using in-tree! 
> > > but we package and push it to pypi so that some outside-tree test suites can
> > > take advantage of the test framework.
> > 
> > Ok, good to know! Us Camera devs are Python n00bs, so its also good news to
> > know that QA is working on getting the JS integration tests working in CI :-)
> 
> FTW - you can see the definition of the goal around this here -
> https://wiki.mozilla.org/QA/Goals/2014q3#Firefox_OS

Meant to say FWIW, not FTW
Keywords: regression
Rob, can you take a look at this when you get back?
Flags: needinfo?(rwood)
Indra, it will be good to know if things changed based on our offline discussion about CAF running the tests per recommendation in comment #6.

In parallel, we have someone from our A-team investigating this as well.
Flags: needinfo?(ikumar)
(In reply to Jonathan Griffin (:jgriffin) from comment #16)
> Rob, can you take a look at this when you get back?

We fixed this on master inside the framework, no need for Rob to investigate.
Flags: needinfo?(rwood)
blocking-b2g: 2.0? → 2.0+
(In reply to Zac C (:zac) from comment #18)
> (In reply to Jonathan Griffin (:jgriffin) from comment #16)
> > Rob, can you take a look at this when you get back?
> 
> We fixed this on master inside the framework, no need for Rob to investigate.

Will that also be patched on the gaia 2.0v branch which is where CAF pulls from and is hitting this issue ?
Flags: needinfo?(zcampbell)
> 
> We fixed this on master inside the framework, no need for Rob to investigate.
:zac -- Can you please also point us to the bug with the fix.
Flags: needinfo?(ikumar)
The patch was unrelated to this bug but it was put on v2.0 about a month ago.
Flags: needinfo?(zcampbell)
Can we close this then?
Flags: needinfo?(zcampbell)
Yes please.
Status: NEW → RESOLVED
Closed: 6 years ago
Flags: needinfo?(zcampbell)
Resolution: --- → FIXED
No STR is present to create test case to address bug.
QA Whiteboard: [QAnalyst-Triage?]
Flags: needinfo?(ktucker)
Flags: in-moztrap?
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+]
Flags: needinfo?(ktucker)
Flags: in-moztrap?
Flags: in-moztrap-
You need to log in before you can comment on or make changes to this bug.