Closed Bug 930556 (sms-integration-tests) Opened 8 years ago Closed 5 years ago

[Meta][Messages][Tests] Improve integration test coverage

Categories

(Firefox OS Graveyard :: Gaia::SMS, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: caiolima, Unassigned)

References

Details

(Keywords: meta)

Let's start to develop UI-tests to messaging application. Nowadays there are only few tests on https://github.com/mozilla-b2g/gaia/tree/master/tests/python/gaia-ui-tests/gaiatest/apps/messages
Duplicate of this bug: 930552
Component: Gaia::SMS → Gaia::UI Tests
Caio, what more tests do you have in mind?
I'd like that we (as in the "comms apps team") do Javascript-based marionette tests instead of Python UI tests.

Of course it makes sense to have supplementary tests and not the same one, but as a first step, it would already be good to have at least one test, so that it's easy to add more then.
(In reply to Julien Wajsberg [:julienw] from comment #3)
> I'd like that we (as in the "comms apps team") do Javascript-based
> marionette tests instead of Python UI tests.
> 
> Of course it makes sense to have supplementary tests and not the same one,
> but as a first step, it would already be good to have at least one test, so
> that it's easy to add more then.

Actually, I was talking about the difference of javascript-based tests(or functional tests) and UI tests. I don't know exactly the difference, only that the Python-based tests can run on device.

But, Why do you prefer the Javascript-based tests? And Why do we have these 2 ways of testing?

(In reply to Zac C (:zac) from comment #2)
> Caio, what more tests do you have in mind?

I'm starting to thinking in them. Until the end of weeke I'll have more especific Tests cases to implement.
But my original Idea is to implement tests to:

1. Test if the app workflow is correct;
2. Teste if the operations like:
  2.1: Send message works correctly(including attachements);
  2.2: Recieve messages have the right behavior(I mean about bugs #902508 or #907652 and others);
  2.3: Delete messages and treads;
  2.4: Activity 'new' and 'share';

I don't know if these test cases can be implemented using the current Javascript-based marionette. But with Python based, I guess they can.

What do you think?
(In reply to Caio Lima(:caiolima) from comment #4)
> (In reply to Julien Wajsberg [:julienw] from comment #3)
> > I'd like that we (as in the "comms apps team") do Javascript-based
> > marionette tests instead of Python UI tests.
> > 
> > Of course it makes sense to have supplementary tests and not the same one,
> > but as a first step, it would already be good to have at least one test, so
> > that it's easy to add more then.
> 
> Actually, I was talking about the difference of javascript-based tests(or
> functional tests) and UI tests. I don't know exactly the difference, only
> that the Python-based tests can run on device.

The javascript marionette tests will eventually run on device too.

> 
> But, Why do you prefer the Javascript-based tests? And Why do we have these
> 2 ways of testing?

This is mostly history. The JS tests are mostly done by Gaia developers, the Python tests are mostly done by the QA team.

> 
> (In reply to Zac C (:zac) from comment #2)
> 
> I don't know if these test cases can be implemented using the current
> Javascript-based marionette. But with Python based, I guess they can.

Yes they can be done with the JS based marionette ;)
This way, we can write using Javascript-based marionette them. BTW, Javascript-based marionette tests are classified as wich type of test? UI test or Integration test?
UI Tests are integration tests ;)
Could we stay on topic here please? Bugzilla is not the place to start a philosophy discussion about JavaScript vs. Python. The reporter here has the right to pick whatever language he wants that he feels comfortable coding in. The point that matters here is that we're increasing automated coverage, independent of the programming language it's done in. The original email Caio sent in regards to this bug targeted writing tests for gaia-ui-tests for the SMS app in the emulator.
Jason, there is nothing philosophical here. Caio didn't know the difference between the various tests we have, and this ignorance made him think the UI Tests was the only way to do integration tests, which is untrue.

To increase automated coverage, we need Gaia developers to write tests. Gaia developers will write them in JS, and it makes a lot of sense to have someone write the first one so that it's easier to write more. Which is the goal for everyone, I guess.

Of course if Caio is more comfortable doing the tests with the Python UI Tests framework, I have nothing against that, please do it !
(In reply to Julien Wajsberg [:julienw] from comment #9)
> Jason, there is nothing philosophical here. Caio didn't know the difference
> between the various tests we have, and this ignorance made him think the UI
> Tests was the only way to do integration tests, which is untrue.
> 
> To increase automated coverage, we need Gaia developers to write tests. Gaia
> developers will write them in JS, and it makes a lot of sense to have
> someone write the first one so that it's easier to write more. Which is the
> goal for everyone, I guess.

Sorry for my ignorance. But I'm thinking in this way.
 
> Of course if Caio is more comfortable doing the tests with the Python UI
> Tests framework, I have nothing against that, please do it !

Actually, I want to increase the automated test coverage, as you mentioned before. So I guess that this discussion is important. Maybe this strategy to implement integrations tests in Javascript can be better for Gaia developers in general.
I was thinking and I agree with Julien about javascript marionette tests. So, I've decided to start work on what is possible, and maybe work on bug https://bugzilla.mozilla.org/show_bug.cgi?id=932635 to improve the test coverage.
Summary: [meta]Create Gaia-ui-tests for messaging application → [meta]Create marionnete-js integration tests for messaging application
Component: Gaia::UI Tests → Gaia::SMS
Summary: [meta]Create marionnete-js integration tests for messaging application → [Meta][Messages][]
Alias: sms-integration-tests
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: meta
OS: Other → Gonk (Firefox OS)
Hardware: Other → ARM
Summary: [Meta][Messages][] → [Meta][Messages][Tests] Improve integration test coverage
Depends on: 1171989
Depends on: 1200597
Depends on: 943178
Mass closing of Gaia::SMS bugs. End of an era :(
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
Mass closing of Gaia::SMS bugs. End of an era :(
You need to log in before you can comment on or make changes to this bug.