1226638 - Build out Marionette Test Infrastructure for Firefox Accounts and 3rd party services (e.g. pocket)

Status: RESOLVED FIXED

(Cloud Services :: QA: Test Automation, defect)

Description

[John Dorlus [:Silne30]]

This is to house our End to End UI Tests for Firefox Accounts and the Third Party Services we integrate with i.e. Pocket.

These tests differ from the TPS tests because they specifically test certain actions that have to do with Sync. TPS also helps with the profile creation and syncing workflow so those tests are not the same as the UI Tests that will live in Firefox-UI-Tests.

Comment 1

[John Dorlus [:Silne30]]

Attachment: Patch for Test Architecture for Services Tests

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So I have some additional question here to get a vision how your execution strategy will look like.

* Do you have any plans where those tests are running?
* Which platforms do you want to test?
* Are there specific requirements for tests (like credentials etc)

Also as you started to create your tests, all of them are located in a separate top folder beside the ui tests. With that you will need your own harness entry script and maybe other customizations. Skimming over the PR I don't see anything from firefox-ui-tests which is used, except the puppeteer library. So I question the request that this should go into the firefox-ui-tests repository. I feel its better to create a separate repository and have the firefox-ui-tests repository as submodule. Also keep in mind that our tests are moving into mozilla-central in the near future which also affects the puppeteer library. At least then there is no relationship anymore to our tests. See bug 1214372 for further details and ongoing status.

Further adding those tests to our repository now would cause me a lot more work to get all into mozilla-central. So it's another indication for its own repository.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8690154 [details] [review]
Patch for Test Architecture for Services Tests

Resetting request for now until I got the requested information.

Comment 4

[John Dorlus [:Silne30]]

The decision is to house the tests in another as [:whimboo] suggested.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Can you please be more specific? Your answer is very short and doesn't really contain all the information I asked for. Also what does another mean? If you refer to a separate github repository, this won't work due to the reasons I gave above. The firefox-ui-tests repository will not live anymore longer once all trees have the tests landed.

Comment 6

[John Dorlus [:Silne30]]

Gotcha. I think I had a misunderstanding.

1.) We are working on formalizing our test plan as we speak. Right now, our plans for the different applications we support are all scattered across Mozilla Wiki.

2.) We will be testing on Windows, Linux and I believe Mac but I will check with Stuart in the morning on that.

3.) Some tests will have creds required. Services has already started creating some infrastructure to run the tests. 

I think you are absolutely right. As I was typing this, it does seem that this would be most beneficial living in another repo. But, as you said, the repo is on it's deathbed. I thought that I heard (might be mistaken) that Puppeteer was being broken out into it's own module. If that is the case, should I make that the submodule for the project that I house in our own repo? Could you foresee any reason that doing that would not work?

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

firefox_puppeteer will become its own package yes. The ground work for that I'm doing on bug 1232967. Once we got all remaining issues sorted out I will move everything to tree this week. It will then be available via the common.tests.zip file, and you might then be able to include your tests there as well.

Keep in mind that for now we have some remaining extra depencencies between the puppeteer and harness packages. Those I will have to get removed in the next couple of weeks.