1157495 - [meta] Pocket quality and test eng tasks

Status: NEW

(Firefox :: Pocket, defect)

Comment 1

[Stuart Philp :sphilp]

Current plan includes:
- Testing matrix for account and user flows (Softvision) - existing users vs new, various paths to get data into pocket, CTAs, etc.
- FxA Integration (Services QA)
- "Add/Open/Save to Pocket" buttons/menu items (Softvision and Services QA)
- Verify items are added to pocket manually (Softvision and Services QA) 
- e2e test (fxa creation/login -> add item to pocket -> verify item added via api) (Services QA)

Unknowns:
- talk of languages other than en-us?
- are we expecting load in excess of current FxA testing?
- should we test FxA on non-desktop (like mobile web), or even non-mozilla (pocket addons and apps)? pocket is responsible for these, but this reflects on our product as well
- other items?

Given the timelines and lack of beta time, there is some risk to this being adequately tested. We will likely prioritize manual testing above all else as that will be the most effective use of time right now.

Comment 2

[Stuart Philp :sphilp]

Some other thoughts. 

Are pocket ready to handle a potential surge (do we have a number in mind)?  What kind of load testing do they do?

I assume we have guarantees/SLAs in place with them, what are they? Average API request time, round trip for add and then get an item (is there a delay or instant, can we guarantee this?). API availability/outage guarantees.

Comment 3

[Stuart Philp :sphilp]

Updates:

- Only en-us locale for June 2
- jrgm is investigating FxA load testing needs, will add some oauth testing for pocket into fxa test suite as well
- need to see nightly/beta builds to get a complete idea for manual testing plans
- No SLA in place, so I'd like to find out what their expectations are here to at least be on the same page
- will arrange time with pocket qa to get an idea of what they are doing on their side (ownership of properties, integrations,

Comment 4

[Thomas Huelbert]

    some thoughts around mobile testing/pocket testing ownership: there seem to be 2 drivers for updating mobile apps - changing to keep up with the ecosystem (for example, ios8 release or android 5) as well as changes made for features/bug fixes. Understanding the cadence of shipping and the ownership around regression of the mobile clients would be good.

The impact of a bad user experience on these mobile clients could negatively impact the adoption/usage of pocket in firefox.

Comment 5

[John Dorlus [:Silne30]]

Hi, Stuart. The tests I am standing up for services-marionette. Should they block this issue?

Comment 6

[Stuart Philp :sphilp]

Up to you. This bug was mainly for the initial release, so the pocket tests now are just for sanity checks to make sure it doesn't break.