Closed Bug 1652074 Opened 5 years ago Closed 5 years ago

Request to re-implement Firefox application update tests

Categories

(Testing :: General, enhancement, P3)

Product:
Testing
Component:
General
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: tracy, Unassigned)

Details

Filing this request on behalf of the Product Integrity manual test team.

For several years, Mozilla had automated testing of Firefox application updates. Two to three years ago that automation was disabled and the supporting tool deprecated. Since then, QA has been manually covering update tests on each Firefox Beta build across all supported platforms. The QA team would like to request that update test automation be re-implemented to relieve personnel of the time consuming task of Firefox application update testing (approximately 5-6 man hours per Beta, three times per week).

Fortunately, requirements and project planning documents, for the creation of the original tests, in the now deprecated Mozmill, still exist for the first go around of Firefox update test automation. The documentation is old, but primarily still applies and could be useful guides.

Here are the basic Firefox Updates Testing types. In addition to the test types described there, major, minor, partial and complete, each tier 1 locale is included as part of a matrix of tests for each build in the Firefox Beta test cycle.

Here is the original On Demand Testing project plan and documentation. Some of the underlying technical solutions have likely changed. However, the scope of what would have to be accomplished is still relevant. Related bugs: bug 628659, bug 657081 and bug 1355026

Lastly, as part of the update testing, the manual QA team also runs checks for basic browser functionality (it can navigate to a page), search code validation and what's new page validation, after each update applied. Including those checks in this automation would be ideal, if possible. If not, the basic update test automation would greatly help the QA team redirect personnel to more valuable testing efforts.

If I understand correctly, part of the reason support for these tests was dropped was difficulty in maintenance. Would the cost of re-implementing and supporting these tests again be greater than the value of saving 15-20 QA man hours of update testing per week?

As mentioned the update tests have been stopped running. That was a decision after a very long discussion with application update developers and various managers. No-one felt that these tests were useful. So the final stop has been done via bug 1386628 about 2 years ago. Based on that feedback we got I don't understand why update tests are still run and than even manually.

Nevertheless, code-wise the firefox-ui harness, which handled the update tests, has been removed via bug 1573406. Also the firefox-puppeteer Python package is no longer present (bug 1573383). Further we have the intention to completely kill the firefox-ui harness itself; currently we still run some UI tests but these are getting moved to Marionette proper. As such no easy re-implementation will be possible at all.

Further please note that we are currently working on the Remote Protocol, and Marionette will finally be decommissioned once the BiDi protocol is in-place. As such I highly advice to not re-add the previous features, which would only block this transition.

A new type of test probably based on geckodriver might have to be added. But this is outside of our team's scope. As such I will move this bug to the general testing component and CC Geoff and Andrew for further brainstorming.

Component: Firefox UI Tests → General
QA Contact: hskupin

I'm not very familiar with update testing, but at a glance this seems like a significant project. I would refer this to ajones to consider priority and resourcing; unfortunately, I think he has just started pto.

Flags: needinfo?(ajones)
Severity: -- → S3
Priority: -- → P3
Flags: needinfo?(ajones)

Dave, can your team look at this? First off, would the cost of re-implementing and supporting the application update tests again be greater than the value of saving 15-20 QA man hours of update testing per week?

Flags: needinfo?(dave.hunt)

This isn't something that would fall under the performance test engineering team. Before kicking off such a project, we should seek buy-in from application update engineering. As Henrik mentioned in comment 1 the previous tests were not considered to be useful by these engineers, so it would be critical to identify and address these concerns up-front.

Flags: needinfo?(dave.hunt)

Henriks comment is misleading. The tests were valuable until they became unreliable. I asked mhowell what happened those few years ago; why the automation of the update tests was shutdown. Her response, "Yeah, I think the request to stop running them (the automated tests) came from our side; we had concerns about them being brittle because there was some subtle dependency on implementation details, and we weren't convinced at the time that the results were valuable enough to keep fixing them.

Then I asked if she thought having update automation tests would be useful particularly since they are working on a background update agent. Her response was, "Good question. Since this is going to be a bunch of new code, I do think it would be great to have something running the whole process end to end, at least for a while until we've built up some confidence in it."

Having the tests automated again would save many manual hours of testing every cycle.

Molly, It seems this has circled back around to you and the update team.

Flags: needinfo?(mhowell)

(In reply to Tracy Walker [:tracy] from comment #5)

Henriks comment is misleading. The tests were valuable until they became unreliable.

I don't want to bring-up this topic again. But there was nothing unreliable! They were working fine across platforms and locales. Not sure who actually started to make this claim from the application update team but at the end we just gave up.

For anything wanted /needed in the future lets build upon some other harness / framework but not Marionette harness. It's support is fading out.

I'm not sure if a specific question is being asked of me, so I'll just provide my thoughts on what's been said here. First of all, I confirm the quotes from me in comment 5 are accurate, just to be totally transparent about that.

"Unreliable" isn't the word I would use; that implies the tests were intermittent or something similar, which I do not recall being the case. The problem as I remember it (this was years ago and I was not the point person in these conversations) was that they were dependent on details of how some UI worked, those details changed, and when the task of fixing the tests fell mostly to us (the app update team), we were not able to do so on a reasonable time frame. It is possible that similar tests written against a different framework (as is being discussed here) may not be as susceptible to the same problems, I really have no idea.

(In reply to Tracy Walker [:tracy] from comment #5)

Then I asked if she thought having update automation tests would be useful particularly since they are working on a background update agent. Her response was, "Good question. Since this is going to be a bunch of new code, I do think it would be great to have something running the whole process end to end, at least for a while until we've built up some confidence in it."

The basic architectural design for the background update agent has changed since that conversation, such that we now plan to reuse existing update service code instead of rebuilding it, so the bunch of new code I was talking about there is no longer going to get written, and the lack of confidence I was discussing is no longer an issue.

Flags: needinfo?(mhowell)

Thank you Molly and Henrik for the clarification on history.

The ask here is still valid. If not the test automation team or the application update team, who should I seek to automate the update tests?

Hi everyone. Resurrecting this conversation, trying to understand things from the Release Engineering perspective.

Can we please assume for a second that update tests are already implemented as of yesterday and are a blackbox that has an input (OS, locale, update-channel) and an output? (True/False for successful update test). If we had this blackbox, I think the next step would be to:

  1. get them running at all in taskcluster
  2. figure out good chunking/scheduling for them to be enabled for all types of releases: release, ers, beta and nightly builds.

Now, going back to the blackbox, what I understand from this bug is:
a) blackbox is useful as it automates 15-20h/w worth of QA manual work
b) we used to have this blackbox a few years back and it relied on tools such as firefox-ui harness, firefox-puppeteer, MozMill and Jenkins. it was discontinued at some point.
c) reimplementing a new blackbox is useful today but can't rely on previous tools as their support/maintenance has been discontinued
d) rebuilding this blackbox would take significant effort

Questions:

  1. did I get this right?
  2. can someone please tl;dr in a nutshell what the blackbox is actually doing: does it drop a mar in place and tests what happens when you apply/restart?

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9)

  1. can someone please tl;dr in a nutshell what the blackbox is actually doing: does it drop a mar in place and tests what happens when you apply/restart?

No, it's simulating what a user would actually do to update Firefox. As such it downloaded/installed a previous version of Firefox (a locale as specified). Then the About dialog was opened, and a check for updates has been performed.

Then we had two scenarios:

  1. A direct update that downloaded the partial/full update and applied it
  2. A fallback update that also downloaded the partial/full update but invalided it, so that another full update gets downloaded

After the restart some basic checks (version, buildid, locale, no other updates) have been done.

Okay, sounds good. Thank you for clarifications.
Is my understanding correct that most of the code for all of that is now deprecated and rebuilding that solution today is a complex task?

Flags: needinfo?(hskupin)

The Firefox ui update tests and its Firefox ui harness additions have been removed on bug 1573406. Also the used Firefox-Puppeteer Python package is gone since bug 1573383. Eventually we will also get rid of the remaining traces of the Firefox UI harness over on bug 1613338.

That said we aren't going to re-add those removed code. A custom repository outside of mozilla-central could be created, to re-integrate all that. Might be a bit of work but could still work. But remember it would remain on unsupported code, and manual synchronization for Marionette updates would be required.

Flags: needinfo?(hskupin)

Found in triaging. IIRC, we agreed for now we won't proceed further with this. Please reopen if I'm wrong.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.