756302 - (daily-beta-qa) Automate all applicable smoke tests to be able to scale to the daily betas process

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[u279076]

The following is a bug to track QA concerns and dependencies for rapid betas. If you have a concern, please file a new bug and mark it as a dependency of this one.

Thanks

Comment 1

[Jason Smith [:jsmith]]

Anthony - Could we move this under the Mozilla QA product? Firefox --> General has a ton of bugs, so it may be hard to track this here.

Comment 2

[u279076]

We don't have a Mozilla QA : General component and none of the Mozilla QA components are appropriate for this bug. Firefox : General is the most appropriate. It should be easy enough to track by alias, "rapid-beta-qa". 

If we can get a Mozilla QA : General component added then we can move it there, but for now we will keep it in Firefox : General.

Comment 3

[Justin Dolske [:Dolske]]

Core::Tracking? Doesn't matter a a lot, though.

Comment 4

[u279076]

(In reply to Justin Dolske [:Dolske] from comment #3)
> Core::Tracking? Doesn't matter a a lot, though.

Yup, doesn't matter, and I don't want being righteous about bugzilla components to distract us from the mission at hand: figuring out rapid betas without sacrificing quality. So let's end the "bugzilla component" discussion here. Thanks.

Comment 5

[Jason Smith [:jsmith]]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #4)
> (In reply to Justin Dolske [:Dolske] from comment #3)
> > Core::Tracking? Doesn't matter a a lot, though.
> 
> Yup, doesn't matter, and I don't want being righteous about bugzilla
> components to distract us from the mission at hand: figuring out rapid betas
> without sacrificing quality. So let's end the "bugzilla component"
> discussion here. Thanks.

Fair enough. For rapid betas, known dependencies may include:

Any automation implemented and fixed at the smoke-test level (functional & update) needed in CI

The recent discussion we had for smoke tests review pointed out the following tests that did not have automation below. We need to get either bugs that already exist linked here or logged for tracking the implementation of these smoke tests in Mozmill.

- Install Firefox
- Detach Tab with Flash Running On Tab and Drop Onto Desktop
- Play popular video formats (e.g. ogv)
- Quit Firefox
- Share location in geolocation
- Print common file formats (e.g. PDF)
- Uninstall Firefox
- Breakpad functions and submits reports
- Startup Firefox
- Switch to tab
- Open common file formats (e.g. PDF)

Are there any infrastructure requirements or modifications we need? Such as something with mozmill ci? Modifications to Mozmill itself? Creation of something else entirely? Henrik - Thoughts?

We also have to address the outstanding item involving priority on this, given that k9o and basecamp deserves QA's top priority focus. Alex - Thoughts on priority against other factors (e.g. k9o, basecamp)?

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Jason Smith [:jsmith] from comment #5)
> - Install Firefox

We are doing that via silent installers for Mozmill tests. But there is no way for us to test the installer ui itself.

> - Detach Tab with Flash Running On Tab and Drop Onto Desktop

Not doable yet because it would need platform handling code.

> - Play popular video formats (e.g. ogv)

Shouldn't this be an unit test? 

> - Quit Firefox

There is nothing which blocks us to get this turned into a Mozmill test.

> - Share location in geolocation

We already have such a test, don't have we?

> - Print common file formats (e.g. PDF)

Not doable and similar to the drag and drop thingy.

> - Uninstall Firefox

Same as for install

> - Breakpad functions and submits reports

First is implemented. We do not test submitting reports. Shouldn't this be an unit test too? 

> - Startup Firefox

I miss details but I think we have this automated.

> - Switch to tab

We have a couple of tests for switching tabs.

> - Open common file formats (e.g. PDF)

With the integrated PDF reader probably possible but not with plugins.

> Are there any infrastructure requirements or modifications we need? Such as
> something with mozmill ci? Modifications to Mozmill itself? Creation of
> something else entirely? Henrik - Thoughts?

Biggest blocker would be operations on the operating system side which we do not support yet via Mozmill.

Comment 7

[Alex Keybl [:akeybl]]

(In reply to Jason Smith [:jsmith] from comment #5)
> We also have to address the outstanding item involving priority on this,
> given that k9o and basecamp deserves QA's top priority focus. Alex -
> Thoughts on priority against other factors (e.g. k9o, basecamp)?

Rapid betas take a back seat to these other efforts if we can only do them serially. All teams are working on these other efforts, however, and have so far been able to find the bandwidth.

I'd like to note that I don't believe the tests below:

> - Detach Tab with Flash Running On Tab and Drop Onto Desktop
> - Share location in geolocation
> - Print common file formats (e.g. PDF)
> - Uninstall Firefox
> - Breakpad functions and submits reports
> - Open common file formats (e.g. PDF)

need to happen more frequently than they do today. We've almost never broken these after landing on m-c and uplifting to m-a before landing on m-b, and these are not absolutely critical if broken for a day.

> - Install Firefox
> - Quit Firefox
> - Play popular video formats (e.g. ogv)
> - Switch to tab

The above are a bit more critical and should probably have automated QA testing.

And I'm not sure what this test is:

> - Startup Firefox

Comment 8

[juan becerra [:juanb]]

It feels like this should be in a wiki and not a bug.

If the purpose of this is to file bugs against products and components that when fixed would allow us to create Mozmill tests for what we think a smoketest should be, then sure. But I wouldn't expect those to be fixed before in time for k90 or beyond.

If this is about process, I think meeting with the right people and getting things on wikis would take care of this.

Comment 9

[Jason Smith [:jsmith]]

(In reply to juan becerra [:juanb] from comment #8)
> It feels like this should be in a wiki and not a bug.
> 

Right, the general point originally in the bug isn't appropriate to track in a bug.

> If the purpose of this is to file bugs against products and components that
> when fixed would allow us to create Mozmill tests for what we think a
> smoketest should be, then sure. But I wouldn't expect those to be fixed
> before in time for k90 or beyond.

Let's re-purpose the bug then, as that's makes more sense as a bug to use as a tracker bug. I'll change the title to reflect that.

> 
> If this is about process, I think meeting with the right people and getting
> things on wikis would take care of this.

Agreed.

Comment 10

[u279076]

Automation is but one piece of the puzzle for QA. I'd really prefer we revert the summary back to a meta QA bug and file a blocker automation bug.

Comment 11

[Jason Smith [:jsmith]]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #10)
> Automation is but one piece of the puzzle for QA. I'd really prefer we
> revert the summary back to a meta QA bug and file a blocker automation bug.

Let's talk about that at the desktop meeting - there's a disagreement here on how to track this bug.

Comment 12

[Lukas Blakk [:lsblakk] use ?needinfo]

If it helps, we'll be checking in on the status of this initiative at our weekly channel meetings.  Each team of stakeholders has a status/health and tracked bugs section like this: https://wiki.mozilla.org/Release_Management/Rapid_Betas#QA_2

It would be great to have a list of the bugs QA files & fixes to get their part of implementing (and automating) the QA needs of daily betas completed.  These bugs can be added to the wiki page and, since the query is dynamic, we'll have up to date information at each channel meeting.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

From all the possible smoketests only the following two are missing.

> - Play popular video formats (e.g. ogv)

Bug 489474 and should be done soon.

> - Share location in geolocation

Bug 758187, which is blocked by an issue with the notification element if Firefox hasn't the focus. Needs to be fixed first, so we are blocked on it.

Comment 14

[Sylvestre Ledru [:Sylvestre]]

Mozmill is dead, WONTFIX the remaining bugs.