Closed Bug 1298772 Opened 8 years ago Closed 8 years ago

Permafail test_fallback_update.py TestFallbackUpdate.test_update | AttributeError: 'Wizard' object has no attribute 'updates_found_billboard'

Categories

(Testing :: Firefox UI Tests, defect, P1)

Product:
Testing
Component:
Firefox UI Tests
Version:
Version 3
Type:
defect

Tracking

(firefox51 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla51
Tracking Flags:
Tracking Status
firefox51 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: whimboo)

References

Details

(Keywords: intermittent-failure, regression)

Attachments

(1 file)

Bustage fix
1.19 KB, patch
whimboo
: review+
Details | Diff | Splinter Review 
It looks like we didn't remove all occurrences for the billboard from our update tests via bug 1296677. It means the final removal of it inside of Firefox (bug 1296207) caused this perma fail for update tests on mozilla-central.

Here the stack:

 06:40:07     INFO - TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | AttributeError: 'Wizard' object has no attribute 'updates_found_billboard'
 06:40:07     INFO - Traceback (most recent call last):
 06:40:07     INFO -   File "/home/mozauto/jenkins/workspace/mozilla-central_update/build/venv/local/lib/python2.7/site-packages/marionette/marionette_test.py", line 351, in run
 06:40:07     INFO -     testMethod()
 06:40:07     INFO -   File "/home/mozauto/jenkins/workspace/mozilla-central_update/build/tests/firefox-ui/tests/update/fallback/test_fallback_update.py", line 21, in test_update
 06:40:07     INFO -     self.download_and_apply_forced_update()
 06:40:07     INFO -   File "/home/mozauto/jenkins/workspace/mozilla-central_update/build/venv/local/lib/python2.7/site-packages/firefox_ui_harness/testcases.py", line 350, in download_and_apply_forced_update
 06:40:07     INFO -     self.download_update(dialog)
 06:40:07     INFO -   File "/home/mozauto/jenkins/workspace/mozilla-central_update/build/venv/local/lib/python2.7/site-packages/firefox_ui_harness/testcases.py", line 274, in download_update
 06:40:07     INFO -     download_via_update_wizard(window)
 06:40:07     INFO -   File "/home/mozauto/jenkins/workspace/mozilla-central_update/build/venv/local/lib/python2.7/site-packages/firefox_ui_harness/testcases.py", line 234, in download_via_update_wizard
06:40:07 INFO - dialog.wizard.updates_found_billboard,
Assignee: nobody → hskupin
Blocks: 1296207
Status: NEW → ASSIGNED
status-firefox51: --- → affected
Depends on: 1296677
Keywords: regression
Attached patch Bustage fix — — Splinter Review 
We left behind a billboard related line which caused fallback update tests to fail. With its removal all is fine again.

I will go and push this directly to mozilla-central, so that the next tests for Nightly should be fixed.
Attachment #8785907 - Flags: review+
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/4f72b1d05267
Bustage fix for firefox-ui-update tests for billboard removal. r=me, a=testonly
Severity: normal → major
Priority: -- → P1
Summary: Intermittent test_fallback_update.py TestFallbackUpdate.test_update | AttributeError: 'Wizard' object has no attribute 'updates_found_billboard' → Permafail test_fallback_update.py TestFallbackUpdate.test_update | AttributeError: 'Wizard' object has no attribute 'updates_found_billboard'
Should be fixed now.
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
status-firefox51: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla51
Why didn't your try push per bug 1296677 comment #13 find this?
Real update tests we can only run on Nightly builds, so it won't work on try. Also for now we do not have test coverage for the harness and the testcase classes given that I work alone on all those pieces and haven't had the time for that. It might be a good idea to raise a mentored bug in the near future to get such tests added, now that we also got Marionette harness tests lately.
We already have tests that thoroughly verify the app update ui using test updates that run on every test run. I think it would be better to back off from testing the same app update ui twice instead of fixing these cases. If there is something that is tested by these tests beyond the testing of actual updates that can be added to the existing tests then that should be added to the existing tests instead.
We started this topic a couple of times. Compared to your tests we ensure that the ui is working across nearly all locales. This is nothing any of the basic developer tests (xpshell, browser chrome, mochitests) can do.

I think that once we figured out owners of the tests via our email thread we can start planning what's necessary and what's not. Any of the possible changes I won't be able to do all on my own due to work load reasons.

In case of other concerns please respond to this mentioned thread, so we can keep this together at a single place. Thanks.
A couple of more questions before going back to the email thread.

Do these tests check all of the Firefox UI?

Why aren't these only running on l10n builds then?

Do these tests connect to the network other than the test http server?
Flags: needinfo?(hskupin)
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #9)
> Do these tests check all of the Firefox UI?

We check the about window and in case of fallback updates we also check the old software update dialog for the possible steps. The puppeteer tests are basic tests which simply check that the correct wizard pages etc exist. 

> Why aren't these only running on l10n builds then?

Not sure I fully understand. So what we do is to run them first for en-US Nightly builds across all platforms on central and aurora. Then once locale updates have been made available by funsize, we test a handful of locales only due to machine capacity issues (we will increase the number once we can run them all via Taskcluster). For Release builds we trigger tests at once when the update channel got opened. This is happening across all supported platform versions, for various older releases (max 3 versions back, but including the last ESR release), and a handfull of locales.

> Do these tests connect to the network other than the test http server?

There is no local HTTP server in use for those tests. All is happening with the real AUS servers and the CDN network. This ensures that the channel to test really offers the correct partial and complete MAR files.
Flags: needinfo?(hskupin)
(In reply to Henrik Skupin (:whimboo) from comment #10)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #9)
> > Do these tests check all of the Firefox UI?
> 
> We check the about window and in case of fallback updates we also check the
> old software update dialog for the possible steps. The puppeteer tests are
> basic tests which simply check that the correct wizard pages etc exist. 
I'm asking because you stated "Compared to your tests we ensure that the ui is working across nearly all locales. This is nothing any of the basic developer tests (xpshell, browser chrome, mochitests) can do."

If the end goal is to make sure the Firefox UI is working in all locales then just doing it for the UI used by app update wouldn't make sense.

> > Why aren't these only running on l10n builds then?
> 
> Not sure I fully understand. So what we do is to run them first for en-US
> Nightly builds across all platforms on central and aurora. Then once locale
> updates have been made available by funsize, we test a handful of locales
> only due to machine capacity issues (we will increase the number once we can
> run them all via Taskcluster). For Release builds we trigger tests at once
> when the update channel got opened. This is happening across all supported
> platform versions, for various older releases (max 3 versions back, but
> including the last ESR release), and a handfull of locales.
It also can run on try
https://treeherder.mozilla.org/#/jobs?repo=try&revision=35d115e21d9e

Do these tests run with the other tests on m-c? It looks like they do based on the presence of the tc-Fxfn-r[tier 2] (en-US) tests.
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=12fa18765018

> > Do these tests connect to the network other than the test http server?
> 
> There is no local HTTP server in use for those tests. All is happening with
> the real AUS servers and the CDN network. This ensures that the channel to
> test really offers the correct partial and complete MAR files.
Do these tests connect to the network in the cases I noted above?
Flags: needinfo?(hskupin)
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #11)
> > We check the about window and in case of fallback updates we also check the
> > old software update dialog for the possible steps. The puppeteer tests are
> > basic tests which simply check that the correct wizard pages etc exist. 
> I'm asking because you stated "Compared to your tests we ensure that the ui
> is working across nearly all locales. This is nothing any of the basic
> developer tests (xpshell, browser chrome, mochitests) can do."
> 
> If the end goal is to make sure the Firefox UI is working in all locales
> then just doing it for the UI used by app update wouldn't make sense.

That's correct. To be able to eg. go through all wizard pages we would have to simulate an update process in Puppeteer unit tests. That's not what we have right now. In the current state we only check for the presence of those pages, to at least ensure via try and integration branch builds that our update tests are not broken once code reached mozilla-central and a Nightly. Specifically for this bug we failed because the left-over code was part of the update process.

Regarding l10n coverage, we normally add locale specific behavior to our functional tests which get run per commit. 

> > Not sure I fully understand. So what we do is to run them first for en-US
> > Nightly builds across all platforms on central and aurora. Then once locale
> > updates have been made available by funsize, we test a handful of locales
> > only due to machine capacity issues (we will increase the number once we can
> > run them all via Taskcluster). For Release builds we trigger tests at once
> > when the update channel got opened. This is happening across all supported
> > platform versions, for various older releases (max 3 versions back, but
> > including the last ESR release), and a handfull of locales.
> It also can run on try
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=35d115e21d9e
> 
> Do these tests run with the other tests on m-c? It looks like they do based
> on the presence of the tc-Fxfn-r[tier 2] (en-US) tests.
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> central&revision=12fa18765018

fxfn are the functional tests. Those are divided into two groups - fxfn-l (with local test data - reporting as tier-1) and fxfn-r (requiring network access - reporting as tier-2). Both type of tests get run per checkin as stated above, and that for all kinds of builds on Linux64 only at the moment. 

Update tests listed as fxup are run for nightlies and that mostly via mozmill-ci except for Linux64 which is on Taskcluster. All those teste results are reported as tier-3 like here: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=1a5b53a831e5a6c20de1b081c774feb3ff76756c&filter-searchStr=fxup&filter-tier=1&filter-tier=2&filter-tier=3

> > > Do these tests connect to the network other than the test http server?
> > 
> > There is no local HTTP server in use for those tests. All is happening with
> > the real AUS servers and the CDN network. This ensures that the channel to
> > test really offers the correct partial and complete MAR files.
> Do these tests connect to the network in the cases I noted above?

If my answer didn't cover what you expected, I may not understand your question. What kind of test HTTP server do you mean?
Flags: needinfo?(hskupin)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: