625826 - Update failures due to wizard pages not set correctly for direct and fallback updates

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[u279076]

http://mozmill-release.brasstacks.mozilla.com/#/update/overview?branch=4.0&channel=beta&from=2010-12-19&to=2011-01-22

Sometimes when we run update tests, the new version is the same as the old version.  As a result, we get false-negatives in the dashboard.  This causes great confusion among the release-drivers.  It would be very helpful if we can fix these types of failures.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Something is wrong with the wizard. The expected page isn't set for direct and fallback updates, so we fail right before we apply the update.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: More descriptive failure message

We need more information to be able to analyze/fix this issue as fast as possible.

Comment 3

[u279076]

Comment on attachment 503920 [details] [diff] [review]
More descriptive failure message

>-    this._controller.waitFor(function() {
>+    this._controller.waitFor(function () {
>       return this.currentPage != WIZARD_PAGES.dummy;
>-    }, "Dummy wizard page has been made invisible.", undefined, undefined, this);
>+    }, "Dummy wizard page has been made invisible - got " + this.currentPage +
>+      ", expected not " + WIZARD_PAGES.dummy, undefined, undefined, this);

Should this be "expected" or "expected not"?  Don't we want all error messages to be "got X, expected Y"?

Other than that, r+.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #3)
> >       return this.currentPage != WIZARD_PAGES.dummy;

This should also be !==

> >+    }, "Dummy wizard page has been made invisible - got " + this.currentPage +
> >+      ", expected not " + WIZARD_PAGES.dummy, undefined, undefined, this);
> 
> Should this be "expected" or "expected not"?  Don't we want all error messages
> to be "got X, expected Y"?

We don't really know which page will be displayed after the dummy page. So this test just checks if another page has been selected. Because of that situation there is no way to say "expected" only.

Comment 5

[u279076]

If we don't know what is expected, I'd rather see just "got".

"expected not" is confusing and makes me think "we do not expect to see this page" -- which is not the case.

I'd say just use "got" in this case.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: More descriptive failure message v2 [checked-in]

Taking over r+ with the changed made.

Landed as:
http://hg.mozilla.org/qa/mozmill-tests/rev/e648179371a3 (default)

Anthony, please re-run the update tests for b9 for the releasetest channel and post the results. I will check it tomorrow.

Comment 7

[u279076]

San Jose downtime is preventing me from testing this right now.  I'll check in the morning.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Were you able to check it Anthony? Right now I cannot open brasstacks due to bug 626147.

Comment 9

[u279076]

(In reply to comment #8)
> Were you able to check it Anthony? Right now I cannot open brasstacks due to
> bug 626147.

No, it seems to be up now, but it's getting a bit late in my day.  I will redo the check tomorrow morning.

Comment 10

[u279076]

http://mozmill-release.brasstacks.mozilla.com/#/update/detail?branch=4.0&channel=releasetest&from=2011-01-16&to=2011-01-17&target=4.0b9

The logs indicate all passes, but it seems some of these did not get reported to brasstacks.  As you can see, I only have results for b7 and b8.  I'm missing results for a5, b6, and mac-b7.  There are PASS results for these in the logs though.

I'll re-run this again tomorrow to verify.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #10)
> http://mozmill-release.brasstacks.mozilla.com/#/update/detail?branch=4.0&channel=releasetest&from=2011-01-16&to=2011-01-17&target=4.0b9
> 
> The logs indicate all passes, but it seems some of these did not get reported
> to brasstacks.  As you can see, I only have results for b7 and b8.  I'm missing

As already said last week this view is not that helpful as you expect at the moment. Given you are querying for a specific target build, all update paths which have been failed will not be visible.

For now you really should use the overview list:
http://mozmill-release.brasstacks.mozilla.com/#/update/overview?branch=4.0&channel=releasetest&from=2011-01-17&to=2011-01-18

Looking at the failures we only get ones for fallback updates in two cases:

> New wizard page has been selected - got errors, expected finished 
> testFallbackUpdate/test2.js:79

So this indicates that we expected the finished wizard page to appear? What? If you check the code you can see that we expect to see the errors page:

>79: update.waitForWizardPage(softwareUpdate.WIZARD_PAGES.errorPatching);

So why it expects the finished page?

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Fix for download progress value which is a string [checked-in]

Ironically the value of the download progress element is not a number but a string. That means my change in the last patch from '== 100' to '=== 100' fails now because we expect a number and not a string. As result we time out and don't invalidate the patch.

I have checked-in that patch. Anthony, please test again with now hopefully better results.

http://hg.mozilla.org/qa/mozmill-tests/rev/01058ffc7cb9

Comment 13

[u279076]

http://mozmill-release.brasstacks.mozilla.com/#/update/overview?branch=4.0&channel=releasetest&from=2011-01-18&to=2011-01-19

Looks like an improvement, however we still see one instance of a b8 -> b8 and b7 -> b7 update path.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We also have updates from b6 and 3.7a5 which go straight to b8 only. Why those don't get updates for b9? Anthony, can you please check the wiki output of the test runs to make sure we are seeing really the correct bit on the Dashboard?

Further the following test is meant to be a restart test but it doesn't get marked as such:

http://mozmill-release.brasstacks.mozilla.com/#/update/report/e232550601ce3a3b5d3363646f056ea8

Not sure what's going on here. But that would explain the failure. I will have to check that.


"updates": [
    {
      "fallback": false,
      "build_pre": {
        "buildid": "20101104142426",
        "disabled_addons": "",
        "locale": "fr",
        "version": "4.0b7",
        "user_agent": "Mozilla/5.0 (Windows NT 5.1; rv:2.0b7) Gecko/20100101 Firefox/4.0b7"
      },
      "build_post": {
        "buildid": "20101104142426",
        "disabled_addons": "",
        "locale": "fr",
        "version": "4.0b7",
        "user_agent": "Mozilla/5.0 (Windows NT 5.1; rv:2.0b7) Gecko/20100101 Firefox/4.0b7"
      },
      "success": false,
      "patch": {
        "buildid": "20110110191547",
        "url": "http://ftp.df.lth.se/mozilla/firefox/releases/4.0b9/update/win32/fr/firefox-4.0b9.complete.mar",
        "download_duration": 360833,
        "type": "minor",
        "is_complete": true,
        "channel": "releasetest",
        "size": 15186784
      }
    }
  ],

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, lets keep by the original report for this bug. There are two reasons why we have failed:

1. There were instances when we tried to download an update but we failed, and ended up on the error wizard page. I was able to reproduce that once manually some hours ago. Sadly I don't know which mirror I hit, because logging was disabled. I would suggest if that happens again we should check for the reason immediately.

2. We failed to download the update because of a missing billboard page for beta releases. See bug 626839. Christian pushed those pages about 5 minutes ago and they should be available soon. That means we don't see the second problem anymore.

Anthony, can you please check those updates tomorrow again? I have copied the release_update.py script to test_update.py, which reports to mozmill-crowd instead. Would be great to see results for all platforms.

I don't think we have to do any more work on this bug.

Comment 16

[u279076]

(In reply to comment #15)
> Anthony, can you please check those updates tomorrow again? I have copied the
> release_update.py script to test_update.py, which reports to mozmill-crowd
> instead. Would be great to see results for all platforms.

I'll test this with 4.0b10 updates when they come out later today.

Comment 17

[u279076]

Seems to be ok:
http://mozmill-release.brasstacks.mozilla.com/#/update/detail?branch=4.0&channel=releasetest&from=2011-01-25&to=2011-01-26&target=4.0b10

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Backport (1.9.2)

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I would tend to say that we do not backport those changes to 1.9.1. It's not worth doing it because we don't have any failures and the code path is too different. Lets better kill that branch. :)

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Backport landed as:
http://hg.mozilla.org/qa/mozmill-tests/rev/772ceed777c3 (1.9.2)

No more work will happen on this bug, now that all the problems have been solved and we do not test alpha builds anymore. Lets close it.