1355026 - Update Firefox UI update tests for new simplified update UI

Status: RESOLVED WONTFIX

(Testing :: Firefox UI Tests, enhancement)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Bug 893505 landed over the weekend, which means that we have a new updater UI by default now. To ensure that our update tests are testing what the end-users see, the tests would have to be updated.

For now we will disable the new ui via bug 1355009.

Florin, I would like to know which parts we have to test, and which elements to interact with. Bug 893505 has some manual testcases at the very end. So maybe you can check what would be necessary for automation, and what gets done by manual testing if those things will still happen? Thanks.

Comment 1

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

I don't think we need additional automated cases in our tests because of this change. From what I see this is for prompting some users to update. The scenarios from https://bugzilla.mozilla.org/show_bug.cgi?id=893505#c197 should be covered manually, and also I see we have some new automated tests here: https://reviewboard.mozilla.org/r/100222/diff/34#index_header - maybe you could check these out Henrik?

Manually testing this, it seems to me that only the Fallback Update scenario has changed: following the steps from https://bugzilla.mozilla.org/show_bug.cgi?id=1333803#c2 I no longer got the window that tells a user to download the complete.mar file... in fact the window seems to not show at all after this change - but if you go to the About window you can see that the complete mar is being downloaded, and the restart button shows after the download ends. It seems to me that this is the only change that we would need for our tests.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #1)
> should be covered manually, and also I see we have some new automated tests
> here: https://reviewboard.mozilla.org/r/100222/diff/34#index_header - maybe
> you could check these out Henrik?

Sorry but I don't have the time to check other existing automated or manual tests. I hardly can work on patches to keep the update tests working. So I will trust your judgement.

> Manually testing this, it seems to me that only the Fallback Update scenario
> has changed: following the steps from
> https://bugzilla.mozilla.org/show_bug.cgi?id=1333803#c2 I no longer got the
> window that tells a user to download the complete.mar file... in fact the
> window seems to not show at all after this change - but if you go to the
> About window you can see that the complete mar is being downloaded, and the
> restart button shows after the download ends. It seems to me that this is
> the only change that we would need for our tests.

Is this behavior happening for both partial and complete patches downloaded in the first step for a fallback update?

Comment 3

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

(In reply to Henrik Skupin (:whimboo) from comment #2)
> Sorry but I don't have the time to check other existing automated or manual
> tests. I hardly can work on patches to keep the update tests working. So I
> will trust your judgement.

Makes sense, it looked like the tests should be good so I think we can work with that assumption (Kanchan who owns the QA part for this should also know more).

> > Manually testing this, it seems to me that only the Fallback Update scenario
> > has changed: following the steps from
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1333803#c2 I no longer got the
> > window that tells a user to download the complete.mar file... in fact the
> > window seems to not show at all after this change - but if you go to the
> > About window you can see that the complete mar is being downloaded, and the
> > restart button shows after the download ends. It seems to me that this is
> > the only change that we would need for our tests.
> 
> Is this behavior happening for both partial and complete patches downloaded
> in the first step for a fallback update?

Doesn't seem to be the same behaviour - the only way I could test what happens after we get the complete mar was to follow the same steps as above (https://bugzilla.mozilla.org/show_bug.cgi?id=1333803#c2), and then when the complete mar download completed I closed Firefox, edited the update.status file again (failed: 6) and reopened Firefox. In this case I got the dropdown saying that "Nightly can't update to the latest version", with a button called "Download Nightly" taking me to https://www.mozilla.org/en-US/firefox/channel/desktop/#nightly.

I haven't seen the above drop-down for the partial update case (in that case, after updating update.status and restaring Firefox, I went to the About window and saw that the complete mar was already downloading). Hope this helps.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hm, that's interesting. I wonder if that is really expected that when a complete update is faulty, we do not actually retry to download it again. But ask the user to download a copy of Firefox manually.

If that is really the case a fallback update for complete updates could be stripped by only checking for that popup.

Matt, do you have that information or whom else we could ask?

Comment 5

[Molly Howell (she/her) (mostly inactive)]

That is indeed the expected behavior; the complete update is exactly what the fallback mechanism is trying to fall back to, so if that one doesn't apply correctly, there's nothing else left to try except asking the user to reinstall.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Matt Howell [:mhowell] from comment #5)
> That is indeed the expected behavior; the complete update is exactly what
> the fallback mechanism is trying to fall back to, so if that one doesn't
> apply correctly, there's nothing else left to try except asking the user to
> reinstall.

Do you remember since which release we are actually doing that? I ask because it would be great to not have to add too many fallback cases due to releases we have to test which do not include this path.

Comment 7

[Molly Howell (she/her) (mostly inactive)]

The most recent thing I can see that might have changed that behavior (but I'm not sure it did) was bug 530872. So it's been... a while.

I haven't really been following the new update UI work (bug 893505) that just landed on nightly, so I'm not sure what effects it would have here.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

> The most recent thing I can see that might have changed that behavior (but
> I'm not sure it did) was bug 530872. So it's been... a while.

Ok, thinking more about all this... 45esr is getting desupported soon. Also running update tests for those kind of releases only result in a 52esr build at maximum (when we upgrade users to the newer esr branch). So if 52esr has this behavior we can perfectly get rid of the old behavior for complete fallback updates.

> I haven't really been following the new update UI work (bug 893505) that
> just landed on nightly, so I'm not sure what effects it would have here.

Thanks Matt! Ok, so lets ask Doug who mainly worked on the new UI. Doug, it would be great if you could go through the existent comments and tell us if we are on the right track for updating the firefox ui update tests for adding support of the new simplified update ui. Thanks.(In reply to Matt Howell [:mhowell] from comment #7)

Comment 9

[Alex Thayer [:alexical] (she/her)]

(In reply to Henrik Skupin (:whimboo) from comment #8)
> Thanks Matt! Ok, so lets ask Doug who mainly worked on the new UI. Doug, it
> would be great if you could go through the existent comments and tell us if
> we are on the right track for updating the firefox ui update tests for
> adding support of the new simplified update ui. Thanks.(In reply to Matt
> Howell [:mhowell] from comment #7)

From what I can see, this all sounds right, but a few clarifications:

- If the patch fails to apply, the user should be prompted to download the installer.

- - However if the patch fails to apply due to a failure to elevate to admin privileges, then we should just prompt the user to restart again, until it's failed app.update.elevate.maxAttempts times, at which point the user should be prompted to download the installer. 

- If the patch fails _during_ download (both the partial and complete downloads fail), then the user is prompted to try to download the patch again app.update.download.maxAttempts times, after which the user should be prompted to download the installer.

Comment 10

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

As requested by Henrik I did some additional testing today to see how Firefox behaves when we initially download the partial mar and when we initially download the complete mar. 

The two cases are different in terms of displaying a prompt, but in both cases it seems that the download of the complete mar is triggered in the background from the moment we restart Firefox. See below in more detail what was tested and the results that I got.

OS: Win 8.1 x64, Firefox 32-bit build

Partial MAR:
1. Installed Nightly 55 from May 01 - https://archive.mozilla.org/pub/firefox/nightly/2017/05/2017-05-01-03-02-04-mozilla-central/
2. Opened Firefox (clean profile) and went to Help -> About Firefox to trigger the update download -> partial mar was downloaded (~7 Mb)
3. After download finished I closed Firefox
4. Went to C:\Users\florin.mezei\AppData\Local\Mozilla\updates\2A4B9A4EFEA5EE5D\updates\0
5. Edited the update.status file and replaced the content with: failed: 6 (with a final new line)
6. Opened Firefox again

Result:
NO prompt window appears telling me to download the complete mar. 
Note that if I instead go to the About window, I can see that the complete mar is being downloaded (~48 Mb), and then, when the download completes, the <Restart to update Nightly> button shows. After restarting from the button, Firefox opens and I can see that it's the latest (May 02).

=====

Complete MAR:
1. Installed Nightly 55 from Apr 24 - https://archive.mozilla.org/pub/firefox/nightly/2017/04/2017-04-24-03-02-11-mozilla-central/
2. Opened Firefox (clean profile) and went to Help -> About Firefox to trigger the update download -> complete mar was downloaded (~48 Mb)
3. After download finished I closed Firefox
4. Went to C:\Users\florin.mezei\AppData\Local\Mozilla\updates\2A4B9A4EFEA5EE5D\updates\0
5. Edited the update.status file and replaced the content with: failed: 6 (with a final new line)
6. Opened Firefox again

Result:
A doorhanger appears under the hamburger menu: "Nightly can't update to the latest version. Download a fresh copy of Nightly and we'll help you to install it. See what's new. <Download Nightly>, <Not Now>" (see attached screenshot below). 
If I click on <Download Nightly>, then I'm taken to https://www.mozilla.org/en-US/firefox/channel/desktop/#nightly.
If I instead go to the About window, I can see that the complete mar is being downloaded (~48 Mb), and then, when the download completes, the <Restart to update Nightly> button shows. After restarting from the button, Firefox opens and I can see that it's the latest (May 02). Also note that if I wait for a little while, and only go to the About window after a few minutes, I can still see the <Restart to update Nightly> button, which means that after step 6 the download of the complete mar starts in the background anyway.

Comment 11

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Attachment: complete_mar_prompt.png

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We are not planning to spend any time on that feature addition for now. If this will change in the future lets reopen this bug.