1901954 - Beta versions prior to Firefox 127.0b8 fail to update to Beta 128.0b1

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect, P1)

Description

[Simona Badau, Desktop QA]

Attachment: maintenanceservice-2.log

Found in

• when testing updates for Firefox 128.0b1

Affected versions

• beta versions prior to Firefox 127.0b8

Tested platforms

• Affected platforms: Windows 10, Windows 11.
• Unaffected platforms: Ubuntu 22.04, macOS 13

Steps to reproduce

1. Install a beta version that is older than Firefox 127.0b8 (for eg Firefox 127.0b7)
2. Trigger an update on the default beta channel.

Expected result

• Firefox should be updated to Firefox 127 build 2 and then to Firefox 128.0b1.

Actual result

• After triggering the update the 'Firefox Update' dialog with the message 'Firefox is installing your updates and will start in a few moments' is displayed and stalls.
• After approx 20 minutes, the 'Firefox update' dialog is dismissed and Firefox is restarted to Firefox 127 build 2, but the update to 128.0b1 fails. - please see the attached maintenanceservice log.
• Some Firefox process remain open in the background, if I manually close those processes then the update to Firefox 128 beta is done.

Regression range

• All the beta versions prior to Firefox 127.0b8 seems to be affected

Additional notes

• The issue is reproducible only with the MSI and EXE installers.
• Update are properly done on ZIP versions.
• Updates from versions older than Firefox 127.0b7 to Firefox 128 beta fail on the second attempt. The first attempt is successful.

Comment 1

[Simona Badau, Desktop QA]

Attachment: last-update.log

Also attaching the updates logs from: C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\308046B0AF4A39CB

Comment 2

[Simona Badau, Desktop QA]

Attachment: last-update-elevated.log

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox127 (release), tracked for firefox128 (beta) and tracked for firefox129 (nightly). However, the bug still isn't assigned.

:Amir, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 4

[Simona Badau, Desktop QA]

Attachment: Screen recording with the Firefox Update dialog.mp4

Comment 5

[Simona Badau, Desktop QA]

I am also adding a link to a screen recording from after the Firefox update dialog was auto-dismissed. I waited for approximately 20 minutes, then Firefox restarted to version 127 RC2. Any attempt to update to Firefox 128.0b1 fails until I manually close some processes in the Task Manager. After that, the update to Firefox 128.0b1 completes successfully.

https://drive.google.com/file/d/1ZaN_-uvO9J1cv5_fXIr1ROnY-lGE6xBq/view

Comment 6

[Nick Alexander :nalexander [he/him]]

The last-update.log error is 58, SERVICE_UPDATE_STATUS_UNCHANGED (see https://searchfox.org/mozilla-central/rev/4582d908c17fbf7924f5699609fe4a12c28ddc4a/toolkit/mozapps/update/common/updatererrors.h#93).

2024-06-12 11:38:13+0300: Writing status to file: failed: 58

The last-elevated-update.log looks normal; the only thing that might be off is

2024-06-12 11:38:21+0300: calling QuitProgressUI
2024-06-12 11:38:21+0300: Running LaunchCallbackAndPostProcessApps
2024-06-12 11:38:21+0300: No callback arg. Skipping LaunchWinPostProcess and LaunchCallbackApp

That surprises me; I expected to run post-update. But this might be expected with an elevated install, I'm not 100% on these details. Finally, the maintenance service is clearly unhappy:

2024-06-12 10:36:00+0300: Checking for Maintenance Service registry. key: 'SOFTWARE\Mozilla\MaintenanceService\f9b87e891978e3145f0f8f9953eadc00'
2024-06-12 10:36:00+0300: updater.exe was compared successfully to the installation directory updater.exe.
2024-06-12 10:36:00+0300: The updater.exe application contains the Mozilla updater identity.
2024-06-12 10:36:00+0300: The file "C:\Program Files\Mozilla Firefox\updater.exe" is signed and the signature was verified.
2024-06-12 10:36:00+0300: Passed in path: 'C:\Program Files\Mozilla Firefox\updater.exe' (ignored); Install dir has: 'C:\Program Files\Mozilla Firefox\updater.exe'; Using this path for updating: 'C:\Program Files (x86)\Mozilla Maintenance Service\update\updater.exe'.
2024-06-12 10:36:00+0300: updater.exe was compared successfully to the installation directory updater.exe.
2024-06-12 10:36:00+0300: The updater.exe application contains the Mozilla updater identity.
2024-06-12 10:36:00+0300: The file "C:\Program Files (x86)\Mozilla Maintenance Service\update\updater.exe" is signed and the signature was verified.
2024-06-12 10:36:00+0300: Starting update process as the service in session 0.
2024-06-12 10:36:00+0300: Starting service with cmdline: "C:\Program Files (x86)\Mozilla Maintenance Service\update\updater.exe" C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\308046B0AF4A39CB\updates\0 "C:\Program Files\Mozilla Firefox" "C:\Program Files\Mozilla Firefox" 6276 "C:\Program Files\Mozilla Firefox" "C:\Program Files\Mozilla Firefox\firefox.exe"
2024-06-12 10:36:00+0300: Process was started... waiting on result.
*** Warning: Error running update process. Updating update.status  (0)***
*** Warning: Could not delete service updater path: 'C:\Program Files (x86)\Mozilla Maintenance Service\update\updater.exe'.***
2024-06-12 10:51:00+0300: Service command software-update complete.
2024-06-12 10:51:00+0300: service command MozillaMaintenance complete with result: Failure.

It's not clear to me why we would hang out for 15 minutes in that context.

Comment 7

[Robin Steuber (they/them) [:bytesized]]

(In reply to Nick Alexander :nalexander [he/him] from comment #6)

It's not clear to me why we would hang out for 15 minutes in that context.

Probably this, I would bet.

Comment 8

[Robin Steuber (they/them) [:bytesized]]

I marked all but the maintenance service log as obsolete because they aren't really going to help us solve this problem. The reason for this is that the logs do not all seem to be from the same update. Logically, the order of log messages should be

This in last-update.log:

2024-06-12 11:38:13+0300: Performing a staged update

This in maintenanceservice-2.log:

2024-06-12 10:36:00+0300: Process was started... waiting on result.

next, the entirety of last-update-elevated.log, including this line:

2024-06-12 11:38:14+0300: Going to update via this updater instance.

Then this in maintenanceservice-2.log:

2024-06-12 10:51:00+0300: Service command software-update complete.

But the timestamps are wrong. They go 11:38, 10:36, 11:38, 10:51. It seems pretty clear to me that the update logs and the maintenance service logs are the results of two different updates, so trying to analyze these logs together will probably not be productive. The maintenance service log does show the 15 minute hang happening, so I think it's worth keeping that one until we have a more complete set of logs to analyze.

It is possible that last-update-elevated.log just won't be created in this error case. I'm not positive that we successfully launch it at all. If it is being launched, it may still not be copying them from C:\Program Files (x86)\Mozilla Maintenance Service\UpdateLogs to the update directory properly.

---

I have so far been unable to reproduce this problem.

@sbadau - Is there any more information you can give about reproducing this? Are you able to reproduce this problem consistently? Could we get some logs that do not suffer from the problems described above (i.e. logs that are all describing the same update).

Comment 9

[Nick Alexander :nalexander [he/him]]

In Windows Sandbox, I can repro a different and possibly anticipated issue which might be also happening here. N.b.: Windows Sandbox does not require UAC elevation and thus is not the same as a regular environment.

1. Install Firefox 127.0b7 from https://archive.mozilla.org/pub/firefox/releases/127.0b7/win64/en-CA/Firefox%20Setup%20127.0b7.exe.
2. Launch Firefox; witness 127.0b7 in about:support.
3. In About > Help, wait for update to download. Witness pending version 127.0 in active-update.xml.
4. Witness successful application (2024-06-12 15:20:03-0700: Writing status to file: applied) in last-update.log.
5. Restart Firefox to apply update.
6. Witness 127.0 in about:support.
7. Witness successful update but failed maintenance service update in last-update.log:

2024-06-12 15:21:26-0700: Starting Service Update before launching callback app
*** Warning: Certificate did not match issuer or name.  (1168)***
*** Warning: Error on certificate check.  (1168)***
2024-06-12 15:21:26-0700: The file "C:\Program Files (x86)\Mozilla Maintenance Service\maintenanceservice_tmp.exe" is signed and the signature was verified.

That suggests that the Maintenance Service Installer keys didn't get revved as expected during the Authenticode key rollout.

bhearsum: jcristau: is this expected, and maybe wouldn't repro with 127.0b8?

I'll see if I can repro what QA witnesses outside of Windows Sandbox, which (again) is an irregular environment.

Comment 10

[Nick Alexander :nalexander [he/him]]

A little more on this: inspecting the code suggests these warnings might just be non-matching but known certificates. Totally possible.

However, I can verify that the MMS did not get updated: same version from Properties > Details, same hash on disk. That surprises me, although I suppose it's possible that the MMS wasn't revved between 127.0b7 and 127.0. I'd expect the signature, at least, to be different. I'll try to run that down now.

Comment 11

[Nick Alexander :nalexander [he/him]]

(In reply to Nick Alexander :nalexander [he/him] from comment #10)

A little more on this: inspecting the code suggests these warnings might just be non-matching but known certificates. Totally possible.

However, I can verify that the MMS did not get updated: same version from Properties > Details, same hash on disk. That surprises me, although I suppose it's possible that the MMS wasn't revved between 127.0b7 and 127.0. I'd expect the signature, at least, to be different. I'll try to run that down now.

127.0 and 127.0b7 do install different MMS EXEs, with different version numbers. This does suggest the MMS isn't getting updated in the way that I expect.

Comment 12

[Nick Alexander :nalexander [he/him]]

I can't repro anything outside of Windows Sandbox -- my update from 127.0b7 to 127.0 seems fine. However, my MMS version is 128.0* from some other installation -- I'll try again later after forcing the 127.0b7 version of the MMS.

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Nick Alexander :nalexander [he/him] from comment #9)

In Windows Sandbox, I can repro a different and possibly anticipated issue which might be also happening here. N.b.: Windows Sandbox does not require UAC elevation and thus is not the same as a regular environment.

1. Install Firefox 127.0b7 from https://archive.mozilla.org/pub/firefox/releases/127.0b7/win64/en-CA/Firefox%20Setup%20127.0b7.exe.
2. Launch Firefox; witness 127.0b7 in about:support.
3. In About > Help, wait for update to download. Witness pending version 127.0 in active-update.xml.
4. Witness successful application (2024-06-12 15:20:03-0700: Writing status to file: applied) in last-update.log.
5. Restart Firefox to apply update.
6. Witness 127.0 in about:support.
7. Witness successful update but failed maintenance service update in last-update.log:

2024-06-12 15:21:26-0700: Starting Service Update before launching callback app
*** Warning: Certificate did not match issuer or name.  (1168)***
*** Warning: Error on certificate check.  (1168)***
2024-06-12 15:21:26-0700: The file "C:\Program Files (x86)\Mozilla Maintenance Service\maintenanceservice_tmp.exe" is signed and the signature was verified.

That suggests that the Maintenance Service Installer keys didn't get revved as expected during the Authenticode key rollout.

I don't think this has anything to do with switching to the new cert - that cut over didn't happen on Beta until 128.0b1. It certainly seems related to some of the prep work and other patches done in advance of that, though.

I also have a slightly different interpretation of that log.

*** Warning: Error on certificate check. (1168)***

This is clearly coming from https://searchfox.org/mozilla-central/rev/346ac9335cbc33e3124a4541c938a0dc455e785e/toolkit/mozapps/update/common/registrycertificates.cpp#128

While this:

2024-06-12 15:21:26-0700: The file "C:\Program Files (x86)\Mozilla Maintenance Service\maintenanceservice_tmp.exe" is signed and the signature was verified.

Is clearly from https://searchfox.org/mozilla-central/rev/346ac9335cbc33e3124a4541c938a0dc455e785e/toolkit/mozapps/update/common/certificatecheck.cpp#230, via https://searchfox.org/mozilla-central/rev/346ac9335cbc33e3124a4541c938a0dc455e785e/toolkit/mozapps/update/common/registrycertificates.cpp#133 - which is only reachable if we get past the attribute check.

This suggests to me that the first check of the attributes fails, and then the second one succeeds. This is what I would expect to see if the maintenance service was signed with the second certificate listed in the registry. This is what we would expect to see for a fresh install of any 127.0 Beta, as the new registry entries landed in the 127.0 Nightly cycle. (Updates to 127.0 Betas on the other hand would only get them after 128.0b8, due to https://bugzilla.mozilla.org/show_bug.cgi?id=1896944.)

Of course, you've confirmed in a later comment that the maintenance service wasn't updated. I took a look at the version information for b7 and RC2, and the version certainly looks updated:
~/tmp/2024-06-12 ❯ peres -v beta7/core/maintenanceservice.exe
DEBUG: Length=20, String=LIMITEDACCESSFEATURE
DEBUG: Length=8, String=IDENTITY
File Version: 65263.1213.1.0
Product Version: 127.0.0.8913
~/tmp/2024-06-12 ❯ peres -v rc/core/maintenanceservice.exe
DEBUG: Length=20, String=LIMITEDACCESSFEATURE
DEBUG: Length=8, String=IDENTITY
File Version: 65263.1213.1.0
Product Version: 127.0.0.8923

bhearsum: jcristau: is this expected, and maybe wouldn't repro with 127.0b8?

If this is only happening on builds prior to 127.0b8 that's very notable, as those are the builds that are funneled through a watershed on 127.0 RC2. This watershed is to ensure that they pick up https://bugzilla.mozilla.org/show_bug.cgi?id=1896944, which will ensure they get the new maintenance service registry entries (the registry strings themselves were updated in https://bugzilla.mozilla.org/show_bug.cgi?id=1894689, which landed in the 127.0 Nightly cycle, but were not updated in earlier builds due to the other bug).

However, I don't really understand why 127.0b7 and earlier would be failing to update the maintenance service as part of the 127.0 RC2 update. That build is still signed with the old certificate. Inspecting https://archive.mozilla.org/pub/firefox/candidates/127.0-candidates/build2/win64/en-US/Firefox%20Setup%20127.0.exe I find:

        cert_info: 
          version: 2
          serialNumber: 16100418757095927544930732507868760829
          signature: 
            algorithm: sha256WithRSAEncryption (1.2.840.113549.1.1.11)
            parameter: NULL
          issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert SHA2 Assured ID Code Signing CA
          validity: 
            notBefore: Apr  9 00:00:00 2021 GMT
            notAfter: Jun 19 23:59:59 2024 GMT
          subject: C=US, ST=California, L=Mountain View, O=Mozilla Corporation, OU=Firefox Engineering Operations, CN=Mozilla Corporation

Which is quite clearly our old, about-to-expire, certificate, which lines up with my interpretation of the log above. I don't see any reason related to the signing certificate why the maintenance service is not updating.

I'm also a bit confused about how this ties into the updater hanging - I would assume that the updater would simply log an error or warning and move on rather than sit and spin, but I admit I haven't verified that assumption.

Comment 14

[[e-mail] (:bhearsum)]

This suggests to me that the first check of the attributes fails, and then the second one succeeds. This is what I would expect to see if the maintenance service was signed with the second certificate listed in the registry. This is what we would expect to see for a fresh install of any 127.0 Beta, as the new registry entries landed in the 127.0 Nightly cycle. (Updates to 127.0 Betas on the other hand would only get them after 128.0b8, due to https://bugzilla.mozilla.org/show_bug.cgi?id=1896944.)

One possibility that comes to mind is that falling back to the second entry in the registry is broken somehow. However, this would only make sense if 127.0b8 -> 127.0 RC2 updates were also failing to update the maintenance service, as the fresh install state of the registry entries for all 127.0 Betas is the same. (This could probably be easily tested by installing a 127.0 Beta, swapping the values of the entries to put the old certificate info first, and seeing if the maintenance service updates correctly.)

I tried to test this myself, but I was unable to reproduce in a regular Windows 11 VM even after ensuring the installed MMS was 127.0b7.

Comment 15

[Simona Badau, Desktop QA]

Attachment: logs.zip

(In reply to Robin Steuber (they/them) [:bytesized] from comment #8)

@sbadau - Is there any more information you can give about reproducing this? Are you able to reproduce this problem consistently? Could we get some logs that do not suffer from the problems described above (i.e. logs that are all describing the same update).

I reproduce this issue every time I try to update Beta older than version 127.0b8.

Precondition: Uninstalled all Firefox versions from my Windows 10 x64 machine, and made sure that the Mozilla Maintenance Service folder is removed during the uninstallation process.

Steps to reproduce:

1. Install Firefox version 127.0b5 from here
2. Open Firefox with a new profile.
3. Select Open application menu -> Help -> About Firefox.
4. Wait for the update to download, then click on the Restart to update Firefox button.
5. Wait until the Firefox update dialog auto-dismisses and select Yes when the UAC prompt requests permissions.
6. After Firefox restarts, go back to Help -> About Firefox to continue the update process.
7. Click on Restart to update Firefox again.
8. When UAC prompts, select Yes.
9. Check the Firefox version.

Expected results: Firefox should update to version 128.0b1.
Actual results: Firefox remains at version 127RC2.

I'm attaching the logs folder from C:\Program Files (x86)\Mozilla Maintenance Service.

Comment 16

[Simona Badau, Desktop QA]

Attachment: updates.zip

I'm also attaching the updates logs from C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\308046B0AF4A39CB

Another thing worth mentioning is that from our investigations we noticed that the issue is reproducible only from the second attempt. This means that the first time we update a beta version older than 127.0b8, it works, but the second time it doesn't.

Comment 17

[Wayne Mery (:wsmwk)]

Should we expect Thunderbird to also be affected?

Comment 18

[Simona Badau, Desktop QA]

I reproduced this issue also on Nightly when updating from older builds.

These are the steps to reproduce:

1. Install EXE or MSI Nightly 127.0a1 from here
2. Select Open application menu -> Help -> About Nightly.
3. Wait for the update to download, then click on the Restart to update Nightly button.
4. Wait until the Nightly update dialog auto-dismisses and select Yes when the UAC prompt requests permissions.
5. After Nightly restarts, check the version.
   ---->Expected: Nightly is updated to version 128.0a1 (Build ID: 20240527092611)
6. Go back to Help -> About Nightly to continue the update process.
7. Click on Restart to continue with the Nightly update.
8. When UAC prompts, select Yes.
9. Check the Nightly version.

Expected results: Nightly should update to version 129.0b1
Actual results: Nightly remains at version 128.0a1 (Build ID: 20240527092611)

Please note that I ran this scenario twice to be able to reproduce it.

Comment 19

[Wayne Mery (:wsmwk)]

vlad, can you reproduce with Thunderbird?

Comment 20

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello,

This issue does not seem to affect Thunderbird.

I have tried the scenarios from Description, Comment 15 and Comment 18 using .zip, .msi and .exe installers and the update is always successfully done.

I have tried updating from 127.0b1 through 127.0b5 , as well as from 126.0b1 through 126.0b3 and Nightly builds, and the issue is not present for Windows 11 and Windows 10.

Comment 21

[Johan Lorenzo [:jlorenzo]]

Bumping the severity to S1 until we have some clarity on the impact of this bug. We may need to publish a fix before Jun 19 23:59:59 2024 GMT, which leaves little leeway. Happy to downgrade it back to S2 if this bug turns out to affect less than 25% of our user base.

Comment 22

[Robin Steuber (they/them) [:bytesized]]

Ah! Yes, removing the maintenance service does allow me to reproduce the problem.

@sbadau I do want to make sure, however, that the problem that I've identified is the same problem as the one QA is experiencing. Could you reproduce the problem again and, while the updater is hanging for 15 minutes, could you use Process Explorer to check if you see a Firefox process with the command line: firefox.exe --backgroundtask defaultagent unregister-task <install hash>?

---

Let me explain a bit further what I believe is going on here. Until recently, elevated Post Update has been broken. We simply never ran Post Update with elevation. One of the things that we do in PostUpdate is to update the Scheduled Task registration of the WDBA. However, in the year during which this was broken and didn't run, we started converting the WDBA from being a standalone binary to being a background task. And the background task seems to be hanging.

There is kind of a lot to unpack here. These questions immediately come to my mind:

1. Is it possible to reproduce this problem on Nightly? I don't see any compelling reason why this should be a beta-specific problem.
2. Is the reason that this isn't more widespread and that I couldn't reproduce without uninstalling the Maintenance Service that the updater is not successfully updating the Maintenance Service?
3. What is the role of this version of the Maintenance Service in reproducing this? I was originally thinking that it made sense because that's where we fixed Bug 1896944. But that's not true; that fix was in NSIS, not in the Maintenance Service.
4. If we really need to do this bit of registration with elevation, it's not really appropriate to do it from a background task. Background tasks are Firefox processes which try to de-elevate their privileges by default. Potentially we could bypass this, but routinely running Firefox as SYSTEM is a really bad idea.
5. Why is the background task hanging?

It's very likely that we could fix things for the time being by just removing that Background Task invocation. Ideally we really should be doing something to potentially fix that task registration, but since we apparently haven't been doing it for the last year and no one even noticed, it might not be the end of the world if we kept not doing it for a while longer.

I'm going to try to find answers for questions 1, 2, 3, and 5 and will report back with my findings.

Comment 23

[Johan Lorenzo [:jlorenzo]]

1. Is it possible to reproduce this problem on Nightly?

Yes. QA reproduced the bug on beta between 124 and 127 and on nightly between 126 and 129[1].

[1] https://mozilla.slack.com/archives/C071RPEUTEH/p1718292079758599?thread_ts=1718291607.431009&cid=C071RPEUTEH

Comment 24

[Jens Stutte [:jstutte]]

5. Why is the background task hanging?

In general the updater hangs quite often, see bug 1741675. IIRC it blocks on the instantiation of a COM object most of the times. I have no idea if this is related here, of course.

Comment 25

[Robin Steuber (they/them) [:bytesized]]

(In reply to Jens Stutte [:jstutte] from comment #24)

5. Why is the background task hanging?

In general the updater hangs quite often, see bug 1741675. IIRC it blocks on the instantiation of a COM object most of the times. I have no idea if this is related here, of course.

That seems unrelated to me.

Comment 26

[Nick Alexander :nalexander [he/him]]

sbadau: I have a theory about what's happening. Could you reproduce in the way that you usually do, and -- when the installer is hung -- could you use procexp to take a screenshot of the maintenanceservice.exe process tree? I expect to see:

maintenanceservice.exe
  uninstall\helper.exe /PostUpdate
    default-browser-agent.exe unregister-task ...
      firefox.exe --backgroundtask defaultagent unregister-task ...
        firefox.exe --backgroundtask defaultagent unregister-task ...

After this, could you then:

1. Return to an earlier Beta (or Nightly) version.
2. Running as Administrator, remove the directory C:\Windows\System32\config\systemprofile\AppData\Roaming\Mozilla entirely. You might even need to be SYSTEM to do this; use psexec if you need to.
3. Try to reproduce going forward only, i.e., without downgrading.
4. Report if you can reproduce. I expect you will not reproduce, because deleting the directory will avoid a profile downgrade issue.

Thanks for all your effort tracking this down!

Comment 27

[Johan Lorenzo [:jlorenzo]]

Per :nalexander on Slack:

We’ve isolated at least one issue: the fix for Bug 1896944 starts running PostUpdate elevated, but that can run firefox.exe --backgroundtask defaultagent ... and that can hang. This isn’t strictly needed and we expect that removing this affordance will address the issue.

Marking bug 1896944 as a regressing bug.

Comment 28

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1896944

Comment 29

[Simona Badau, Desktop QA]

Attachment: 2024-06-14_12h27_38.png

(In reply to Nick Alexander :nalexander [he/him] from comment #26)

sbadau: I have a theory about what's happening. Could you reproduce in the way that you usually do, and -- when the installer is hung -- could you use procexp to take a screenshot of the maintenanceservice.exe process tree? I expect to see:

maintenanceservice.exe
  uninstall\helper.exe /PostUpdate
    default-browser-agent.exe unregister-task ...
      firefox.exe --backgroundtask defaultagent unregister-task ...
        firefox.exe --backgroundtask defaultagent unregister-task ...

Yes, please see the attached screenshot.

After this, could you then:

1. Return to an earlier Beta (or Nightly) version.
2. Running as Administrator, remove the directory C:\Windows\System32\config\systemprofile\AppData\Roaming\Mozilla entirely. You might even need to be SYSTEM to do this; use psexec if you need to.
3. Try to reproduce going forward only, i.e., without downgrading.
4. Report if you can reproduce. I expect you will not reproduce, because deleting the directory will avoid a profile downgrade issue.

This is what I did:

1. Installed Firefox 127.0b5.
2. Successfully deleted the Mozilla directory from C:\Windows\System32\config\systemprofile\AppData\Roaming
3. Opened Firefox 127.0b5 -> Updated to 127RC2 -> successfully updated to 128.0b2. - with no hangs.

Comment 30

[Brindusa Tot, Desktop QA]

Tested more today on Windows 10 (physical and VM machine) and when user get stuck with Firefox 127.0, after a windows restart and clicking twice OK on the "newer version profile" error, the updater is back in working state and Firefox is successful updated to 128.0b2. Here is a link to a screen recording after user is updated to 127:

https://drive.google.com/file/d/1NKbpPuVc3Or-NSr3I9VlPDA5IjkwmsxW/view?usp=drive_link

Comment 31

[Johan Lorenzo [:jlorenzo]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #21)

Bumping the severity to S1 until we have some clarity on the impact of this bug. We may need to publish a fix before Jun 19 23:59:59 2024 GMT, which leaves little leeway. Happy to downgrade it back to S2 if this bug turns out to affect less than 25% of our user base.

Status update: This incident is also being downgraded to S3 because test and data have shown it only impacts a small amount of users. Moreover, there's a simple workaround which is to reboot Windows (see comment 30).

Comment 32

[Robin Steuber (they/them) [:bytesized]]

Attachment: Bug 1901954 - Don't invoke any background tasks from the MMS r=nalexander!

Comment 33

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/71d0793fa963
Don't invoke any background tasks from the MMS r=nalexander

Comment 34

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/71d0793fa963

Comment 35

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:bytesized, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox128 to wontfix.

For more information, please visit BugBot documentation.

Comment 36

[Simona Badau, Desktop QA]

After the fix we can no longer reproduce this issue on the latest Nightly on Windows 10 and Windows 11.

These are the steps we took:

1. Installed Nightly 127 from April 23 (Build ID:20240423094521 ) / Nightly 128 from May 20 (Build ID: 20240520093945)
2. Updated and through the watershed we reached Nightly 128 from May 27 (Build ID: 20240527092611).
3. Updated again and reached the latest Nightly 129 from June 16 (Buid ID: 20240616215431).

We then uninstalled Nightly and repeated these steps several times with both versions 127 and 128 mentioned above - and each time we reached the latest Nightly 129 without hangs.

@jlorenzo, are there any other ways that we ca use to verify this fix?

Comment 37

[Robin Steuber (they/them) [:bytesized]]

Attachment: beta.patch

Approval Request Comment
[Feature/Bug causing the regression]:

Bug 1896944

[User impact if declined]:

Downgrading Firefox could break application update.

[Is this code covered by automated tests?]:

Unfortunately not. We still have no test harness that covers NSIS code.

[Has the fix been verified in Nightly?]:

Yes.

[Needs manual test from QE? If yes, steps to reproduce]:

Yes. STR in Comment 15.

[List of other uplifts needed for the feature/fix]:

None.

[Is the change risky?]:

Low Risk.

[Why is the change risky/not risky?]:

We simply do not call the problematic background task. We do need to come up with a fix rather than just not calling this (Bug 1902719). But we hadn't been successfully calling this for a year until Bug 1896944 was fixed, so ceasing to call it now is unlikely to cause immediate problems.

[String changes made/needed]:

None

Comment 38

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9407962 [details] [diff] [review]
beta.patch

Approved for 128.0b5.

Comment 39

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/b157b36d8a7e

Comment 40

[Johan Lorenzo [:jlorenzo]]

@jlorenzo, are there any other ways that we ca use to verify this fix?

I'm sorry, I'm not knowledgeable enough about this. As discussed on Slack I defer to :bytesized and :nalexander. Based on the reactmoji, I understand :bytesized also defers to :nalexander. NI'ing Nick.

Comment 41

[[e-mail] (:bhearsum)]

(In reply to Johan Lorenzo [:jlorenzo] from comment #40)

@jlorenzo, are there any other ways that we ca use to verify this fix?

I'm sorry, I'm not knowledgeable enough about this. As discussed on Slack I defer to :bytesized and :nalexander. Based on the reactmoji, I understand :bytesized also defers to :nalexander. NI'ing Nick.

I think we could probably set up an update channel that would route you through an affected version to be able to reproduce still. What exactly are you looking to verify that? That someone who hits the reported issue can update later? Please poke me on Slack and we can try to sort this out.

Comment 42

[Nick Alexander :nalexander [he/him]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #40)

@jlorenzo, are there any other ways that we ca use to verify this fix?

I'm sorry, I'm not knowledgeable enough about this. As discussed on Slack I defer to :bytesized and :nalexander. Based on the reactmoji, I understand :bytesized also defers to :nalexander. NI'ing Nick.

:bhearsum chimed in but I'll add that I can't think of any additional testing of much value.

Comment 43

[Simona Badau, Desktop QA]

We are unable to reproduce this issue after updating to the latest Firefox 128.0b5 or Firefox Developer Edition 128.0b5, following the steps from Comment 15.

Testing was conducted on both Windows 10 and Windows 11, involving downgrading and upgrading of Firefox when using the beta version installers (.exe, .msi, .zip, and .exe-stub). For further details, please refer to the document.

Please let us know if there are additional scenarios or methods to verify this.