725568 - MozillaMaintenance service can crash after applying updates

Status: VERIFIED FIXED

(Toolkit :: Application Update, defect)

Description

[Gian-Carlo Pascutto [:gcp]]

I installed a Nightly on this machine, which already had an Aurora build on it that I'm using. After some very brief usage, I uninstalled the Nightly.

Every time I start Firefox now, I get a "Nightly: This application has stopped responding and has been closed". Checking the event log shows:

Faulting application maintenanceservice.exe, version 13.0.0.4419, time stamp 0x4f2fcdba, faulting module ntdll.dll, version 6.0.6002.18541, time stamp 0x4ec3e39f, exception code 0xc0000374, fault offset 0x000abc4f, process id 0xc08, application start time 0x01cce6efe87d2a61.

This means that:

1) Uninstalling Nightly failed because it left behind this Mozilla Maintenance Service.
2) This leftover service is launching with Firefox Aurora and crashing.

Comment 1

[Robert Strong (they/them - no direct email)]

Bug 720521 is for uninstalling the maintenance service and you likely don't have the fix from that bug. Since there are two different problems reported with one of them likely fixed I am removing the not uninstalled issue from this bug. If it is still present a new bug would need to be submitted anyways.

Comment 2

[Gian-Carlo Pascutto [:gcp]]

I'll uninstall the Maintenance service, reinstall Nightly, and see if I get a crash when launching Aurora.

Comment 3

[Gian-Carlo Pascutto [:gcp]]

No crash in that use case.

I also added a dummy entry to HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService, uninstalled Nightly, verified that the Maintenance Service stays around, and then launched Aurora. No crash. We're good.

Comment 4

[Gian-Carlo Pascutto [:gcp]]

Attachment: WER crash report

Comment 5

[Gian-Carlo Pascutto [:gcp]]

It did happen again. I have Nightly installed (from yesterday), and with it MaintenanceService. I started Aurora, and got an error "Nightly has caused an error and had to be closed".

Comment 6

[Brian R. Bondy [:bbondy]]

For the most recent time you reproduced, is it happening each time you open? Or did it just happen once?

Comment 7

[Gian-Carlo Pascutto [:gcp]]

So far only seems to have happened once. I'm 80% sure this was just after updating Aurora.

Comment 8

[Brian R. Bondy [:bbondy]]

I'll add crash reporting this week and send you a new maintenancesevice.exe binary.  Likely it will just put the crash report to your hard drive and you can send me that.

Comment 9

[Gian-Carlo Pascutto [:gcp]]

Okay. If the crash is reproducible on the next update, is there anything I can do to help you already? This Windows box has development tools on it (but I didn't get offered a chance to jump in the debugger when that service went down).

Comment 10

[Brian R. Bondy [:bbondy]]

Let's see if we can reproduce this consistently.  
When you go to the About dialog, and it says there is an update available... 

Before applying the update, could you go to: 
C:\Users\<username>\AppData\Local\Mozilla\Firefox and create a zip of the "Nightly" 
folder?

Then hopefully when you apply the update it'll crash on the restart. 
Then you should be able to unzip that folder again to reproduce it again.

If you can reproduce it consistently that way then perhaps we could do a gotomeeting, or logmein, or I could just send you some binaries with some logged output that would tell me where it crashes.

Comment 11

[Brian R. Bondy [:bbondy]]

Also if you have no update staged, one thing you can try quickly is to open up a command prompt and type:
net start MozillaMaintenance.

If it says "The process terminated unexpectedly then it is crashes really early on".
If it says 
"The Mozilla minatenance Service service is starting.
The Mozilla Maintenance service could not be started.
The service did not report an error."

Then that is normal and the crash doesn't happen that early on.

Comment 12

[Gian-Carlo Pascutto [:gcp]]

Attachment: Contents of Programdata\Mozilla\logs

Comment 13

[Brian R. Bondy [:bbondy]]

SO from the logs I can see that the update completes successfully but it crashes after that. 

(snip)...
> Process was started... waiting on result.
> Process finished with return code 0.
> updater.exe returned status: succeeded
> 
> Launching post update process as the service in session 0.
> updater.exe was launched and run successfully!


But you should have a couple extra lines at the end like this:

(snip)...
> Process was started... waiting on result.
> Process finished with return code 0.
> updater.exe returned status: succeeded
> 
> Launching post update process as the service in session 0.
> updater.exe was launched and run successfully!
> Service command software-update complete.
> service command MozillaMaintenance complete with result: Success.

Comment 14

[Brian R. Bondy [:bbondy]]

Attachment: Patch v1.

I caught up with gcp on IRC and added logging between each line after updates. He tested it and reported back the results.  The crash happened at the very end when trying to free the buffer in this patch.

This buffer used to be freed because we allocated it on the heap after parsing the command line arguments from the work item file.  But later we stopped using work item files and just used the argc and argv directly, so it should no longer be freed.

Comment 15

[Brian R. Bondy [:bbondy]]

gcp confirmed on irc that the fixed build no longer crashes by manually staging an update first and doing the reproducible steps.

Comment 16

[Brian R. Bondy [:bbondy]]

Thanks for the quick review, pushed to inbound:
http://hg.mozilla.org/integration/mozilla-inbound/rev/7e3d5c2879a8

Comment 17

[Marco Bonardo [:mak]]

https://hg.mozilla.org/mozilla-central/rev/7e3d5c2879a8

Comment 18

[Brian R. Bondy [:bbondy]]

http://hg.mozilla.org/releases/mozilla-aurora/rev/18840d884c97

Comment 19

[u279076]

Gian-Carlo, can you verify if this works for you now in Firefox 12b3 and Firefox 13.0a2?

Comment 20

[Gian-Carlo Pascutto [:gcp]]

No more crashes with Aurora. Beta doesn't seem to crash either.

Comment 21

[u279076]

Marking verified based on comment 20 -- thanks Gian-Carlo