862747 - Firefox not responding error while updating localized build

Status: RESOLVED WORKSFORME

(Toolkit :: Application Update, defect, P1)

Description

[mario garbi]

Attachment: Screenshot of Not Responding Firefox during update

We encountered this issue while updating Firefox localized release builds using CI. The update froze and download stopped while the Firefox process being tagged as "Not responding" in Task Manager as indicated in the screenshots.

We caught this while investigating bug 803489

Firefox Dump download link - 170Mb 
https://dl.dropboxusercontent.com/u/37788888/Dump/firefox.DMP

Comment 1

[mario garbi]

Attachment: Task Manager Screen Shot

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Before someone asks I was also able to see that on a 32bit version of Windows 8 by running the updates manually. So this not depends on the Mozmill CI.

Comment 3

[Robert Strong (they/them - no direct email)]

Brian, could you take a look at this?

Comment 4

[Brian R. Bondy [:bbondy]]

Does this happen on every update attempt?  My first thought is some kind of unicode path problem.

Comment 5

[Brian R. Bondy [:bbondy]]

Also could you please attach a log?  (First set app.update.log to true)

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

When I was running several updates on another win8 machine I have seen this on and off. It was not always reproducible.

Brian, the log you can find at the following URL after you VPN'ed into MPT:
http://mm-ci-master.qa.scl3.mozilla.com:8080/job/ondemand_update/7421/console

If you don't have access to don't worry, I will upload it as attachment to this bug. Please keep in mind that this log also contains the test output from Mozmill.

Comment 7

[Brian R. Bondy [:bbondy]]

I don't have access, could you upload to this bug?

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: update log

Comment 9

[Brian R. Bondy [:bbondy]]

The original problem reported in Comment 0 seems to be a hang when downloading the update.

What the log shows is that the update is donwloaded and applied fine but then there is some error in test3.js.  

> Timeout: bridge.execFunction("66004d91-a695-11e2-8963-005056bb7a83", 
> bridge.registry["{6914a0ca-1258-4c62-a001-3a9bdca58261}"]["runTestFile"], 
> ["c:\\users\\mozauto\\appdata\\local\\temp\\tmp8cjm7a.mozmill-tests\\tests
> \\update\\testFallbackUpdate\\test3.js"])
>
> TEST-UNEXPECTED-FAIL | Disconnect Error: Application unexpectedly 
> closed
> INFO Passed: 5
> INFO Failed: 1
> INFO Skipped: 0


whimboo: could you describe the problem that you seen on and off? The log here doesn't seem to match what I think you're reporting.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I had different manifestations of this issue. It happened during downloading but also while the update got applied in the background. As the attached screenshot shows the Firefox process no longer responded, and needed to be killed.

The failure you can see in our test results is simply that effect. We have a global timeout of 60 seconds in which the application has to respond (like a heartbeat). If that is not happening, which is the case here, Mozmill kills the Firefox process and shows that application disconnect error. 

Brian, what would be helpful for you to get? What could Mario do to give you more information?

Comment 11

[Brian R. Bondy [:bbondy]]

It sounds like there was a timeout in the mozmmill script but I don't think it is from a hang while in the middle of downloading. It could just be some other random shutdown hang unrelated to updates.

> It happened during downloading but also while the update got 
> applied in the background.

I don't understand this.  The process is first to download, and then after that is done to update. I don't understand how both can happen at the same time.

> We have a global timeout of 60 seconds in which the application has to 
> respond (like a heartbeat).

Do you know how the implementation of this heartbeat works more precisely?

> Firefox Dump download link - 170Mb 
> https://dl.dropboxusercontent.com/u/37788888/Dump/firefox.DMP

Which Firefox was this dump created with?

> Brian, what would be helpful for you to get? What could Mario do to 
> give you more information?

Info on the build used and possibly additional logs.  Also if you have a consistent way to reproduce this, or way that I can run the mozmill steps to reproduce even 1 out of X times, then that would be helpful.

Comment 12

[mario garbi]

Attachment: Ondemand Config file

I will try to provide more info the best I can:
Build used - Firefox 19.0 (19.0, zh-TW, 20130215130331)
Platform - Windows NT 6.2.9200 (x86_64)

We managed to reproduce this issue every ondemand update testrun on CI using the config file attached (might need updating of the target build id with relevant one when you run it) 

When we collected the information Firefox was not responding and I'm not sure how that could have been a result of Mozmill testruns. 

Please let me know if there are any specific informations I can provide.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mario, please run those update tests manually with the affected build. That way we can ensure that it's a standalone issue. I also cannot see any relation to Mozmill here given that we never have seen this before and no other tests are affected. But as said, lets get Mozmill out of the scope here once you got it by running manually. Once the application is in such a state let me know and I can have a look at the machine. Keep in mind to enable update logs before.

Comment 14

[mario garbi]

Attachment: Update logs from Firefox zh-TW 19.0.2 build

I managed to reproduce it by running the tests manually outside of jenkins and we still reproduced the issue 8/10 times. I copied the update logs from AppData/local/Mozilla/firefox and attached them.

Firefox became unresponsive during download and had to be closed by End Task. It had a Not Reponding message as before.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to mario garbi from comment #14)
> Created attachment 739489 [details]
> Update logs from Firefox zh-TW 19.0.2 build

Those logs show nothing. The update process was successful. 

> I managed to reproduce it by running the tests manually outside of jenkins
> and we still reproduced the issue 8/10 times. I copied the update logs from

Not sure what manual means here, but have you ran those updates fully by hand? If not, that was something I told yesterday in our meeting. Once the browser is in such a state please let me know and I can collect all the necessary information.

Comment 16

[mario garbi]

I've tried to reproduce it manually on the machine that failed by using these steps:

1.Install build of firefox using either custom location, default location or using the automated script install in temp folder.
2.Opened Firefox and pressed Ctrl-Shift-J to bring up the Error Log
3.Called the update wizard using:
" top.opener.Components.classes["@mozilla.org/updates/update-prompt;1"].createInstance(top.opener.Components.interfaces.nsIUpdatePrompt).checkForUpdates() "
4.Updated firefox using the update wizard window.

Expected results:
-Firefox Not Reponding while downloading the update or right after

Actual results:
-The update did not hanged and Firefox was updated without becoming unresponsive

After each update I've used the control panel to uninstall Firefox.

Not sure if I missed something in the manual testing of this issue, that is why I presented the steps I've took.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I will try to get back to this then when I have the time. It may take some days.

Comment 18

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This blocks a P1 so raising priority and setting assignee to Henrik according to his comment 17

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Dave Hunt (:davehunt) from comment #18)
> This blocks a P1 so raising priority and setting assignee to Henrik
> according to his comment 17

No, this is an application updater bug I'm not going to work on. This bug has to be fixed by a Firefox developer. I'm happy to be the QA contact and further investigate the problem.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Could all be related to the problems we have seen with bug 918676.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mario, could you please try if you can still reproduce this problem? If not I would tend to say we can close it as fixed by bug 919676.

Comment 22

[mario garbi]

In order to attempt to reproduce it I'd need to take a CI machine offline and they have been running jobs today. I will take one offline the first thing in the morning tomorrow and report back.

Comment 23

[mario garbi]

I tried a couple of times to reproduce the bug using the steps in comment 16 on mm-win-8-64-2 with Firefox 19.0.2 zh-TW and the updates were fine every time.

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+1]]

You could have closed the bug as WFM. Doing that now. If we can reproduce it again we will reopen the bug.