811102 - frameworker reload failure

Status: RESOLVED FIXED

(Firefox Graveyard :: SocialAPI, defect, P1)

Description

[Shane Caraveo (:mixedpuppy)]

Facebook reported that the worker reload seemed to be happening twice, and that the second load caused the ports to fail.

A quick look verifies that the first reload does happen twice.  It then seems the second reload happens once, and after that reload no longer works correctly.  I have not looked into the ports having an issue yet.

Comment 1

[Shane Caraveo (:mixedpuppy)]

The unload handler is removing the worker from the global list of workers, then closing all ports.  This caused two issues:

- The next call to getFrameWorkerHandle created a new frameworker instance, we would end up with multiple frameworkers per provider
- the closed ports cannot be reused (portid is set to null, never reset, then you cannot postmessage across the port)

The reason for the change over number of reloads seems to be that the worker can no longer send a reload message over the ports.

Comment 2

[Shane Caraveo (:mixedpuppy)]

Attachment: reload fix

The first part of the patch is code removal.  

Unfortunately the diff doesn't show enough context, so you need to look at the unload handler to see that we handle what I am removing later in the handler.

The second part of the patch causes the unload handler to run during termination.  If we do not close ports here, we run into the cause of our "re activation" bug because the worker was not getting the port-close message on termination.  We could live without the second part for 17 of the patch if we must.

I think that the code I removed may have been in the terminate function prior to adding the unload handler, but I didn't dig into history to verify that.

Comment 3

[Shane Caraveo (:mixedpuppy)]

Attachment: reload fix

better fix after better understanding of what is happening there.

Comment 4

[:Gavin Sharp [email: [e-mail]]]

Attachment: patch per IRC discussion

This doesn't fix the "close ports on terminate()" issue, but we can followup with that.

Comment 5

[:Gavin Sharp [email: [e-mail]]]

Comment on attachment 680826 [details] [diff] [review]
reload fix

I also omitted the "ports = []" change in my patch, since I think that line should just be removed (reload() already clears .ports). We need more tests here.

Comment 6

[Shane Caraveo (:mixedpuppy)]

bug 811134 created for the terminate fix

Comment 7

[Shane Caraveo (:mixedpuppy)]

https://tbpl.mozilla.org/?tree=Try&rev=8fcb227ed3ee

Comment 8

[:Gavin Sharp [email: [e-mail]]]

Attachment: updated patch with test

Comment 9

[:Gavin Sharp [email: [e-mail]]]

Attachment: updated patch with additional test

This other test is from Mark, for the ".loaded" change.

Comment 10

[:Gavin Sharp [email: [e-mail]]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/85b3e7667edc

Comment 11

[:Gavin Sharp [email: [e-mail]]]

Comment on attachment 680898 [details] [diff] [review]
updated patch with additional test

[Triage Comment]
Try looks good, and Shane got confirmation from Facebook engineers that this addresses their problem, so I'm going to land this on beta.

Comment 12

[:Gavin Sharp [email: [e-mail]]]

https://hg.mozilla.org/releases/mozilla-beta/rev/731c85bd8e9e
https://hg.mozilla.org/releases/mozilla-aurora/rev/ea619b3b2d3c

Comment 13

[:Gavin Sharp [email: [e-mail]]]

followup test fix:
https://hg.mozilla.org/releases/mozilla-beta/rev/a5ec489a0ab9
https://hg.mozilla.org/releases/mozilla-aurora/rev/3b91e7e2c30a
https://hg.mozilla.org/integration/mozilla-inbound/rev/2e23d71714b4
https://hg.mozilla.org/integration/mozilla-inbound/rev/cab9c764a898

Comment 14

[:Gavin Sharp [email: [e-mail]]]

Attachment: followup leak fix #2

First followup leak fix wasn't enough. Removing the deletion of the entry in the workerCache from the unload handler exposed a previously-existing problem: we were leaking the workers from the tests for bug 804910 until shutdown via the workerCache. This didn't come up until now because that line in the unload handler would clear the reference during shutdown, when the hidden window was unloaded.

Comment 15

[:Gavin Sharp [email: [e-mail]]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a06a34a84955
https://hg.mozilla.org/releases/mozilla-aurora/rev/56e3981e015f
https://hg.mozilla.org/releases/mozilla-beta/rev/be03b1063c9a

Comment 16

[:Gavin Sharp [email: [e-mail]]]

Not my night tonight:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a1e3296bcd21
https://hg.mozilla.org/releases/mozilla-aurora/rev/0fd81b763e50
https://hg.mozilla.org/releases/mozilla-beta/rev/9f4f7fb0216d

Comment 17

[:Gavin Sharp [email: [e-mail]]]

(thankfully all of the followup landings are test-only issues - no need to worry about them for b6)

Comment 18

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/85b3e7667edc
https://hg.mozilla.org/mozilla-central/rev/2e23d71714b4
https://hg.mozilla.org/mozilla-central/rev/cab9c764a898
https://hg.mozilla.org/mozilla-central/rev/a06a34a84955
https://hg.mozilla.org/mozilla-central/rev/a1e3296bcd21

Comment 19

[u279076]

Is there a way to test this? Any exploratory testing QA can do in Beta 6 to either verify this is fixed or check for potential regressions?

Comment 20

[Mark Hammond [:markh] [:mhammond]]

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #19)
> Is there a way to test this? Any exploratory testing QA can do in Beta 6 to
> either verify this is fixed or check for potential regressions?

Not really - the "reload" feature is not exposed via the UI (ie, there is nothing a user can do to invoke a reload).  The tests should now reliably cover this though.

Comment 21

[u279076]

Okay, thanks Mark. Flagging this qa- for verification.