1479209 - Frequently getting "You're offline" when performing Google searches

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Marcia Knous [:marcia]]

Attachment: Screenshot of issue on Android P

While using the latest nightly on my Pixel 2 running Android P (now Friday's build), I have been received the attached screenshot.

STR:
1. Type a Google search in the URL bar or in Google search bar on a page
2. Receive the attached screenshot

I am definitely not offline, as I can surf websites using other browsers as well as loading other websites on the same nightly. It seems I only get the message when performing a Google search for now.

Comment 1

[Marcia Knous [:marcia]]

Still getting this in the latest nightly. Will look in the console for errors.

Comment 3

[Dennis Schubert [:denschub]]

We're seeing lots of reports here on webcompat.com, but sadly, I am unable to reproduce... For context: This is probably related to an experiment we're running right now (see bug 1453691) where we spoof as Chrome in Fennec Nightly to discover breakage with Google's Tier 1 experience.

I'd like to debug this, but as I am not able to reproduce, this is hard... Marcia, can you also reproduce after clearing caches and cookies? Can you try in a private tab? Do you see any console errors?

Tom, do you have any idea what may cause this?

Comment 4

[Dennis Schubert [:denschub]]

See previous comment.

Comment 5

[Thomas Wisniewski [:twisniewski]]

I wonder if it's related to any ServiceWorkers... some users of my Google Search Fixer addon have had weird issues due to Google's ServiceWorker having webcompat issues, but they were not related to them "being offline".

However, I don't see any ServiceWorkers in about:serviceworkers, and this issue isn't reproducing for me. Perhaps there's some A/B test going on at Google, and others do have a ServiceWorker installed that's causing issues?

Comment 6

[Mike Taylor [:miketaylr]]

(In reply to Dennis Schubert [:denschub] from comment #3)
> Marcia, can you also reproduce after clearing caches and cookies? Can you
> try in a private tab? Do you see any console errors?

Also, marcia, do you see any service workers related to google.com in about:serviceworkers, and if so, take a screenshot?

Comment 7

[Mike Taylor [:miketaylr]]

Also, do you have any addons installed in Fennec Nightly?

Comment 8

[Pascal Chevrel:pascalc]

Tracking for 63

Comment 9

[Mike Taylor [:miketaylr]]

(In reply to Pascal Chevrel:pascalc from comment #8)
> Tracking for 63

Note: this is a Nightly-only experiment.

Comment 10

[Pascal Chevrel:pascalc]

(In reply to Mike Taylor [:miketaylr] (62 Regression Engineering Owner) from comment #9)
> (In reply to Pascal Chevrel:pascalc from comment #8)
> > Tracking for 63
> 
> Note: this is a Nightly-only experiment.

Are you sure that the experiment is the cause of the bug?

Comment 11

[Mike Taylor [:miketaylr]]

(In reply to Pascal Chevrel:pascalc from comment #10)
> (In reply to Mike Taylor [:miketaylr] (62 Regression Engineering Owner) from
> comment #9)
> > (In reply to Pascal Chevrel:pascalc from comment #8)
> > > Tracking for 63
> > 
> > Note: this is a Nightly-only experiment.
> 
> Are you sure that the experiment is the cause of the bug?

Not 100% certain, but all the reports we have are on Nightly, so it seems likely "pretending to be Chrome to get the nice version of web search" is related. But who knows. :)

Comment 12

[Tom S]

I also run into this frequently and unregistering the service worker for google.com fixed it.

Comment 13

[Thomas Wisniewski [:twisniewski]]

That's good to know, thanks evilpie. Do you find that you have to repeatedly remove the worker, or was once enough?

I wonder if there is a reliable way for our addon to unregister the serviceworker if it's detected...

Comment 14

[Tom S]

It's probably to easy to tell, but a handful so far seem to be working fine.

Comment 15

[Tom S]

Eh, I meant s/to easy/too soon/ of course.

Comment 16

[Mike Taylor [:miketaylr]]

Attachment: about:serviceworkers

Strange fact: fennec is telling me service workers are not enabled, and I have none registered. But in fact, I do have some registered -- and I can see dom.serviceWorkers.enabled is true in about:config

https://searchfox.org/mozilla-central/source/toolkit/content/aboutServiceWorkers.js#19-25

Comment 17

[Thomas Wisniewski [:twisniewski]]

I see the same message in about:serviceworkers as MikeT in comment 16, but then it lists serviceworkers regardless so I think that's a UI bug.

That said, there might be a bug regardless, as I have the Google search serviceworker installed on release Fennec on my phone, but when I load up a search page and call "navigator.serviceWorker.getRegistrtation().then(function(worker) { console.log(worker) })" it shows "undefined" (no registered workers).

Comment 18

[Jan Henning [:JanH]]

(In reply to Mike Taylor [:miketaylr] (62 Regression Engineering Owner) from comment #16)
> Created attachment 8996363 [details]
> about:serviceworkers
> 
> Strange fact: fennec is telling me service workers are not enabled, and I
> have none registered. But in fact, I do have some registered -- and I can
> see dom.serviceWorkers.enabled is true in about:config
> 
> https://searchfox.org/mozilla-central/source/toolkit/content/
> aboutServiceWorkers.js#19-25

Bug 1354779.

Comment 19

[Mike Taylor [:miketaylr]]

Thinking about this from the angle that we changed something related to ServiceWorker that left some people in a confused state (which is solved by a hard refresh)...

Boris, I see https://hg.mozilla.org/mozilla-central/rev/1c52d8baf789 landed a few days ago. Can you think of any scenario related to this that might cause a google.com service worker to think the user was offline (until they hard-refreshed)?

Unfortunately nobody on our team has managed to get in a state to reproduce or observe error messages, so debugging is thus far guess-work.

Comment 20

[Boris Zbarsky [:bzbarsky]]

It's _possible_ that that patch could have weird effects if sandboxes are not in fact correctly holding on to their globals.

We could try backing that patch out and seeing whether the problem goes away, I guess.

Comment 21

[Dale Harvey (:daleharvey)]

I have been seeing this for a day now, I have a serviceworker enabled for google.com, and did a quick log of attempting a search

https://gist.github.com/daleharvey/b4ae9e39c89f2a08fc899466065edc0c

Comment 22

[Dale Harvey (:daleharvey)]

Few more logs but cant see anything obvious in there - https://gist.github.com/daleharvey/32bc0e1268dd0f25629b2545bf9ddfde

Comment 23

[Thomas Wisniewski [:twisniewski]]

jchen just mentioned that he's running into this as well on Fennec nightly, but that things work fine in a private-browsing tab.

Comment 24

[Marcia Knous [:marcia]]

(In reply to Dennis Schubert [:denschub] from comment #3)
> We're seeing lots of reports here on webcompat.com, but sadly, I am unable
> to reproduce... For context: This is probably related to an experiment we're
> running right now (see bug 1453691) where we spoof as Chrome in Fennec
> Nightly to discover breakage with Google's Tier 1 experience.
> 
> I'd like to debug this, but as I am not able to reproduce, this is hard...
> Marcia, can you also reproduce after clearing caches and cookies? Can you
> try in a private tab? Do you see any console errors?
> 
> Tom, do you have any idea what may cause this?

My about:serviceworkers says "Service Workers are Not enabled"

After clearing cache and cookies the issue went away. But right before that I was in a persistent state of getting the offline message. I can attach my logcat if it useful (while the issue was present).

Comment 25

[Sergiu Logigan [:sergiu]]

Attachment: serviceworkers

So...
Looks like Unregistering the google.com serviceworker makes the issue go away.

Tested on a OnePlus 5, Android 8.0.1 and Nightly 63.

Comment 26

[Jan Henning [:JanH]]

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #24)
> My about:serviceworkers says "Service Workers are Not enabled"

As mentioned above, you can't rely on that text on Android because of bug 1354779.

Comment 27

[:gerard-majax]

(In reply to Sergiu Logigan [:sergiu] from comment #25)
> Created attachment 8996615 [details]
> serviceworkers
> 
> So...
> Looks like Unregistering the google.com serviceworker makes the issue go
> away.
> 
> Tested on a OnePlus 5, Android 8.0.1 and Nightly 63.

Same here, removed the Service Workers registered on Google.com, and issue worked around, nightly from today, android 8.0.0 (XZ2 Compact)

Comment 28

[Mike Taylor [:miketaylr]]

(In reply to Sergiu Logigan [:sergiu] from comment #25)
> Created attachment 8996615 [details]
> serviceworkers
> 
> So...
> Looks like Unregistering the google.com serviceworker makes the issue go
> away.
> 
> Tested on a OnePlus 5, Android 8.0.1 and Nightly 63.

Tom, uh, could we somehow do a one-off "unregister the google serviceworker" hack here in the addon to fix currently busted instances? >_>

Comment 29

[Thomas Wisniewski [:twisniewski]]

Hopefully? I'm investigating that approach now.

Comment 30

[Thomas Wisniewski [:twisniewski]]

mythmon just tried an experiment with me, and ran this command in the remote devtools on his device on the google tab:

>navigator.serviceWorker.getRegistrations().then(regs => { for (const reg of regs) reg.update(); })

It didn't help, and neither did the same command using "unregister()" instead of "update()".

I'm not sure why it wouldn't help, given that unregistering the SW in about:serviceworkers fixes the problem for others. I've also tested that the commands should work, as they remove the SW installed by https://mdn.github.io/sw-test for me (unfortunately Google's SW isn't installing for me for whatever reason, so I can't test it there).

Comment 31

[Thomas Wisniewski [:twisniewski]]

Dale, would you mind testing a long-shot theory for us? (assuming you are still experience this issue)

1) Disable our Google search experiment in about:config, by setting to false the pref extensions.gws-and-facebook-chrome-spoof.enabled
2) Do a Google search in another tab, and confirm that it is now showing the "old" Firefox Google search experience.
3) Refresh that tab with a long-press on the refresh button.
4) Re-enable our experiment again (flip the same pref in step 1 back to true).
5) Do a Google search again, confirm that the Chrome site experience is back.
6) Do another long-press refresh just in case.
7) Confirm whether the "you're offline" problem is gone.

Thanks in advance!

Comment 32

[Thomas Wisniewski [:twisniewski]]

For the record, mythmon just tried the above steps in comment 31, and it looks like the Google serviceworker lingers and keeps the new Google search experience going even when our experiment is disabled.

Moreover, mythmon learned something interesting: attempts to unregister the serviceworker (as per comment 30) are never actually resolving or rejecting. There is also nothing interesting in the tab's web console or the main process console, and I'm not sure how to investigate such an issue.

baku, bkelly, any ideas on why a serviceworker's unregister promise might never complete? Is there something mythmon should try out here to help find out?

Comment 33

[Ben Kelly [:bkelly, not reviewing]]

Miscleared the flag...

If unregister() and update() are not resolving then some kind of internal error is happening.  It could be a corrupt profile or something causing the job queue to break.

Comment 34

[Andrew Sutherland [:asuth] (he/him)]

If I fetch the serviceworker URL from https://bugzilla.mozilla.org/attachment.cgi?id=8996615 via:
curl -H "Service-Worker: script" -A "Mozilla/5.0 (Android 4.4; Mobile; rv:41.0) Gecko/63.0 Firefox/63.0" "https://www.google.com/serviceworker?pwa=search&hl=en&gl=RO"

Or just also via desktop nightly, I get the following payload:

(function(){var a=self;a.skipWaiting();a.addEventListener("activate",function(b){b.waitUntil(a.clients.claim().then(function(){return a.registration.unregister()}))});}).call(this);

So indeed we'd expect this registration to clear itself up in general unless clients.claim() is broken or, as :bkelly suggests, the job queue is broken.

Comment 35

[Mike Taylor [:miketaylr]]

One thing to note, in Fennec Nightly, we're spoofing as Chrome Mobile (approximately), to get the good search experience:

https://searchfox.org/mozilla-central/source/mobile/android/extensions/gws-and-facebook-chrome-spoof/bootstrap.js#24

Comment 36

[Blake Kaplan (:mrbkap) (inactive)]

(In reply to Mike Taylor [:miketaylr] (62 Regression Engineering Owner) from comment #35)
> One thing to note, in Fennec Nightly, we're spoofing as Chrome Mobile
> (approximately), to get the good search experience:

Changing the user agent doesn't appear to change the service worker that gets returned.

Comment 37

[Thomas Wisniewski [:twisniewski]]

Dale, Andreas, anyone, if you're still running into this problem, I'd like to test whether the following script fragment fixes it. You'd need to fire up the remote devtools and run this code in the main process:

>  var gSWM = Cc["@mozilla.org/serviceworkers/manager;1"]
>               .getService(Ci.nsIServiceWorkerManager);
>  var regs = gSWM.getAllRegistrations();
>  for (let i = 0; i < regs.length; ++i) {
>    let info = regs.queryElementAt(i, Ci.nsIServiceWorkerRegistrationInfo);
>    if (info.principal.origin.match(/^https?:\/\/(www|encrypted|maps)\.google\..*/)) {
>      console.info("Updating ServiceWorker for", info.principal.origin);
>      gSWM.propagateSoftUpdate(info.principal.originAttributes, info.scope);
>    }
>  }

With luck, it should be able to update the service worker, and the problem should go away. With a little less luck, re-running the fragment with the "propagateSoftUpdate" line replaced with this bit should at least work (by unregistering the worker entirely, and letting it reinstall upon your next Google search):

>  let cb = {
>    unregisterSucceeded() {
>      console.log("Succeeded removing serviceworker for", info.principal.origin);
>    },
>    unregisterFailed() {
>      console.log("Failed removing serviceworker for", info.principal.origin);
>    },
>    QueryInterface: ChromeUtils.generateQI([Ci.nsIServiceWorkerUnregisterCallback])
>  };
>  gSWM.propagateUnregister(info.principal, cb, info.scope);

This is basically the same code that about:serviceworkers runs, so if it doesn't work then I'm not sure what else we could try.

Comment 38

[Osmose [:osmose, :mkelly]]

(In reply to twisniewski from comment #37)
> Dale, Andreas, anyone, if you're still running into this problem, I'd like
> to test whether the following script fragment fixes it. You'd need to fire
> up the remote devtools and run this code in the main process:
> 
> >  var gSWM = Cc["@mozilla.org/serviceworkers/manager;1"]
> >               .getService(Ci.nsIServiceWorkerManager);
> >  var regs = gSWM.getAllRegistrations();
> >  for (let i = 0; i < regs.length; ++i) {
> >    let info = regs.queryElementAt(i, Ci.nsIServiceWorkerRegistrationInfo);
> >    if (info.principal.origin.match(/^https?:\/\/(www|encrypted|maps)\.google\..*/)) {
> >      console.info("Updating ServiceWorker for", info.principal.origin);
> >      gSWM.propagateSoftUpdate(info.principal.originAttributes, info.scope);
> >    }
> >  }

I was experiencing the error, and ran this snippet; it did not fix my issues and I observed a service worker for Google search in about:serviceworkers.

> With luck, it should be able to update the service worker, and the problem
> should go away. With a little less luck, re-running the fragment with the
> "propagateSoftUpdate" line replaced with this bit should at least work (by
> unregistering the worker entirely, and letting it reinstall upon your next
> Google search):
> 
> >  let cb = {
> >    unregisterSucceeded() {
> >      console.log("Succeeded removing serviceworker for", info.principal.origin);
> >    },
> >    unregisterFailed() {
> >      console.log("Failed removing serviceworker for", info.principal.origin);
> >    },
> >    QueryInterface: ChromeUtils.generateQI([Ci.nsIServiceWorkerUnregisterCallback])
> >  };
> >  gSWM.propagateUnregister(info.principal, cb, info.scope);
> 
> This is basically the same code that about:serviceworkers runs, so if it
> doesn't work then I'm not sure what else we could try.

Modifying the script with this second bit successfully removed the service workers and resolved the issue for me.

Comment 39

[Thomas Wisniewski [:twisniewski]]

Thanks a bunch, osmose! Based on that, here's a summary of what we've found out so far:

- updating the serviceworker does not help.
- unregistering it does help.
- the web APIs are unable to unregister the serviceworker (the promise never resolves/rejects).
- about:serviceworkers (or the equivalent code) can unregister the worker.

So based on this, we have a path forward here.

Mike, I could have our GWS system addon remove the serviceworker as it starts up (but only if it has not done so before, using an about:config option to record that fact).

That way, when the user restarts Firefox (presumably for the next nightly upgrade) it should do the trick. Does that sound good to you?

Comment 40

[Mike Taylor [:miketaylr]]

SGTM. Thanks!

Comment 41

[Thomas Wisniewski]

Attachment: Bug 1479209 - Update GWS experiment in bz1453691 to clear the GWS serviceworker.

Review commit: https://reviewboard.mozilla.org/r/261232/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/261232/

Comment 42

[Thomas Wisniewski [:twisniewski]]

rhelmer, this patch just updates our GWS/Facebook experiment Fennec system addon to clear the GWS service worker once on the next update (or if the user toggles the relevant about:config pref).

This is to mitigate the issue in this bug, where users have to manually remove the service worker or they cannot use GWS, as it presumes they're offline. It seems to be a temporary problem that is now resolved, but has still required some users to manually clear the worker.

Comment 43

[Robert Helmer [:rhelmer]]

Comment on attachment 8997484 [details]
Bug 1479209 - Update GWS experiment in bz1453691 to clear the GWS serviceworker.

https://reviewboard.mozilla.org/r/261232/#review268298

::: mobile/android/extensions/gws-and-facebook-chrome-spoof/bootstrap.js:92
(Diff revision 1)
> +        console.info(ServiceWorkerFixNotice);
> +        noticeShown = true;
> +      }
> +      let cb = {
> +        unregisterSucceeded() { },
> +        unregisterFailed() { },

Is it worth console-logging this, just to get diagnostic info on the off chance that it happens for QA or some user?

::: mobile/android/extensions/gws-and-facebook-chrome-spoof/bootstrap.js:128
(Diff revision 1)
>    disable();
>  };
>  
>  this.startup = () => {
>    Services.prefs.addObserver(EnabledPref, checkIfEnabled);
> +  Services.prefs.addObserver(ServiceWorkerFixPref, checkIfServiceWorkerFixNeeded);

Huh I thought observer objects required an `observe` function, but I see other code in-tree that does this so as long as it works that's cool.

Comment 44

[Thomas Wisniewski]

Comment on attachment 8997484 [details]
Bug 1479209 - Update GWS experiment in bz1453691 to clear the GWS serviceworker.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/261232/diff/1-2/

Comment 45

[Thomas Wisniewski [:twisniewski]]

Thanks for the quick review, rhelmer!

Requesting check-in.

Comment 46

[Mike Taylor [:miketaylr]]

(I clicked "Land commits" from treeherder, removing checkin-needed)

Comment 47

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/11996094e70c
Update GWS experiment in bz1453691 to clear the GWS serviceworker. r=rhelmer

Comment 48

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/11996094e70c

Comment 49

[Jovan Gerodetti]

This does not fix the issue since the service worker is registered again once I visit https://google.com. It just allows to perform one search query until the offline message is shown again.

Comment 50

[Thomas Wisniewski [:twisniewski]]

Thanks for the report, Jovan! It could help a great deal if you might help us diagnose a potential lead: we think this is related to Google's service worker presuming that support is available for the ReadableStreams web standard.

Are you handy with the remote developer tools? It would be useful to confirm whether you see any interesting console messages while being told that you're offline on a Google Search tab. (If you're not sure how to use the remote devtools, but would like to, please hop onto the #webcompat Mozilla IRC channel and we'll try to step you through it ASAP).

If you're willing to at least try enabling our (very experimental) support for ReadableStreams, that might also help confirm if it's due to them not being enabled yet on Firefox. ReadableStreams can be enabled in about:config with the dom.streams.enabled setting (though I wouldn't recommend leaving them on as they're not really ready for general usage).

Comment 51

[Thomas Wisniewski [:twisniewski]]

Google tells me that they've pushed a fix over the past 12 hours or so, and we've seen a corresponding decrease in the number of reports we're getting at webcompat.com, so I think we might be good now.

If anyone is still experiencing this issue, please let us know! Fixing it should just require un-registering the serviceworker one more time, which can be done through about:serviceworkers, or by resetting this about:config flag: extensions.gws-and-facebook-chrome-spoof.serviceworker-fix-applied (note that the flag should be immediately set back to true upon being reset).

Comment 52

[Jovan Gerodetti]

I also no longer can reproduce the issue.

Comment 53

[Daniel Holbert [:dholbert]]

I was able to reproduce the issue last night and on my first Google Search this morning (after comment 51 was posted), but when I tried another search a few minutes later, the issue had been resolved and my search was successful.

So if anyone's still hitting this, it's possible that a reload or a second attempt (or a few minutes' wait) will fix it.  That seemed to be the case for me, at least.

Comment 55

[Mike Taylor [:miketaylr]]

OK, let's go ahead and close this -- reports have stopped coming in on webcompat.com. Thanks for everyone's help on this.