Closed Bug 1479209 Opened 6 years ago Closed 6 years ago

Frequently getting "You're offline" when performing Google searches

Categories

(Firefox for Android Graveyard :: General, defect)

Product:
Firefox for Android Graveyard
Component:
General
Platform:
Unspecified
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox63+ fixed)

Status:
RESOLVED FIXED
Milestone:
Firefox 63
Tracking Flags:
Tracking Status
firefox63 + fixed

People

(Reporter: marcia, Assigned: wisniewskit)

References

(Depends on 1 open bug)

Details

(Whiteboard: [webcompat:p1])

Attachments

(4 files)

Screenshot of issue on Android P
147.26 KB, image/png
Details
about:serviceworkers
180.77 KB, image/png
Details
serviceworkers
134.29 KB, image/jpeg
Details
Bug 1479209 - Update GWS experiment in bz1453691 to clear the GWS serviceworker.
59 bytes, text/x-review-board-request
rhelmer
: review+
Details 
While using the latest nightly on my Pixel 2 running Android P (now Friday's build), I have been received the attached screenshot.

STR:
1. Type a Google search in the URL bar or in Google search bar on a page
2. Receive the attached screenshot

I am definitely not offline, as I can surf websites using other browsers as well as loading other websites on the same nightly. It seems I only get the message when performing a Google search for now.
Still getting this in the latest nightly. Will look in the console for errors.
OS: Unspecified → Android
We're seeing lots of reports here on webcompat.com, but sadly, I am unable to reproduce... For context: This is probably related to an experiment we're running right now (see bug 1453691) where we spoof as Chrome in Fennec Nightly to discover breakage with Google's Tier 1 experience.

I'd like to debug this, but as I am not able to reproduce, this is hard... Marcia, can you also reproduce after clearing caches and cookies? Can you try in a private tab? Do you see any console errors?

Tom, do you have any idea what may cause this?
Flags: needinfo?(mozillamarcia.knous)
Whiteboard: [webcompat]
See previous comment.
Flags: needinfo?(twisniewski)
I wonder if it's related to any ServiceWorkers... some users of my Google Search Fixer addon have had weird issues due to Google's ServiceWorker having webcompat issues, but they were not related to them "being offline".

However, I don't see any ServiceWorkers in about:serviceworkers, and this issue isn't reproducing for me. Perhaps there's some A/B test going on at Google, and others do have a ServiceWorker installed that's causing issues?
Flags: needinfo?(twisniewski)
Depends on: 1472220
(In reply to Dennis Schubert [:denschub] from comment #3)
> Marcia, can you also reproduce after clearing caches and cookies? Can you
> try in a private tab? Do you see any console errors?

Also, marcia, do you see any service workers related to google.com in about:serviceworkers, and if so, take a screenshot?
Also, do you have any addons installed in Fennec Nightly?
Tracking for 63
tracking-firefox63: --- → +
(In reply to Pascal Chevrel:pascalc from comment #8)
> Tracking for 63

Note: this is a Nightly-only experiment.
(In reply to Mike Taylor [:miketaylr] (62 Regression Engineering Owner) from comment #9)
> (In reply to Pascal Chevrel:pascalc from comment #8)
> > Tracking for 63
> 
> Note: this is a Nightly-only experiment.

Are you sure that the experiment is the cause of the bug?
(In reply to Pascal Chevrel:pascalc from comment #10)
> (In reply to Mike Taylor [:miketaylr] (62 Regression Engineering Owner) from
> comment #9)
> > (In reply to Pascal Chevrel:pascalc from comment #8)
> > > Tracking for 63
> > 
> > Note: this is a Nightly-only experiment.
> 
> Are you sure that the experiment is the cause of the bug?

Not 100% certain, but all the reports we have are on Nightly, so it seems likely "pretending to be Chrome to get the nice version of web search" is related. But who knows. :)
I also run into this frequently and unregistering the service worker for google.com fixed it.
That's good to know, thanks evilpie. Do you find that you have to repeatedly remove the worker, or was once enough?

I wonder if there is a reliable way for our addon to unregister the serviceworker if it's detected...
Flags: needinfo?(evilpies)
It's probably to easy to tell, but a handful so far seem to be working fine.
Flags: needinfo?(evilpies)
Eh, I meant s/to easy/too soon/ of course.
Attached image about:serviceworkers — 
Strange fact: fennec is telling me service workers are not enabled, and I have none registered. But in fact, I do have some registered -- and I can see dom.serviceWorkers.enabled is true in about:config

https://searchfox.org/mozilla-central/source/toolkit/content/aboutServiceWorkers.js#19-25
I see the same message in about:serviceworkers as MikeT in comment 16, but then it lists serviceworkers regardless so I think that's a UI bug.

That said, there might be a bug regardless, as I have the Google search serviceworker installed on release Fennec on my phone, but when I load up a search page and call "navigator.serviceWorker.getRegistrtation().then(function(worker) { console.log(worker) })" it shows "undefined" (no registered workers).
(In reply to Mike Taylor [:miketaylr] (62 Regression Engineering Owner) from comment #16)
> Created attachment 8996363 [details]
> about:serviceworkers
> 
> Strange fact: fennec is telling me service workers are not enabled, and I
> have none registered. But in fact, I do have some registered -- and I can
> see dom.serviceWorkers.enabled is true in about:config
> 
> https://searchfox.org/mozilla-central/source/toolkit/content/
> aboutServiceWorkers.js#19-25

Bug 1354779.
Thinking about this from the angle that we changed something related to ServiceWorker that left some people in a confused state (which is solved by a hard refresh)...

Boris, I see https://hg.mozilla.org/mozilla-central/rev/1c52d8baf789 landed a few days ago. Can you think of any scenario related to this that might cause a google.com service worker to think the user was offline (until they hard-refreshed)?

Unfortunately nobody on our team has managed to get in a state to reproduce or observe error messages, so debugging is thus far guess-work.
Flags: needinfo?(bzbarsky)
It's _possible_ that that patch could have weird effects if sandboxes are not in fact correctly holding on to their globals.

We could try backing that patch out and seeing whether the problem goes away, I guess.
Flags: needinfo?(bzbarsky)
I have been seeing this for a day now, I have a serviceworker enabled for google.com, and did a quick log of attempting a search

https://gist.github.com/daleharvey/b4ae9e39c89f2a08fc899466065edc0c
Few more logs but cant see anything obvious in there - https://gist.github.com/daleharvey/32bc0e1268dd0f25629b2545bf9ddfde
jchen just mentioned that he's running into this as well on Fennec nightly, but that things work fine in a private-browsing tab.
(In reply to Dennis Schubert [:denschub] from comment #3)
> We're seeing lots of reports here on webcompat.com, but sadly, I am unable
> to reproduce... For context: This is probably related to an experiment we're
> running right now (see bug 1453691) where we spoof as Chrome in Fennec
> Nightly to discover breakage with Google's Tier 1 experience.
> 
> I'd like to debug this, but as I am not able to reproduce, this is hard...
> Marcia, can you also reproduce after clearing caches and cookies? Can you
> try in a private tab? Do you see any console errors?
> 
> Tom, do you have any idea what may cause this?

My about:serviceworkers says "Service Workers are Not enabled"

After clearing cache and cookies the issue went away. But right before that I was in a persistent state of getting the offline message. I can attach my logcat if it useful (while the issue was present).
Flags: needinfo?(mozillamarcia.knous)
Attached image serviceworkers — 
So...
Looks like Unregistering the google.com serviceworker makes the issue go away.

Tested on a OnePlus 5, Android 8.0.1 and Nightly 63.
(In reply to Marcia Knous [:marcia - needinfo? me] from comment #24)
> My about:serviceworkers says "Service Workers are Not enabled"

As mentioned above, you can't rely on that text on Android because of bug 1354779.
(In reply to Sergiu Logigan [:sergiu] from comment #25)
> Created attachment 8996615 [details]
> serviceworkers
> 
> So...
> Looks like Unregistering the google.com serviceworker makes the issue go
> away.
> 
> Tested on a OnePlus 5, Android 8.0.1 and Nightly 63.

Same here, removed the Service Workers registered on Google.com, and issue worked around, nightly from today, android 8.0.0 (XZ2 Compact)
(In reply to Sergiu Logigan [:sergiu] from comment #25)
> Created attachment 8996615 [details]
> serviceworkers
> 
> So...
> Looks like Unregistering the google.com serviceworker makes the issue go
> away.
> 
> Tested on a OnePlus 5, Android 8.0.1 and Nightly 63.

Tom, uh, could we somehow do a one-off "unregister the google serviceworker" hack here in the addon to fix currently busted instances? >_>
Flags: needinfo?(wisniewskit)
Hopefully? I'm investigating that approach now.
Flags: needinfo?(wisniewskit)
mythmon just tried an experiment with me, and ran this command in the remote devtools on his device on the google tab:

>navigator.serviceWorker.getRegistrations().then(regs => { for (const reg of regs) reg.update(); })

It didn't help, and neither did the same command using "unregister()" instead of "update()".

I'm not sure why it wouldn't help, given that unregistering the SW in about:serviceworkers fixes the problem for others. I've also tested that the commands should work, as they remove the SW installed by https://mdn.github.io/sw-test for me (unfortunately Google's SW isn't installing for me for whatever reason, so I can't test it there).
Dale, would you mind testing a long-shot theory for us? (assuming you are still experience this issue)

1) Disable our Google search experiment in about:config, by setting to false the pref extensions.gws-and-facebook-chrome-spoof.enabled
2) Do a Google search in another tab, and confirm that it is now showing the "old" Firefox Google search experience.
3) Refresh that tab with a long-press on the refresh button.
4) Re-enable our experiment again (flip the same pref in step 1 back to true).
5) Do a Google search again, confirm that the Chrome site experience is back.
6) Do another long-press refresh just in case.
7) Confirm whether the "you're offline" problem is gone.

Thanks in advance!
Flags: needinfo?(dharvey)
For the record, mythmon just tried the above steps in comment 31, and it looks like the Google serviceworker lingers and keeps the new Google search experience going even when our experiment is disabled.

Moreover, mythmon learned something interesting: attempts to unregister the serviceworker (as per comment 30) are never actually resolving or rejecting. There is also nothing interesting in the tab's web console or the main process console, and I'm not sure how to investigate such an issue.

baku, bkelly, any ideas on why a serviceworker's unregister promise might never complete? Is there something mythmon should try out here to help find out?
Flags: needinfo?(ben)
Flags: needinfo?(amarchesini)
Flags: needinfo?(mrbkap)
Flags: needinfo?(ben)
Flags: needinfo?(amarchesini)
Flags: webcompat+
Whiteboard: [webcompat] → [webcompat:p1]
Flags: needinfo?(mrbkap)
Miscleared the flag...

If unregister() and update() are not resolving then some kind of internal error is happening.  It could be a corrupt profile or something causing the job queue to break.
Flags: needinfo?(mrbkap)
If I fetch the serviceworker URL from https://bugzilla.mozilla.org/attachment.cgi?id=8996615 via:
curl -H "Service-Worker: script" -A "Mozilla/5.0 (Android 4.4; Mobile; rv:41.0) Gecko/63.0 Firefox/63.0" "https://www.google.com/serviceworker?pwa=search&hl=en&gl=RO"

Or just also via desktop nightly, I get the following payload:

(function(){var a=self;a.skipWaiting();a.addEventListener("activate",function(b){b.waitUntil(a.clients.claim().then(function(){return a.registration.unregister()}))});}).call(this);

So indeed we'd expect this registration to clear itself up in general unless clients.claim() is broken or, as :bkelly suggests, the job queue is broken.
One thing to note, in Fennec Nightly, we're spoofing as Chrome Mobile (approximately), to get the good search experience:

https://searchfox.org/mozilla-central/source/mobile/android/extensions/gws-and-facebook-chrome-spoof/bootstrap.js#24
(In reply to Mike Taylor [:miketaylr] (62 Regression Engineering Owner) from comment #35)
> One thing to note, in Fennec Nightly, we're spoofing as Chrome Mobile
> (approximately), to get the good search experience:

Changing the user agent doesn't appear to change the service worker that gets returned.
Dale, Andreas, anyone, if you're still running into this problem, I'd like to test whether the following script fragment fixes it. You'd need to fire up the remote devtools and run this code in the main process:

>  var gSWM = Cc["@mozilla.org/serviceworkers/manager;1"]
>               .getService(Ci.nsIServiceWorkerManager);
>  var regs = gSWM.getAllRegistrations();
>  for (let i = 0; i < regs.length; ++i) {
>    let info = regs.queryElementAt(i, Ci.nsIServiceWorkerRegistrationInfo);
>    if (info.principal.origin.match(/^https?:\/\/(www|encrypted|maps)\.google\..*/)) {
>      console.info("Updating ServiceWorker for", info.principal.origin);
>      gSWM.propagateSoftUpdate(info.principal.originAttributes, info.scope);
>    }
>  }

With luck, it should be able to update the service worker, and the problem should go away. With a little less luck, re-running the fragment with the "propagateSoftUpdate" line replaced with this bit should at least work (by unregistering the worker entirely, and letting it reinstall upon your next Google search):

>  let cb = {
>    unregisterSucceeded() {
>      console.log("Succeeded removing serviceworker for", info.principal.origin);
>    },
>    unregisterFailed() {
>      console.log("Failed removing serviceworker for", info.principal.origin);
>    },
>    QueryInterface: ChromeUtils.generateQI([Ci.nsIServiceWorkerUnregisterCallback])
>  };
>  gSWM.propagateUnregister(info.principal, cb, info.scope);

This is basically the same code that about:serviceworkers runs, so if it doesn't work then I'm not sure what else we could try.
Flags: needinfo?(abovens)
(In reply to twisniewski from comment #37)
> Dale, Andreas, anyone, if you're still running into this problem, I'd like
> to test whether the following script fragment fixes it. You'd need to fire
> up the remote devtools and run this code in the main process:
> 
> >  var gSWM = Cc["@mozilla.org/serviceworkers/manager;1"]
> >               .getService(Ci.nsIServiceWorkerManager);
> >  var regs = gSWM.getAllRegistrations();
> >  for (let i = 0; i < regs.length; ++i) {
> >    let info = regs.queryElementAt(i, Ci.nsIServiceWorkerRegistrationInfo);
> >    if (info.principal.origin.match(/^https?:\/\/(www|encrypted|maps)\.google\..*/)) {
> >      console.info("Updating ServiceWorker for", info.principal.origin);
> >      gSWM.propagateSoftUpdate(info.principal.originAttributes, info.scope);
> >    }
> >  }

I was experiencing the error, and ran this snippet; it did not fix my issues and I observed a service worker for Google search in about:serviceworkers.

> With luck, it should be able to update the service worker, and the problem
> should go away. With a little less luck, re-running the fragment with the
> "propagateSoftUpdate" line replaced with this bit should at least work (by
> unregistering the worker entirely, and letting it reinstall upon your next
> Google search):
> 
> >  let cb = {
> >    unregisterSucceeded() {
> >      console.log("Succeeded removing serviceworker for", info.principal.origin);
> >    },
> >    unregisterFailed() {
> >      console.log("Failed removing serviceworker for", info.principal.origin);
> >    },
> >    QueryInterface: ChromeUtils.generateQI([Ci.nsIServiceWorkerUnregisterCallback])
> >  };
> >  gSWM.propagateUnregister(info.principal, cb, info.scope);
> 
> This is basically the same code that about:serviceworkers runs, so if it
> doesn't work then I'm not sure what else we could try.

Modifying the script with this second bit successfully removed the service workers and resolved the issue for me.
Thanks a bunch, osmose! Based on that, here's a summary of what we've found out so far:

- updating the serviceworker does not help.
- unregistering it does help.
- the web APIs are unable to unregister the serviceworker (the promise never resolves/rejects).
- about:serviceworkers (or the equivalent code) can unregister the worker.

So based on this, we have a path forward here.

Mike, I could have our GWS system addon remove the serviceworker as it starts up (but only if it has not done so before, using an about:config option to record that fact).

That way, when the user restarts Firefox (presumably for the next nightly upgrade) it should do the trick. Does that sound good to you?
Flags: needinfo?(miket)
Flags: needinfo?(dharvey)
Flags: needinfo?(abovens)
SGTM. Thanks!
Flags: needinfo?(miket)
rhelmer, this patch just updates our GWS/Facebook experiment Fennec system addon to clear the GWS service worker once on the next update (or if the user toggles the relevant about:config pref).

This is to mitigate the issue in this bug, where users have to manually remove the service worker or they cannot use GWS, as it presumes they're offline. It seems to be a temporary problem that is now resolved, but has still required some users to manually clear the worker.
Comment on attachment 8997484 [details]
Bug 1479209 - Update GWS experiment in bz1453691 to clear the GWS serviceworker.

https://reviewboard.mozilla.org/r/261232/#review268298

::: mobile/android/extensions/gws-and-facebook-chrome-spoof/bootstrap.js:92
(Diff revision 1)
> +        console.info(ServiceWorkerFixNotice);
> +        noticeShown = true;
> +      }
> +      let cb = {
> +        unregisterSucceeded() { },
> +        unregisterFailed() { },

Is it worth console-logging this, just to get diagnostic info on the off chance that it happens for QA or some user?

::: mobile/android/extensions/gws-and-facebook-chrome-spoof/bootstrap.js:128
(Diff revision 1)
>    disable();
>  };
>  
>  this.startup = () => {
>    Services.prefs.addObserver(EnabledPref, checkIfEnabled);
> +  Services.prefs.addObserver(ServiceWorkerFixPref, checkIfServiceWorkerFixNeeded);

Huh I thought observer objects required an `observe` function, but I see other code in-tree that does this so as long as it works that's cool.
Attachment #8997484 - Flags: review?(rhelmer) → review+
Thanks for the quick review, rhelmer!

Requesting check-in.
Keywords: checkin-needed
(I clicked "Land commits" from treeherder, removing checkin-needed)
Keywords: checkin-needed
Pushed by mitaylor@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/11996094e70c
Update GWS experiment in bz1453691 to clear the GWS serviceworker. r=rhelmer
https://hg.mozilla.org/mozilla-central/rev/11996094e70c
Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox63: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 63
This does not fix the issue since the service worker is registered again once I visit https://google.com. It just allows to perform one search query until the offline message is shown again.
Flags: needinfo?(twisniewski)
Thanks for the report, Jovan! It could help a great deal if you might help us diagnose a potential lead: we think this is related to Google's service worker presuming that support is available for the ReadableStreams web standard.

Are you handy with the remote developer tools? It would be useful to confirm whether you see any interesting console messages while being told that you're offline on a Google Search tab. (If you're not sure how to use the remote devtools, but would like to, please hop onto the #webcompat Mozilla IRC channel and we'll try to step you through it ASAP).

If you're willing to at least try enabling our (very experimental) support for ReadableStreams, that might also help confirm if it's due to them not being enabled yet on Firefox. ReadableStreams can be enabled in about:config with the dom.streams.enabled setting (though I wouldn't recommend leaving them on as they're not really ready for general usage).
Status: RESOLVED → REOPENED
Flags: needinfo?(twisniewski)
Resolution: FIXED → ---
Google tells me that they've pushed a fix over the past 12 hours or so, and we've seen a corresponding decrease in the number of reports we're getting at webcompat.com, so I think we might be good now.

If anyone is still experiencing this issue, please let us know! Fixing it should just require un-registering the serviceworker one more time, which can be done through about:serviceworkers, or by resetting this about:config flag: extensions.gws-and-facebook-chrome-spoof.serviceworker-fix-applied (note that the flag should be immediately set back to true upon being reset).
Flags: needinfo?(mrbkap)
Depends on: 1387483
I also no longer can reproduce the issue.
I was able to reproduce the issue last night and on my first Google Search this morning (after comment 51 was posted), but when I tried another search a few minutes later, the issue had been resolved and my search was successful.

So if anyone's still hitting this, it's possible that a reload or a second attempt (or a few minutes' wait) will fix it.  That seemed to be the case for me, at least.
OK, let's go ahead and close this -- reports have stopped coming in on webcompat.com. Thanks for everyone's help on this.
Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → FIXED
Assignee: nobody → wisniewskit
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: