1104175 - Continuous access to push.services.mozilla.com

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: SimplePush, defect)

Description

[Boaz Dodin]

For about a month there is a major issue on GP Peak phones - it looks like the phone every six seconds makes a DNS query to resolve the domain: push.services.mozilla.com
(Please take a look at the bug's url for more info on GP forum)

Firstly I though this issue related to some app, but according to this thread on mozillaZine:
http://forums.mozillazine.org/viewtopic.php?f=23&t=2890709
it looks that the same issue present even on Firefox nightly.

So, maybe it's related to some wrong reply from the above server?

Comment 1

[Mossroy]

I have the same issue on my Peak (recent 2.0 build), but don't have it on a ZTE Open C (2.0 build from 20141123).
Could it be Peak-specific?

Comment 2

[msth67]

Hardly so,if it is happening on desktop versions of Nightly as well.

Comment 3

[Boaz Dodin]

After the upgrade of my Peak to the latest 2.1 build (25/12/2014 build), I can confirm that this issue not related to any app.

As per detailed Usage app info, after 5 minutes of not using the phone, mobile data on, wi-fi off:
about 100% of the usage related to System, in total of about 210 kb (or about 42 kb/min).

In other words - this background usage may take about 1.8 gb every month!
This usage may cost $$-$$$ for some users.

I think it should be pretty urgent to at least understand who exactly (how many users) suffer from this issue.

Any ideas who may take a look at this?
Please don't hesitate to use the "need more info" option :)

Comment 4

[Lina Butler [:lina]]

(In reply to Boaz Dodin from comment #0)
> it looks that the same issue present even on Firefox nightly.

Hi Boaz! To clarify, are you referring to Desktop nightly, or Firefox OS nightly? As far as I know, the Push API is only enabled in FxOS.

Firefox Hello on Desktop *does* use the Push infrastructure, but they have a dedicated client and Push cluster (*.push.hello.firefox.com). If you've been able to reproduce this on Desktop, I'm curious where the `updates.push.services.mozilla.com` are coming from.

Comment 5

[Lina Butler [:lina]]

What's even more interesting is that `updates.push.services.mozilla.com` is the endpoint for app servers to send notifications...why is the client connecting to it? I'd expect to see connections to `push.services.mozilla.com`, which is the WebSocket endpoint that clients use to receive notifications.

Comment 6

[Boaz Dodin]

Hi Kit,

I can only confirm it on Firefox OS 2.0 and 2.1, GP Peak.

Desktop - according to the thread on mozillaZine (the link in my bug's description, comment #0), someone suffer from the very similar symptoms on Desktop.

Comment 7

[msth67]

I'm 100% sure that I have seen a background connection to updates.push.services.mozilla.com using a Nightly build not so long ago.
Afterwards I've disabled the service and blanked the server in about:config, and I haven't seen it since.

Comment 8

[Mossroy]

On my Peak device, I had a look at the prefs.js file, and found this line :
user_pref("services.push.userAgentID", "9dcfd169-359d-44e6-9be2-ca28f9f12738@5590323d9cc7c642aab4004884aa1b1263fa710f");

There is no such preference set on my ZTE Open C device, so I tried to remove it :
adb shell
stop b2g
vi /data/b2g/mozilla/*.default/prefs.js
(copy/paste the services.push.userAgentID line in case of, and remove the line)
start b2g
exit

It seems that it solved my issue : I don't see the network access every 5 seconds any more.

After rebooting the device, another pref seems to be generated in prefs.js for this userAgentID : a shorter string like b4eadd5578f1419981ee552339b6a991. So I suppose the issue might come from an invalid userAgentID set in prefs.js? (I don't know where it came from : I did not set it myself)

Comment 9

[Richard Pappalardo [:rpapa][:rpappalardo]]

Kit, is the userAgentID assigned by the push server?

Comment 10

[msth67]

BTW, in https://bugzilla.mozilla.org/show_bug.cgi?id=1104175#c7 I was referring to a Desktop Nightly build.

Comment 11

[Lina Butler [:lina]]

Sorry I missed this, Richard. For the initial handshake (without `services.push.userAgentID` set), yes; the server selects the ID.

For subsequent reconnects, the client provides the assigned ID; if it's still valid, it'll be echoed back by the server. Otherwise—say, if the backing store went down and all registration records were lost—the server will assign the client a new ID.

Comment 12

[Mossroy]

Thanks Kit for the clarification.

So, if I understood correctly, removing the userAgentID in prefs.js (like suggested above) is safe, because it's re-generated on the server side.

Could anyone confirm that it solves the issue, as for me?

Comment 13

[Panos Astithas (he/him) [:past] (please ni?)]

This bug just eliminated my 300 MB data plan for this month in less than two weeks. I'm using a Flame running latest master and I didn't have high data plan usage in the past until now (I would always end the month with 50% usage). The Firefox OS Usage app points to "Others" as the app that ate ~250 MB from my quota and after connecting WebIDE to the device, I can see continuous requests (about every second) for push.services.mozilla.com that get a 101 reply:

Request headers:
Host: push.services.mozilla.com
User-Agent: Mozilla/5.0 (Mobile; rv:40.0) Gecko/40.0 Firefox/40.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Sec-WebSocket-Version: 13
Origin: wss://push.services.mozilla.com/
Sec-WebSocket-Protocol: push-notification
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: sZJkMz+UJNIgO2dLEJbVSQ==
Connection: keep-alive, Upgrade
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket

Response headers:
Connection: Upgrade
Sec-WebSocket-Accept: ARTNqsxha/wX1rHtZQuY1SGC85c=
Sec-WebSocket-Protocol: push-notification
Upgrade: websocket

I have disabled data for now as I don't want to bleed money, but I'm happy to debug further if you give me some pointers.

Comment 14

[Mossroy]

Panos, did you try the workaround I mentioned in comment 8 (removing the pref "services.push.userAgentID") ?
It solved the issue for me

Comment 15

[Lina Butler [:lina]]

Panos, if it's not too late, could you post your `userAgentID` before deleting it, please? I wonder if the server is rejecting it and closing the connection instead of issuing a new one...

Comment 16

[Panos Astithas (he/him) [:past] (please ni?)]

My userAgentID is 55fe4fd6ea2b47bb96987f55b58c853b and I can wait for your debugging before trying the workaround.

Comment 18

[Mossroy]

I'm not sure this is a dupe of bug 1152264.
The symptom is a bit different : a request every 6 seconds (precisely) vs continuous requests
It happened on a 2.0 version in november 2014 vs a recent 2.2
And I did not modify the same preferences to workaround the issue.

In my case, it seemed to come from an invalid userAgentId.

What is common to both issues, is that, for an unknown reason, some bogus prefs have been set (and not by the user)