1022679 - Add network analytics

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: General, defect, P5)

Description

[James Willcox (:snorp) ([e-mail]) (he/him)]

Sometimes we get reports where Fennec causes a ton of network usage, overrunning a users bandwidth allotment. This makes people angry, understandably. We really need to have a way of seeing what was causing all of that. We have about:networking which shows current active connections, but maybe we could extend that to show number of bytes transferred over some period of time (a week? a day?), broken down by page. Not sure of the best way forward here, but we need to have something.

Comment 1

[Wesley Johnston (:wesj)]

B2G has some infrastructure for this. I think maybe?:

http://mxr.mozilla.org/mozilla-central/source/dom/network/interfaces/nsINetworkStatsServiceProxy.idl

Comment 2

[Aaron Train [:aaronmt]]

↳ user files bug report that Firefox uses 1.4GB of cell data overnight (bug 1022569), and we have nothing to help diagnose the problem

Comment 3

[Taha Lukmanji]

Hi guys, I am willing to work on this... need a little bit of guidance on code structure. I am going through the wiki right now

Comment 4

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Taha Lukmanji from comment #3)
> Hi guys, I am willing to work on this... need a little bit of guidance on
> code structure. I am going through the wiki right now

This was just a random thought I had for a way to help debug problems like we had in bug 1022569. It may need some more thought before any coding happens. My initial idea was to just record the number of requests per host for the last....24 hours? And then have some about: page that would show that. There may be some privacy concerns...

Patrick, do you have any ideas here? Is there something reasonable we could do for bugs like 1022569 to help our investigation?

Comment 5

[Patrick McManus [:mcmanus]]

we can track by appid - but afaik appid is only really useful on b2g, right?

you could pretty trivially track by origin by putting a hash table in nshttpconnectionmgr for it.. make sure to consider private browsing. It would be similar to the existing connectionEntry hash but I wouldn't just add it that structure as those things get cleared out based on certain fairly regular events. (e.g. soon based changing from wifi to cell).

you can piggyback off nsHttpTransaction::SaveNetworkStats() which is currently only doing things for b2g and appid.. but it already has the byte counts you're interested in.

reporting can just be added to about:networking, so you don't need to blaze new ground there.

this won't work at all if its not http or not from necko (i.e. from flash). hopefully you can ignore that.

Comment 6

[Taha Lukmanji]

Hi Patrick, dumb question. Is https tracked by nshttptransaction? I havent looked at the code yet.

whats b2g?

Comment 7

[Taha Lukmanji]

it would be safe to assume that we get data from http/s only.

Comment 8

[Patrick McManus [:mcmanus]]

yes nshttptransaction sees https .. (you won't see the overhead bytes, but that is relatively small)

b2g is the original name of firefox os

Comment 9

[Taha Lukmanji]

Sorry guys but I have been a bit busy. Plan is to fix it by the weekend.

Comment 10

[Taha Lukmanji]

Hi Valentin,

I was tracing back to the email you sent regarding the 4 files to look at. I am confused on how does Dashboard.cpp and AboutNetworking.js work together. Could you provide me some guidelines and as the case description states that it would be nice if we display a day, or week or month of accumulative stats so is there a data repo built inside firefox or fennec as we need to save data

Taha

Comment 11

[Valentin Gosu [:valentin] (he/him) <<Parental leave March 1 - August 1>>]

Hi Taha,

Dashboard.cpp (nsIDashboard.idl) provides an interface which AboutNetworking.js uses to display the current state of the networking layer. It displays mostly the instantaneous state of sockets, connections, dns, but there some fields that may be used to get some usage statistics.

Consider the following example:

var dash = Components.classes['@mozilla.org/network/dashboard;1'].getService(Components.interfaces.nsIDashboard);
function callback(res) {
   console.log("Total received: "+res.received);
   console.log("Total sent: "+res.sent);
}

dash.requestSockets(callback);

This can give you access to the total traffic in bytes sent an received by the browser (the real numbers would be slightly higher, as we don't measure everything). If there's anything I can do to help, please let me know.

Comment 12

[Brad Lassey [:blassey] (use needinfo?)]

filter on [mass-p5]

Comment 15

[shivambalikondwar]

hi there i am an undergrad student in computer science,
i will happy yo help on this issue 
the patch i may suggest will be to create a log file for data
that is transmitted every time you start firefox 
so that one may know what went wrong
------------------>
or here is a good approach that to send notifications to the user everytime he hits particular data limit 
that would be counted as a feature though 
but still i think creating log file containing how data is transacted will make sure that where and when it was used .
 I am sure the person that installed backdoor and used 1.4 gb overnight isn't listening :P

Comment 16

[shivambalikondwar]

Attachment: Bug 1022679 add network analytics

i added nothing , for time-being i wasn't able to contact anybody
  regarding this bug over a month!. Its a mentored one as far as i see , there
  is need to change 2-3 files and it will take some time
  i am willing to submit patch for it but i want a mentor who can work
  review and respond in a week !

Comment 17

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to shivambalikondwar from comment #15)
I'm afraid we might not have enough bandwidth right now to shepherd a patch like this right now. I'm flagging Susheel and Valentin, as they might disagree. Generally, though, we're trying to not add new features to Fennec. I could see a generic data usage monitoring tool being useful in Gecko, though.

Comment 18

[:sdaswani]

James has it right - unless Andreas asks us to get this into Fennec, we will probably have to pass on mentoring you.

Comment 19

[Valentin Gosu [:valentin] (he/him) <<Parental leave March 1 - August 1>>]

I don't see it being a high priority on Firefox either. :(

Comment 20

[shivambalikondwar]

OK , but when i wanted any expertise on network related bugs i didn't find anyone 
I am new here which irc do you people are on ?

Comment 21

[Valentin Gosu [:valentin] (he/him) <<Parental leave March 1 - August 1>>]

(In reply to shivambalikondwar from comment #20)
> OK , but when i wanted any expertise on network related bugs i didn't find
> anyone 
> I am new here which irc do you people are on ?

The network IRC channel is #necko on irc.mozilla.org

Comment 22

[Chaitanya Bankanhal]

Hi, is this issue still open? I am new to open source development and would like to work on this. Where do I begin?

Comment 23

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.