Closed Bug 992405 Opened 8 years ago Closed 5 years ago

B2G Spot check (and future automation) system needed to analyze data usage for app interaction

Categories

(Testing :: General, defect, P2)

x86_64
Gonk (Firefox OS)
defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: cmtalbert, Unassigned)

Details

(Keywords: ateam-b2g-device-big, Whiteboard: [ateam-project-request])

= The Problem =
We want to be the best at data usage on Firefox OS. To that end, we would like to optimize so that when a user is performing an action on Firefox OS, they use less data (measured by bits over the wire when running with a Cellular radio) than performing an equivalent operation on our competition.

= The Project Goal Supported =
B2G - helping us scale even further into the low end device space

= Timeline =
* Getting a spot check of where we are is moderately urgent (by May).
* Future automation for b2g is desired to capture the values of data we transfer for a set of common actions (tbd)
* Competition numbers need only be captured intermittently and held as a baseline

= MVP =
* Run a b2g phone over a ceullular network, perform a common interaction on facebook, measure bytes used. 
* Do the same on Android and see how we stack up.

== Automation case ==
* Write a gaia UI test to perform this action on a repeatable basis for b2g, and report its numbers to datazilla

= Owner =
Peter Dolanjski (pdol)

A common, simple place to start would be launching the facebook app, with it configured for auto-sign in and making a comment on a post. Then logging in as the same user and making the same comment on the same post on an android phone. Then measuring the bytes transfered before and after of each of these.
Whiteboard: [ateam-project-request]
And even more details from Peter:
Here is basically what we are looking for:

At a high level, we'd like to compare the following:
Facebook view News Feed, view a photo gallery, upload a photo (using Facebook mobile client)
Email (using stock email client)
Email with 1MB Attachment Sent (using stock email client)
Email with 1MB Attachment Received (using stock email client)
Website browsing (10 specific websites, ensuring the same version is served to each OS) (using stock browser client)
Line IM app, specific message thread with photo send (using Line mobile client)
Line IM app, specific message thread with photo received (using Line mobile client)
Comparable apps using push notifications and background services over a 24 hour period (receiving a specific number of pushes)
 
Software to be used:
Firefox OS 1.3
Android 2.3
Android 4.4

The last test is probably the most ambiguous as we will need to find comparable apps on Android and Firefox OS that make use of push.  In that test, we would also like to monitor general operating system background processes which may hit the network (particularly Google's services).

Let me know if this gives you enough to provide an indication of the viability for doing something like this in house (feel free to state your assumptions) and an estimation of the timeframe that might be required.
Do we already have an easy way of measuring network consumption on B2G?
Priority: -- → P2
(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> Do we already have an easy way of measuring network consumption on B2G?

There is the usage app on the device.  I'm guessing we could use that for FxOS.  I'm not sure if it is accurate enough, though.
Having been playing this game on my personal android phone, I find that a lot of stuff takes place in the background, I have to go and kill my apps so that I don't have some random thing using up my bandwidth.

could we add a leave it alone overnight and measure the idle bandwidth consumption test?
The team is pretty busy in the near term on high priority B2G projects, like the FxOS Branding Certification Suite, standing up automation on Tarakos, and getting gaia-integration tests running on emulators and devices.

The upshot of this is, I don't think we'll have someone to tackle this for the next several weeks at least.  That makes automating a solution to getting a spot check by May pretty unlikely.

Can you perform these tests manually in order to assess the baseline numbers?  We could then look at building out an automated solution in the latter half of Q2.
Doing some manual runs of these tests would also help prove out the data-gathering portions. I'd like to be sure that we can measure the things we care about on both platforms with reasonable precision before we think about automating it.
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.