706704 - drop in adus' after mobile builds switch to native UI

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect, P1)

Description

[chris hofmann]

attached chart built from data at https://metrics.mozilla.com/pentaho/content/pentaho-cdf/RenderHTML?solution=metrics&path=blocklist/blocklist_0&dashboard=template.html&template=new-metrics&title=Firefox%20Usage

shows that we started a drop in ADUs on Fennec 11.0a1 corresponding to the day that nightly users were served a native UI version to update the xul based UI version.

this drop in the metrics data could be explained by the metrics system no longer counting right base on changes that might have been made in the appID or other tracking data that the metrics system looks at, or it could be explained by nightly mobile testers abandoning the night builds for some reason.  

Its not clear from the chart they they went to other experimental build channels.

we need to get to a good explaination of the numbers to understand how many users are actually testing nightly native UI builds at this point.  that will help calibrate crash data and other feedback.

Daniel, can you help look at this from the metrics side?

The nightly native UI build I'm running shows a user agent of:

 Mozilla 5.0 (Android; Linux armv7l; rv:11.0a1) Gecko/20111130 Firefox/11.0a1 Fennec11.0a1

Comment 1

[chris hofmann]

Attachment: mobile adus in the last 30 days preceeding Nov 29

Comment 2

[Robert Kaiser]

The important part is that those are on the nightly-birch channel, not the nightly channel - at least for now. So, if this is per channel, this is expected.

Given that a lot of our Nightly testers are probably in-house, could there be any connection with the Thanksgiving holidays?

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

The application information for the channel nightly-birch is not being properly recognized as part of Fennec.  I will look into this right now.

By any chance, did we change the app name or app id for birch?

Comment 4

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Daniel Einspanjer :dre [:deinspanjer] from comment #3)
> The application information for the channel nightly-birch is not being
> properly recognized as part of Fennec.  I will look into this right now.
> 
> By any chance, did we change the app name or app id for birch?

For Fennec Native the AppName is the same, but the AppID is now {aa3c5121-dab2-40e2-81ca-7ea25febc110}

Comment 5

[Daniel Einspanjer [:dre] [:deinspanjer]]

Ugh.  No one in Metrics knew about this and hence we are not classifying that new app id as being part of Fennec.  We'll get an emergency update in to correct it starting today, but I'd rather not reprocess the previous two weeks of data to pick up the miss-classified ADU.

choffman,  the birch channel ADU was up to ~650 per day last week.

Comment 6

[Martijn Wargers (dead)]

Does comment 5 say this bug has been resolved now?

Comment 7

[chris hofmann]

daniel? ^^

Comment 8

[Erin Lancaster [:elan]]

Daniel: checking in on this. Did the emergency update get deployed on 12/01 per your comment? Thank you.

Comment 9

[Daniel Einspanjer [:dre] [:deinspanjer]]

Hi.  It was deployed, but there seems to have been a problem with it not propigating all the way to the export.  I am working on it now and will put an update in this bug in a couple of hours.

Comment 10

[Daniel Einspanjer [:dre] [:deinspanjer]]

Propagated the change to all systems and verified that the dashboard chart now has corrected data for birch.  Bonus, I had to back-process, so we have data going back to the inception of birch rather than just Dec 1.

Comment 11

[chris hofmann]

did the birch branch users get folded into the 11.a1 counts, or the "other" counts.  I suspect the 11.0a1 counts but just want to confirm.

here is what the recent numbers look like after the rerunning of the data.  It looks like we might have lost 2-300 or so nightly users in the moves from xul to native and around to the birch branch and back to central if they all are being counted as 11.0a1 users in all of these moves.

           11.0a1       other

2011-11-13 496		550
2011-11-14 520		577
2011-11-15 608		575
2011-11-16 672		520
2011-11-17 734		565
2011-11-18 761		553
2011-11-19 831		544
2011-11-20 846		508
2011-11-21 808		568
2011-11-22 899		491
2011-11-23 930		557
2011-11-24 934		524
2011-11-25 1,001	523
2011-11-26 898		534
2011-11-27 988		513
2011-11-28 917		513
2011-11-29 865		516
2011-11-30 891		526  <- migrate xul users to native on birch branch
2011-12-01 846		518
2011-12-02 835		543
2011-12-03 789		492
2011-12-04 747		484
2011-12-05 651		473
2011-12-06 666		440
2011-12-07 771		538
2011-12-08 718		516
2011-12-09 708		536
2011-12-10 740		497
2011-12-11 655		505
2011-12-12 764		482

Comment 12

[Erin Lancaster [:elan]]

Daniel: can you confirm if birch branch users get folded into the 11.a1 counts, or the "other" counts? 

I would like to be able to confirm the ADUs for mobile in tomorrow's Noon (PST) mobile meeting. Many thanks.

Comment 13

[Daniel Einspanjer [:dre] [:deinspanjer]]

Yes, I can confirm that.  It looks like we did dip a little after the merge, but that coincides with holidays also.

http://screencast.com/t/lp7Fdk23ES

Comment 14

[chris hofmann]

here is another view that shows things being roughly back on track when comparing the nightly, aurora, and beta stages of fennec 9 against recent usage levels of fennec 10, 11, and 12.   a bit lower on native, and still very low overall, so thats a problem needing correction and we talked about the need for more users a bit at today's product planning meeting.

http://people.mozilla.org/~chofmann/mobile/mobile-nightly-aurora-adus.png

Comment 15

[chris hofmann]

and here is the ramp up in beta adus that we have seen over the last several releases.    looks like we get up to 12-14k before shipping and putting out the release to final version users. http://people.mozilla.org/~chofmann/mobile/mobile-beta-adus.png