Metrics is reporting abnormal/un-expected number of ADUs for mozilla central builds

VERIFIED FIXED in Unreviewed

Status

Mozilla Metrics
Frontend Reports
VERIFIED FIXED
7 years ago
7 years ago

People

(Reporter: chris hofmann, Unassigned)

Tracking

unspecified
Unreviewed
x86
All
Dependency tree / graph

Details

Attachments

(1 attachment)

(Reporter)

Description

7 years ago
Mozilla Central ADUs as forwarded from metrics to socorro have started reporting unexpectedly low numbers in the past few days.

up to the branch point for beta 9, beta9pre was running 52-75,000 ADUs

2011-01-09	3357	64087	5.24%	52.3819
2011-01-08	3309	64318	5.14%	51.4475
2011-01-07	3921	72848	5.38%	53.8244
2011-01-06	3027	74651	4.05%	40.5487
2011-01-05	2451	75356	3.25%	32.5256
2011-01-04	2345	75234	3.12%	31.1694
2011-01-03	2084	73083	2.85%	28.5155
2011-01-02	1704	59750	2.85%	28.5188
2011-01-01	1588	52213	3.04%	30.4139
2010-12-31	1899	57361	3.31%	33.1061
2010-12-30	2649	67821	3.91%	39.0587
2010-12-29	3901	70457	5.54%	55.3671
2010-12-28	3487	71775	4.86%	48.5824


after the branch point and creation of 4.0b10pre we started to see numbers rise up until Jan. 14 as trunk users started to shift over to b10pre, but since then numbers have declined. 

2011-01-19	1744	10533	16.56%	165.575
2011-01-18	1843	15867	11.62%	116.153
2011-01-17	1662	23857	6.97%	69.6651
2011-01-16	1790	24215	7.39%	73.9211
2011-01-15	2099	42435	4.95%	49.4639
2011-01-14	2083	51809	4.02%	40.2054
2011-01-13	1865	45557	4.09%	40.9377
2011-01-12	986	29412	3.35%	33.5237
2011-01-11	166	3865	4.29%	42.9495

 Having 10,533 mozilla-central users is quite alarming and the crash counts haven't seen a corresponding drop.  This means that either usage is remaining the same and ADU numbers are off, or that usage has declined and we have gotten a lot crashier in the last few days.   Looking at crash data we haven't seen any unusual spiking crashes so my first guess is that we are under counting ADUs

You can see the effect on crashes per 100 users at

http://crash-stats.mozilla.com/daily?form_selection=by_version&p=Firefox&v[]=4.0b10pre&throttle[]=100.00&v[]=4.0b9&throttle[]=100.00&v[]=3.6.13&throttle[]=10.00&v[]=&throttle[]=10.00&hang_type=any&os[]=Windows&os[]=Mac&os[]=Linux&date_start=2011-01-06&date_end=2011-01-20&submit=Generate

Any chance we could have shifted a large body of trunk users around with the update system in the last 5 days?
(Reporter)

Updated

7 years ago
Group: metrics-private
CC list accessible: false
Not accessible to reporter
(Reporter)

Comment 1

7 years ago
bug 600865 has info about an unexpected rise in mozilla-central users back in sept.   this decline puts us at, or lower, than the numbers we've seen since back until then.
(Reporter)

Comment 2

7 years ago
here's a chart from pentaho that shows possible shifts.  

b9 may have ramped a bit faster than expected and maybe we lost some b10pre mozilla-central users to those branched pre-release b9 builds before it was pushed to the beta audience.  is that possible?

	    4.0b8pre	4.0b8 4.0b9pre 4.0b10pre 4.0b9	Other
12/22/10	21707	51845	72939			8462
12/23/10	18486	158369	70762			8370
12/24/10	14828	327616	62641			7892
12/25/10	12066	452057	54636			6726
12/26/10	12317	579535	59339			6873
12/27/10	14598	740120	72608			8457
12/28/10	13385	844929	71779			8172
12/29/10	12310	917221	70462			8107
12/30/10	11256	956933	67824			7855
12/31/10	9119	879472	57364			7034
01/01/11	7447	895783	52219			6073
01/02/11	8646	986896	59753			6527
01/03/11	10944	1145155	73089			7326
01/04/11	10707	1212025	75239			8039
01/05/11	10069	1253137	75364			8110
01/06/11	9375	1275910	74654			7920
01/07/11	8689	1288473	72853			7714
01/08/11	7500	1203676	64323			6853
01/09/11	6977	1224773	64092			6996
01/10/11	8679	1427535	77453			8110
01/11/11	8502	1471795	74923			12391
01/12/11	8079	1502611	49197			39608
01/13/11	7878	1516235	32724	45557	4817	7866
01/14/11	7337	1484306	24732	51809	19349	7798
01/15/11	5940	1166470	17941	42436	201179	6642
01/16/11	5724	794494	15370	24216	632247	6720
01/17/11	7227	736305	16577	23858	946511	8055
01/18/11	6967	523717	14478	15867	1200817	8028
01/19/11	6539	404806	12101	10534	1323453	7802
01/20/11	6303	335779	10489	6970	1408010	7721
(Reporter)

Comment 3

7 years ago
re-ordered the colums to make it easier to see that b8 users are moving over to b9 as expected, but b9pre are not making the transition to b10pre.  The started to, but now we seem to be loosing users from mozilla-central.  Its not clear where they might be going.


	4.0b8	4.0b9	4.0b8pre	4.0b9pr 4.0b10pr Other
12/22/10	51845		21707	72939		8462
12/23/10	158369		18486	70762		8370
12/24/10	327616		14828	62641		7892
12/25/10	452057		12066	54636		6726
12/26/10	579535		12317	59339		6873
12/27/10	740120		14598	72608		8457
12/28/10	844929		13385	71779		8172
12/29/10	917221		12310	70462		8107
12/30/10	956933		11256	67824		7855
12/31/10	879472		9119	57364		7034
01/01/11	895783		7447	52219		6073
01/02/11	986896		8646	59753		6527
01/03/11	1145155		10944	73089		7326
01/04/11	1212025		10707	75239		8039
01/05/11	1253137		10069	75364		8110
01/06/11	1275910		9375	74654		7920
01/07/11	1288473		8689	72853		7714
01/08/11	1203676		7500	64323		6853
01/09/11	1224773		6977	64092		6996
01/10/11	1427535		8679	77453		8110
01/11/11	1471795		8502	74923		12391
01/12/11	1502611		8079	49197		39608
01/13/11	1516235	4817	7878	32724	45557	7866
01/14/11	1484306	19349	7337	24732	51809	7798
01/15/11	1166470	201179	5940	17941	42436	6642
01/16/11	794494	632247	5724	15370	24216	6720
01/17/11	736305	946511	7227	16577	23858	8055
01/18/11	523717	1200817	6967	14478	15867	8028
01/19/11	404806	1323453	6539	12101	10534	7802
01/20/11	335779	1408010	6303	10489	6970	7721
(Reporter)

Comment 4

7 years ago
Created attachment 505937 [details]
formating on that paste not so good. here is a spreadsheet.
I bet it's related to the blocklist ping URL changing on the 14th:
 http://hg.mozilla.org/mozilla-central/rev/c9420f27b9dc
The associated bug is bug 620837.

Daniel, any idea as to what is causing this?
There was a stuck deployment for the code that accepts the new blocklist parameters.  It was tested in staging with manual data but I didn't follow up on the 15th to make sure that the new data was flowing through properly.

I corrected the stuck deployment and am running back processing for the nightly builds for the past week.  That should be finished in the next two hours and will be pushed out the the affected systems tonight.
The machine that was running this last night ran out of memory due to other production processes that were running concurrently.

I am rerunning it on two machines this morning and will post a status when it is caught up.
Everything from the 14th through the 20th is caught up.  The 21st requires a little more work because half the day is right and half of it is wrong. I'll get that worked in tomorrow.
Data mentioned in comment #9 is updated in the Metrics datawarehouse, but it is not yet visible in Socorro due to a config issue in Phoenix, bug 627802.
Depends on: 627802
(Reporter)

Comment 11

7 years ago
seems to hang at the "processing" popup when I try to view  https://metrics.mozilla.com/pentaho/content/pentaho-cdf-dd/Render?solution=metrics&path=&file=Home.wcdf

Comment 12

7 years ago
Investigating. Vertica seems to be having issues

Updated

7 years ago
Blocks: 628145

Comment 13

7 years ago
And it's back up. Thanks Daniel and Jabba for jumping into it
We should have updated ADUs in socorro for the 14th through the 21st now.  Please verify.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Reporter)

Comment 15

7 years ago
yeah, numbers on metrics and crash-stats for b10pre are now tracking closer to what we have seen on mozilla-central for for b9pre, b8pre, ...  Thanks!

2011-01-22 	1,254 	57,075 	100% 	2.2%
2011-01-21 	1,785 	64,805 	100% 	2.75%
2011-01-20 	2,101 	66,761 	100% 	3.15%
2011-01-19 	1,879 	66,076 	100% 	2.84%
2011-01-18 	1,974 	64,934 	100% 	3.04%
2011-01-17 	1,817 	62,674 	100% 	2.9%
2011-01-16 	1,892 	49,421 	100% 	3.83%
2011-01-15 	2,184 	47,073 	100% 	4.64%
2011-01-14 	2,205 	51,842 	100% 	4.25%
2011-01-13 	1,976 	45,557 	100% 	4.34%
2011-01-12 	1,066 	29,412 	100% 	3.62%
2011-01-11 	185 	3,865 	100% 	4.79%
2011-01-10 	- 	- 	- 	-
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.