Closed Bug 876143 Opened 11 years ago Closed 11 years ago

Socorro aggregation data missing for May 21, 25, and 28

Categories

(Data & BI Services Team :: DB: MySQL, task)

Product:
Data & BI Services Team
Component:
DB: MySQL
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: kairo, Assigned: selenamarie)

Details

Aggregation data on Socorro prod for the 25nd is missing. Unfortunately, it looks like we still don't have the ADI yet, though, so that needs to be pushed by metrics first:

breakpad=> SELECT max(date) FROM raw_adu;
    max     
------------
 2013-05-23
(1 row)


We might not need to backfill matviews, as it might self-heal once the ADI data is there, but we should watch out for this.


(Funny that exactly when I have time to take a fast look even though I'm on vacation, it's right when this happens.)
Data for the 24th is still missing in Socorro. Looks like data for the 21st of May is missing as well.
My guess is that deinspanjer is enjoying his weekend and may not see this until Tuesday, because Monday is a statutory US holiday.
FWIW, if metrics does come through and you NEED the backfill done before Tuesday morning, please have the sysadmins page mpressman or me (I'm on flights from 5 pm - midnight Eastern time, so I don't know if I will be around to help if it's later today. I'm around tomorrow though).
Please let us know when the metrics backfill has been pushed, and we'll push the ADU/backfill as needed.
resolving incomplete; if this is being actively worked on, please reopen.
Assignee: server-ops-database → scabral
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Summary: Get ADI pushed and possibly backfill missing Socorro aggregation data for 2013-05-25 → Socorro aggregation data missing for May 21, 25, and 28
I don't care who is needed to fix this, but we are still missing data for 21st, 25th, and 28th, which messes up trend data.
Having a look at those dates to see if we have metrics, and then will run a backfill if we do.
Assignee: scabral → sdeckelmann
Based on raw_adu data, running backfills for 25 and 28, but can't for 21. 

breakpad=# select max(received_at) from raw_adu where date = '2013-05-25'; 
              max              
-------------------------------
 2013-05-26 04:19:10.991168+00
(1 row)

Time: 41.725 ms
breakpad=# select max(received_at) from raw_adu where date = '2013-05-28'; 
              max              
-------------------------------
 2013-05-29 19:22:14.298495+00
(1 row)

Time: 175.610 ms
breakpad=# select max(received_at) from raw_adu where date = '2013-05-21'; 
 max 
-----
 
(1 row)

Time: 0.393 ms
breakpad=# select backfill_matviews('2013-05-25', '2013-05-25');
INFO:  backfilling reports_clean
:mreid had a look into it and we are also missing data for May 24.
Ok! We now have ADI from metrics for all the dates. Still waiting on backfill_matviews() to complete for May 25, then will start the remaining backfilling.

breakpad=# select date, count(*) from raw_adu where date > '2013-05-19' group by date; 
    date    | count 
------------+-------
 2013-05-20 | 44330
 2013-05-21 | 45801
 2013-05-22 | 45559
 2013-05-23 | 45357
 2013-05-24 | 44190
 2013-05-25 | 38920
 2013-05-26 | 39572
 2013-05-27 | 44538
 2013-05-28 | 45361
 2013-05-29 | 45307
 2013-05-30 | 44438
 2013-05-31 | 43958
 2013-06-01 | 38970
 2013-06-02 | 39620
(14 rows)

Time: 373.567 ms
May 25 complete. Started May 21 backfill.
https://crash-stats-django.mozilla.org/daily?p=Firefox shows 21, 25, and 28 data now, but apparently 24 is missing there, as comment #10 states.
Done!
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Done!
yay, thanks!
Status: RESOLVED → VERIFIED
According to https://addons.mozilla.org/en-US/firefox/addon/firefox-2-theme-for-firefox-3x/statistics/?last=90

We still need: apr 29-30, May 1-2, 7-8, 16-19, 21 and 25.
rctgamer3 ping'd me and said that deinspanjer said we could help.
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
(In reply to Sheeri Cabral [:sheeri] from comment #17)
> According to
> https://addons.mozilla.org/en-US/firefox/addon/firefox-2-theme-for-firefox-
> 3x/statistics/?last=90
> 
> We still need: apr 29-30, May 1-2, 7-8, 16-19, 21 and 25.
> rctgamer3 ping'd me and said that deinspanjer said we could help.

That chart pulls from tp-socorro01-master02?
Stats missing/zero:
April 29/30, May 1-30 (1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30)

Extensions:
AdblockPlus: [May 2, 7-8, 16-19, 21, 25]
DownThemAll: [May 2, 4, 6, 14-15, 21, 25, 27-28, 30]
GreaseMonkey: [Apr 29-30] [May 1, 7, 10, 12, 19]
Firebug [Apr 30] [May 5, 10, 12, 19, 22, 25]
FireFTP [May 2, 12, 14, 19, 29]
Stylish [May 16-19, 21, 23, 25, 30]

Themes:
A Blue Fox Theme: [Apr: 30; May: 4-5, 14-17, 20, 26]
Bloody Red theme pack - 5 themes in one [May 4, 11, 14-15, 17, 26]
Classic Compact: [May: 3, 9, 17, 21, 24, 30]
Firefox 2, the theme, reloaded [Apr 29, May 1, 7, 17, 20, 22]
FXChrome [Apr 29] [May 12, 18-20]
Noia 4 [Apr 29, May 4, 18-20, 22, 26]
Silvermel: [May 5, 7, 18-19, 21, 28]
Hi there!

(In reply to rctgamer3 from comment #19)
> Stats missing/zero:
> April 29/30, May 1-30
> (1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/
> 29/30)
> 
> Extensions:
> AdblockPlus: [May 2, 7-8, 16-19, 21, 25]
> DownThemAll: [May 2, 4, 6, 14-15, 21, 25, 27-28, 30]
> GreaseMonkey: [Apr 29-30] [May 1, 7, 10, 12, 19]
> Firebug [Apr 30] [May 5, 10, 12, 19, 22, 25]
> FireFTP [May 2, 12, 14, 19, 29]
> Stylish [May 16-19, 21, 23, 25, 30]
> 
> Themes:
> A Blue Fox Theme: [Apr: 30; May: 4-5, 14-17, 20, 26]
> Bloody Red theme pack - 5 themes in one [May 4, 11, 14-15, 17, 26]
> Classic Compact: [May: 3, 9, 17, 21, 24, 30]
> Firefox 2, the theme, reloaded [Apr 29, May 1, 7, 17, 20, 22]
> FXChrome [Apr 29] [May 12, 18-20]
> Noia 4 [Apr 29, May 4, 18-20, 22, 26]
> Silvermel: [May 5, 7, 18-19, 21, 28]

I don't think this is the right place to be asking for help on this, unfortunately. The system that provides this data to addons is not Socorro.

We get our data from #metrics. I'll open a ticket and ask Daniel what is up.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.