Closed Bug 1159993 Opened 9 years ago Closed 9 years ago

No reported crashes for Firefox for Android 38 beta 8

Categories

(Socorro :: General, task)

Product:
Socorro
Component:
General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(firefox38+ wontfix)

Status:
RESOLVED WONTFIX
Tracking Flags:
Tracking Status
firefox38 + wontfix

People

(Reporter: kbrosnan, Assigned: rhelmer)

References

Details

(In reply to Kevin Brosnan [:kbrosnan] from comment #0)
> https://crash-stats.mozilla.com/topcrasher/products/FennecAndroid/versions/
> 38.0b8
> https://crash-stats.mozilla.com/daily?p=FennecAndroid
> 
> We shipped this release to the Play Store over 24 hours ago.

The build ID Socorro is using is 20150112125706
This is used to distinguish betas since crash reports don't send that info.

Kevin, could you confirm that this is the correct build ID?
Assignee: nobody → rhelmer
Status: NEW → ASSIGNED
Flags: needinfo?(kbrosnan)
(In reply to Robert Helmer [:rhelmer] from comment #1)
> (In reply to Kevin Brosnan [:kbrosnan] from comment #0)
> > https://crash-stats.mozilla.com/topcrasher/products/FennecAndroid/versions/
> > 38.0b8
> > https://crash-stats.mozilla.com/daily?p=FennecAndroid
> > 
> > We shipped this release to the Play Store over 24 hours ago.
> 
> The build ID Socorro is using is 20150112125706
> This is used to distinguish betas since crash reports don't send that info.
> 
> Kevin, could you confirm that this is the correct build ID?

Oops scratch that.. I actually see two different build IDs for beta 8, we should be matching on either:

product_name | version |    platform    |    build_id    | build_type | beta_number |  repository  | update_channel | version_build 
--------------+---------+----------------+----------------+------------+-------------+--------------+----------------+---------------
 mobile       | 38.0    | android-api-9  | 20150426173102 | beta       |           8 | mozilla-beta | beta           | 
 mobile       | 38.0    | android-api-11 | 20150426173102 | beta       |           8 | mozilla-beta | beta           | 
 mobile       | 38.0    | android-x86    | 20150426173102 | beta       |           8 | mozilla-beta | beta           | 
 mobile       | 38.0    | android-api-9  | 20150427090529 | beta       |           8 | mozilla-beta | beta           | 
 mobile       | 38.0    | android-api-11 | 20150427090529 | beta       |           8 | mozilla-beta | beta           | 
 mobile       | 38.0    | android-x86    | 20150427090529 | beta       |           8 | mozilla-beta | beta           |
OK these build IDs seem to match build1 and build3 in http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/38.0b8-candidates/
We do have raw crashes present for these build IDs:

 count | day 
-------+-----
     5 |  27
   115 |  28
  1044 |  29
   170 |  30

I don't see any in our post-processing reports though, will take a look there next.
(In reply to Robert Helmer [:rhelmer] from comment #4)
> We do have raw crashes present for these build IDs:
> 
>  count | day 
> -------+-----
>      5 |  27
>    115 |  28
>   1044 |  29
>    170 |  30
> 
> I don't see any in our post-processing reports though, will take a look
> there next.

Oh, I see the problem - these crashes are all coming in on the "release" channel instead of beta. Here's an example:

https://crash-stats.mozilla.com/report/index/90e3584f-61a4-4a21-a70f-75ca62150428

Kevin, is this a problem in the build?
Flags: needinfo?(kbrosnan)
This is fallout from building Firefox Beta from the https://hg.mozilla.org/releases/mozilla-release/
Sigh.

1) This is bad, as we aren't getting those crashes in any stats atm.
2) This is a product bug in the end, as we never should use a different channel than "beta" when it's beta build. We fortunately have the updates being served by Google Play, otherwise this would disturb even updates. :(

That said, can we do a short-time workaround and rewrite the channel for this beta only on the processor side or so to "beta" so that stats work?
Tracking. Too late to build a beta 9 for mobile :/
status-firefox38: --- → affected
tracking-firefox38: --- → +
I wonder if we should file a separate bug for the releng issue of get the right channel in there.
Depends on: 1160234
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #7)
> Sigh.
> 
> 1) This is bad, as we aren't getting those crashes in any stats atm.
> 2) This is a product bug in the end, as we never should use a different
> channel than "beta" when it's beta build. We fortunately have the updates
> being served by Google Play, otherwise this would disturb even updates. :(
> 
> That said, can we do a short-time workaround and rewrite the channel for
> this beta only on the processor side or so to "beta" so that stats work?

Yes we could do something like this on the Socorro side to work around the problem. I'll start investigating this.
Filed the releng side as bug 1160234, I'm leaving this bug here for a short-term workaround for this build (we basically only care about the 20150427090529 build that shipped).

Rob, can we do a rewrite of crashes with that build ID and channel "release" to set the channel to "beta" instead, reprocess the ones we have and backfill for yesterday?
Flags: needinfo?(rhelmer)
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #11)
> Filed the releng side as bug 1160234, I'm leaving this bug here for a
> short-term workaround for this build (we basically only care about the
> 20150427090529 build that shipped).
> 
> Rob, can we do a rewrite of crashes with that build ID and channel "release"
> to set the channel to "beta" instead, reprocess the ones we have and
> backfill for yesterday?

That is what I was thinking. Let me get the processor rewrite bit figured out and landed, then we can reprocess.
Flags: needinfo?(rhelmer)
Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/9b3a8382408c948a90f7750b1ea7bb0a96992ae1
fixes bug 1159993 - rewrite release channel for certain Fennec

https://github.com/mozilla/socorro/commit/5a90c9068e490ced9856bd72411258c1ecda2b71
Merge pull request #2761 from twobraids/bug-1159993-fennec-on-wrong-channel

fixes bug 1159993 - rewrite release channel for certain Fennec
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
We're preparing a release now, when it's out we can find the jobs that need reprocessing like so:

insert into reprocessing_jobs select uuid from reports_20150427 where product like 'Fennec%' and release_channel = 'release' and build = '20150427090529'
Depends on: 1158124
Release is up, submitted crashes for reprocessing:

breakpad=# insert into reprocessing_jobs select uuid::uuid from reports_20150427 where product like 'Fennec%' and release_channel = 'release' and build = '20150427090529';
INSERT 0 2547
Looks like reprocessing is done, these now say "beta", we have no more bad entries in reports table:

https://crash-stats.mozilla.com/report/index/b2e274c1-1dc7-463a-81e4-9b7dc2150429

I'll go ahead and backfill the reports now.
Backfill running now, will probably take several hours:

for day in {27..29}
do
  psql -c "select backfill_matviews('2015-04-$day')"
done
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #18)
> https://crash-stats.mozilla.com/topcrasher/products/FennecAndroid/versions/
> 38.0b8?days=7 still, doesn't have any data, not even for 4/30 :(

Backfill finished and there still isn't data even though we're now assigning the right channel. I'll investigate this again.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Looks like the problem is that ADI still comes in on the release channel.

Going to test to see if it works if we fix this.

1) took a backup of raw_adi table on prod:

pg_dump -Fc -t raw_adi breakpad > raw_adi-backup-20150501.pgdump

2) tested WHERE clause:

select * from raw_adi
    where product_name = 'FennecAndroid'
          and product_version = '38.0'
          and build = '20150427090529'
          and date > '2015-04-27';

3) ran update with above clause:

update raw_adi
    set update_channel = 'beta'
    where product_name = 'FennecAndroid'
          and product_version = '38.0'
          and build = '20150427090529'
          and date > '2015-04-27';

Result: UPDATE 36
Seeing if backfill for yesterday works now:

select backfill_matviews('2015-04-30');
Looks like the backfill worked after correcting ADI data:

https://crash-stats.mozilla.com/daily?form_selection=by_version&p=FennecAndroid&v=38.0b8&v=&v=&v=&hang_type=any&os=Windows&os=Mac+OS+X&os=Linux&date_range_type=report&date_start=2015-04-17&date_end=2015-05-01&submit=Generate

There is a single data point on the graph and a row in the table at the bottom.

Lars has a PR up to patch up the ADI so it goes to the right channel, equivalent to comment 20.
OK we figured out how to do this, I am going to backfill.

We can put a more permanent solution in place, but I am hesitant to push a build at 5 PM on a Friday, if no one is going to be looking at it over the weekend.

Let's reconvene on Monday and figure out if this is still desired.
I am backfilling the 27th through the 30th now, I can manually fix this over weekend in case anybody's looking (it takes only a minute or so of my time) and we can decide Monday if a permanent fix is warranted.
FYI, we will probably push 38 beta 10 today.
Backfilling has been going on here for a while, we now have April 28-30 done, and May is catching up right now. It should be totally done overnight, I'll check on it this evening and make sure it doesn't slow down our normal run.
Does beta 10 suffer from the same issue?
(In reply to Kevin Brosnan [:kbrosnan] from comment #28)
> Does beta 10 suffer from the same issue?

Well the problem with 8 is that releng shipped it with the release channel "release" instead of "beta"... did that happen again with 10?

According to FTP the 38.0b10 build ID is 20150503163717:
https://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/38.0b10-candidates/build2/android-x86_info.txt

I haven't seen any crashes come in with that ID yet.
(In reply to Kevin Brosnan [:kbrosnan] from comment #28)
> Does beta 10 suffer from the same issue?

It should not, the work to fix it was done in bug 1158124. But we should verify that it has the correct channel set.
We released 38. Not an issue anymore.
status-firefox38: affected → wontfix
Wontfixed per comment 31.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.