842360 - appinfo section is missing from FHR payload

Status: RESOLVED WONTFIX

(Firefox Health Report Graveyard :: Client: Desktop, defect, P4)

Description

[Mark Reid [:mreid]]

Attachment: Payload without an appinfo section

I was just looking at my own FHR payload and noticed that the "appinfo" provider in the "last" section is missing.

Steps to Reproduce:
Follow the instructions at https://gist.github.com/mreid-moz/4979402

Attached are the generated payload as well as the relevant-looking lines from the log.

Comment 1

[Mark Reid [:mreid]]

Attachment: Log messages while generating the payload

Comment 2

[Richard Newman [:rnewman]]

Comment on attachment 715206 [details]
Log messages while generating the payload

Yay, file system. Also yay promise stacks :/

Comment 3

[Gregory Szorc [:gps]]

I see two issues here:

1) Profile age detection isn't robust enough.
2) Profile age error prevents app info submission.

We should defend against #2 by catching errors and continuing with submission. We should try to land #2 before 21 is cut over tomorrow, just to avoid 1 more uplift.

Comment 4

[Gregory Szorc [:gps]]

And why is the profile provider interfering with the app info provider? That shouldn't happen. I thought we had test coverage ensuring provider failures don't impact other providers :/

Comment 5

[Richard Newman [:rnewman]]

See component bugs.

Comment 6

[Mark Reid [:mreid]]

After a restart of Firefox, I can no longer reproduce this.

Comment 7

[Gregory Szorc [:gps]]

I'd still like figures from Metrics on how many submissions are missing the appinfo section.

Comment 8

[Mark Reid [:mreid]]

Looks like of the payloads with a "thisPingDate" in February 2013, 44274 payloads are missing the appinfo section (of 232513 total).

Comment 9

[Gregory Szorc [:gps]]

(In reply to Mark Reid [:mreid] from comment #8)
> Looks like of the payloads with a "thisPingDate" in February 2013, 44274
> payloads are missing the appinfo section (of 232513 total).

Holy crap. That's a lot :(

It sounds like we did (do?) have a bug with provider isolation. I'm very interested in knowing if this percentage goes down with new submissions after the changes we rolled out.

Comment 10

[Mark Reid [:mreid]]

I just ran the counts again (grouped by 'lastPingDate'):

            day    present     absent
            ---    -------     ------
     2013-02-01       3698       2126
     2013-02-02       1748       1646
     2013-02-03       1194       1286
     2013-02-04       1010       1030
     2013-02-05       1113       1288
     2013-02-06       1048       2249
     2013-02-07        892       5007
     2013-02-08       1819       4117
     2013-02-09       3556       2366
     2013-02-10       4100       1738
     2013-02-11       4923       1647
     2013-02-12       6186       1456
     2013-02-13       6965       1607
     2013-02-14       7372       1367
     2013-02-15       7294       1285
     2013-02-16       6962       1230
     2013-02-17       7635       1282
     2013-02-18       8558       1399
     2013-02-19      10351       1519
     2013-02-20      14620       1856
     2013-02-21      39711       2740
     2013-02-22      66926       3448
     2013-02-23        922         69
     2013-02-24         49          5
     2013-02-25         25          5
     2013-02-26         27          2
     2013-02-27         19          4
     2013-02-28         21          5
          total     208744      43779


I filtered just to February, though as you can see there are some bogus future dates in there too.

The ratio seems to be the worst around Feb 7 & 8, and I'm pretty sure the build where I saw the problem initially is from those dates.

Comment 11

[Gregory Szorc [:gps]]

~5% for Feb 22 still seems too high for my liking. Perhaps it's time we added the captured error messages into the payload to help debug issues like this. It might also be worth exploring other avenues for error reporting. e.g. we could pop up a notification bar on Nightly channels and ask users to file a bug attaching a saved log file for debugging. Oh, the possibilities...

Comment 12

[Gregory Szorc [:gps]]

Tomorrow's Nightly should send more errors in the payload per bug 845431. I'm optimistic this will helper answer the riddle of this bug.

Also see bug 845842 for the current leading theory.

Comment 13

[Mark Reid [:mreid]]

Here are some more recent numbers.  The ratio seems to have gone down somewhat.  It's hard to pin down by anything other than absolute numbers, since most of the good slicing and dicing would use stuff inside the appinfo section of the payload.

     2013-02-12       5826       1206
     2013-02-13       6418       1281
     2013-02-14       6682       1064
     2013-02-15       6321        914
     2013-02-16       5748        855
     2013-02-17       5883        819
     2013-02-18       6225        830
     2013-02-19       6701        786
     2013-02-20       7071        755
     2013-02-21       7266        696
     2013-02-22       7157        705
     2013-02-23       7255        676
     2013-02-24       7521        656
     2013-02-25       8238        693
     2013-02-26       8481        661
     2013-02-27       8894        677
     2013-02-28       8741        732
     2013-03-01       9010        718
     2013-03-02       9830        643
     2013-03-03      15640        736
     2013-03-04      20499        781
     2013-03-05      28739        902
     2013-03-06      73336       1404
     2013-03-07     177759       1946

Comment 14

[Gregory Szorc [:gps]]

I almost wonder if it is worth treating app info as a one-off and moving it to the top of the document...

Comment 15

[Mike Connor [:mconnor]]

Is it possible we still have older builds submitting here?  If we filter to build IDs newer than the latest fixes are we still seeing these?

Comment 16

[Gregory Szorc [:gps]]

(In reply to Mike Connor [:mconnor] from comment #15)
> Is it possible we still have older builds submitting here?  If we filter to
> build IDs newer than the latest fixes are we still seeing these?

We can't filter on build IDs because the build ID is part of the missing appinfo section. See comment #14.

Comment 17

[u279076]

Is this still reproducible? I'm seeing an org.mozilla.appInfo.appinfo child in my current payload with Firefox Aurora 21.0a2 2013-03-25.

Comment 18

[Mark Reid [:mreid]]

Here are some updated counts.  I'm going to run another job to see what we are getting the recently added top-level appinfo.

            day    present     absent
            ---    -------     ------
     2013-03-14      21785        384
     2013-03-15      21423        390
     2013-03-16      21050        380
     2013-03-17      21472        405
     2013-03-18      23370        435
     2013-03-19      24870        469
     2013-03-20      25127        480
     2013-03-21      26348        509
     2013-03-22      27834        534
     2013-03-23      28517        506
     2013-03-24      44956        633
     2013-03-25     165492       1190
     2013-03-26     106050        660
     2013-03-27       1365          5

Comment 19

[Mark Reid [:mreid]]

Attachment: Report of missing appinfo sections by submission date and build id.

Here's an updated report on the missing appinfo sections.

Let me know if it would be more useful to break it down by something other than buildid.

Comment 20

[Gregory Szorc [:gps]]

I filtered this on version 2 payloads and the problem essentially does not exist. The percentage of v2 payloads missing appInfo is like 0.05%. This roughly corresponds with the percentage of payloads that have errors. I'm not too worried about this going forward. Unless this presents on newer builds with version 2 payloads, I'm inclined to say this isn't a problem.

Comment 21

[Alessio Placitelli [:Dexter]]

FHR is going away per bug 1209088, wontfix.