830503 - More selective enabling of Firefox Health Report

Status: RESOLVED WONTFIX

(Firefox Health Report Graveyard :: Client: Desktop, defect)

Description

[Gregory Szorc [:gps]]

We currently blanket enable Firefox Health Report (MOZ_SERVICES_HEALTHREPORT=1) in /browser/confvars.sh. We should consider not enabling FHR in a) debug builds b) non-official builds. We probably don't want local builds populating the Metrics data set, right?

Comment 1

[(no longer active)]

IIRC the telemetry folks have solved this problem once before.

Comment 2

[Vladan Djeric (:vladan)]

That's right, we don't want non-official builds submitting data. You can reuse a lot of the Telemetry code for this. 

Btw, is FHR submission disabled on tryserver?

Comment 3

[Gregory Szorc [:gps]]

(In reply to Vladan Djeric (:vladan) from comment #2)
> Btw, is FHR submission disabled on tryserver?

Not exactly. We prevent the notification info bar from being displayed in automation.py.in and various other places. However, there is nothing that explicitly prevents data submission. That being said, data submission cannot occur until at least 24h after first run, so unless you have something that takes 1d or mucks with the system clock, you'll never see FHR submission during automation. Even if we did see submission during automation, in-flight uploads (should) have no bearing on shutdown. And, submissions would likely fail and cause FHR to simply try again later. (We have a back-off mechanism in the client.)

Comment 4

[Daniel Einspanjer [:dre] [:deinspanjer]]

I'd like to question whether we definitely don't want submissions from non-official builds.  The critical aspect is whether we can easily distinguish them from official builds.  If the brand-name, channel, or distribution values are reliably distinct then we could easily exclude those submissions from metrics that should include only official builds while still providing us with the ability to answer other questions such as:

* How much long term testing / usage happens on non-official builds?
* Are profiles used with both official and non-official builds?
* Are performance / stability characteristics significantly different on non-official builds?

It might be that these questions are of little value, in which case we are better off avoiding submission, but since comment #3 mentions that very short lived sessions are unlikely to ever submit data at all, it is potentially useful for us to be able to observe and consider data from these builds if they are actually interacting with the internet on a regular basis.

Comment 5

[Vladan Djeric (:vladan)]

I consider "non-official" builds to be try-builds, regular mozilla-inbound builds, and local developer builds. Nightly/Aurora/etc are termed official builds afaik.

With that said, I don't think FHR data from non-official builds will provide us with reliable answers to Daniel's questions.

> * Are performance / stability characteristics significantly different on
> non-official builds?

We already know building in debug config significantly affects performance and stability (e.g. asserts). Even disabling PGO significantly degrades startup & JS benchmarks. 
Non-official developer builds (try builds of patches & dev builds) will exhibit completely skewed usage patterns. There will be crashes, hangs and bugs that don't exist in release builds. This is especially the case for local builds, e.g. devs setting  breakpoints and debugging their code.
There's also the skew added by the significantly more powerful and atypical developer & test machine hardware.

> * How much long term testing / usage happens on non-official builds?

I think the reports would be hard to interpret. I suspect the amount of uptime experienced by a given build has a very weak correlation with the amount & quality of testing done. I don't think there is any real long-term usage of non-official builds.

> * Are profiles used with both official and non-official builds?

Could you clarify this point?

> It might be that these questions are of little value, in which case we are
> better off avoiding submission, but since comment #3 mentions that very
> short lived sessions are unlikely to ever submit data at all, it is
> potentially useful for us to be able to observe and consider data from these
> builds if they are actually interacting with the internet on a regular basis.

If we're unlikely to ever submit FHR data from our build servers, we should make the behavior deterministic and just disable FHR reporting in that environment

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

I had a bit of a different interpretation for non-official builds.  I think I can easily agree that try-builds, mozilla-inbound builds, and local dev debug builds would be difficult targets from which to extract useful analysis.

I was thinking more about branch builds such as nightly-ux as well as third party builds.

Comment 7

[Gregory Szorc [:gps]]

I'm going to reverse my original opinion: I think we should default to enabled everywhere.

As the maintainer of this feature, I want it enabled in as many places as possible so the testing surface area is as large as possible. If bugs in FHR are found in local developer builds, this is a good thing.

I believe the only real consideration is whether data submission should occur. Metrics may not want data from non-official builds on their servers. And, said data might not mean much.

I believe the importance of a similar-to-production build outweighs the possible insignificance of uploaded data. That leaves "unwanted" data on servers. As Daniel said in comment #4, it sounds like it's pretty easy for Metrics to filter out submissions from these builds by looking at the channel and build info. So, I don't see any concerns here.

In the interest of having local builds more closely match the shipping product, I'm marking this WONTFIX.