Closed Bug 1680025 Opened 4 years ago Closed 4 years ago

Consider to disable FOG in developer builds?

Categories

(Toolkit :: Telemetry, task, P1)

Product:
Toolkit
Component:
Telemetry
Type:
task
Points:
1

Tracking

()

Status:
RESOLVED FIXED
Milestone:
87 Branch
Tracking Flags:
Tracking Status
firefox87 --- fixed

People

(Reporter: janerik, Assigned: chutten)

Details

(Whiteboard: [telemetry:fog:m?])

Attachments

(1 file)

Bug 1680025 - Disable FOG ping upload in developer builds r?janerik!
48 bytes, text/x-phabricator-request
Details | Review

Currently FOG's only preference to control its behavior is datareporting.healthreport.uploadEnabled, which is shared with the old Telemetry system.

However, old Telemetry also has the build-time constant MOZ_TELEMETRY_REPORTING which is not set on local developer builds, but is set for shippable builds ("offiicial builds").
Not-official builds then disable sending any telemetry data.

FOG doesn't care (right now).

Do we need to?

Note: this means that local builds are sending FOG telemetry by default.

The Glean SDK has no distinction between "cannot send" and "cannot record" and provides exactly one lever to control both: set_upload_enabled. If we disable recording on developer-built Firefox, then writing tests for new collection becomes rather tricky. If we don't disable recording, we're enabling sending, and there's just nothing to be done about that...

...except. These days we now have a way to signal at runtime that sending should shortcircuit: setting telemetry.fog.test.localhost_port to a negative value. So I suppose we have built ourselves a way to emulate that behaviour at the FOG level.

Going back in history to the design phase of FOG, the going assumption then was that we need not care. Developer-built Firefoxen have a different channel (default) and so can be ignored easily by analyses. If the volume ever got high enough to be bothersome, that would mean we have a bigger problem than just the data being sent and should be grateful we received the data so we could diagnose it : )

Has this assumption changed?

Flags: needinfo?(jrediger)

(In reply to Chris H-C :chutten from comment #2)

Has this assumption changed?

I think there's another assumption that's not being discussed here: local developers builds won't send telemetry. The reason why we don't send legacy telemetry from developer build is not that we can't filter it at analysis time. It's because we want "no surprises" for contributors or developers.

(In reply to Alessio Placitelli [:Dexter] from comment #3)

I think there's another assumption that's not being discussed here: local developers builds won't send telemetry. The reason why we don't send legacy telemetry from developer build is not that we can't filter it at analysis time. It's because we want "no surprises" for contributors or developers.

In a current build of Firefox, we send and receive a lot of telemetry even if we aren't recording and sending Telemetry. If this assumption is still valid, what makes Telemetry (or Glean) different from the other services sending and receiving data that aren't surprising?

Regardless of that, we could implement this. We could export MOZ_TELEMETRY_REPORTING as a feature to FOG's Rust build and, if telemetry.fog.test.localhost_port is unset, pretend that telemetry.fog.test.localhost_port is set to a negative value which pretends to the Glean SDK that everything has sent properly when in reality we've just dropped the pings on the floor.

This would support local integration tests (since they set localhost_port) and local unit testing... but prevents manual testing. If I add instrumentation and want to see if it recorded an expected value, I can't trigger pings in about:glean and view them in the Glean Debug Ping Viewer. The SDK would need to advertise to the Uploader that a ping was debug-tagged, and then we could rule that debug-tagged pings were allowed to go through.

Is this the route you think we should walk, Alessio? Jan-Erik?

Or does this underscore a need for an about:telemetry-like local data viewer?

Flags: needinfo?(alessio.placitelli)

(In reply to Chris H-C :chutten from comment #4)

(In reply to Alessio Placitelli [:Dexter] from comment #3)

I think there's another assumption that's not being discussed here: local developers builds won't send telemetry. The reason why we don't send legacy telemetry from developer build is not that we can't filter it at analysis time. It's because we want "no surprises" for contributors or developers.

In a current build of Firefox, we send and receive a lot of telemetry even if we aren't recording and sending Telemetry. If this assumption is still valid, what makes Telemetry (or Glean) different from the other services sending and receiving data that aren't surprising?

I'm not sure about the other services you mention, they probably have good reasons to do that or have requested permissions (-:

Is this the route you think we should walk, Alessio? Jan-Erik?

I think we should just be explicit about changing this so that, if we do it, we can reference back to this back with the rationale. You raise a good point about Telemetry not being "different" compared to other services. But then why did we have this requirement in the first place? And who is a good person to ask for relaxing it?

Moreover, what other services are you talking about?

Flags: needinfo?(alessio.placitelli)

If you start up a more-or-less-vanilla build of Firefox, it grabs locale-specific newtab recommendations (including spocs) to display. I haven't dug into the networking side, but I presume this comes about due to an exchange of information.

There's also the "accidental telemetry" leaked from blocklist and friends that we all know and love, but that's not the same thing as telemetry itself, so I'm not counting it here.

As for who can okay this change... well, IANAL. But Emily is, so let's ask.

Emily: At present, if a developer builds a Firefox on their own machine, Telemetry is able to record but not send any data. Some changes we're planning (Project FOG) will, unless we expend some effort, default to being able to record and send data. Is this an okay change to make?

Flags: needinfo?(emily)
Flags: needinfo?(jrediger)

Hi! A few questions. Thanks in advance for answering them!

  1. Would the data that's being recorded and sent be from the developer only or anyone they shared their build with?
  2. If we were to record telemetry, is there any way to partition it so that Mozilla doesn't use it for its purposes? I ask because, technically, if I understand correctly, if a developer builds their own version of the browser, they/users who use their version, won't agree to the Fx Privacy Notice, be able to delete it, etc.

Emily

(In reply to Emily Litka from comment #7)

Hi! A few questions. Thanks in advance for answering them!

  1. Would the data that's being recorded and sent be from the developer only or anyone they shared their build with?

Anyone they share that build with. For the purposes of data collection in Firefox, it'll send the same data that a mozilla-built Firefox would (though with slightly different values).

Interestingly, this is analogous to how some Linux flavours distribute their own Firefox builds. Some developer builds it on their machine and distributes the built artefacts widely. Given the similarity between that workflow and local developer workflows, we've had problems in the past getting good telemetry from Linux.

  1. If we were to record telemetry, is there any way to partition it so that Mozilla doesn't use it for its purposes? I ask because, technically, if I understand correctly, if a developer builds their own version of the browser, they/users who use their version, won't agree to the Fx Privacy Notice, be able to delete it, etc.

We can usually identify a developer-built Firefox by checking for app_channel = 'default'. Devs tend not to customize the release channel so it remains as default which isn't the case for any build Mozilla distributes (or even builds we use in testing. They get special channel names like nightly-try). Analysts tend to specify the specific channel they're interested in, so this data isn't mixed into analysis about "Real Users". However, we aren't partitioning it in its own place (like we do to handle data sent from automation).

And if we want to add additional collection to identify it, we could send the value of MOZILLA_OFFICIAL, which is the current switch in the codebase for determining the difference between "built for me" and "built for others".

As for the Privacy Notice part: unless the developer goes to some effort to disable it, on first run the Firefox Privacy Notice is still shown. The ability to opt out is still present in the Preferences, which will be able to send the "deletion-request" ping just as easily.

Emily: Does this help?

Is there are historical reason (that you're aware of) for why telemetry was disabled for developer builds?

Flags: needinfo?(emily)

(In reply to Emily Litka from comment #9)

Is there are historical reason (that you're aware of) for why telemetry was disabled for developer builds?

I wasn't there, so I can't say for certain. Certainly the product engagement metrics acquired from developer builds are useless for product planning. Most performance metrics as well, considering we often don't know just how the build was built (e.g. maybe the dev swapped out the networking stack for something way slower). The data isn't useful at present, and with a default of "we don't collect it" it never will be.

Also, our contributor base was pretty darn data-averse at the time. I think opinions have softened slightly, but maybe that's still the case.

Also also: if I had to guess, our testing infra might be part of it too. Tests we run are broadly forbidden from accessing the network. Doing so to send Telemetry causes a crash, and a test failure. (We've solved this for FOG in the way that we'd leverage if we had to solve it for developer builds too).

In summary: the data we'd receive likely wouldn't be useful, and it might annoy contributors and cause test failures.

Put this way, it certainly sounds as though even if we were allowed to collect this, we probably still don't want to if there's an easy way to do that which doesn't break local testing.

Flags: needinfo?(emily)

Hi, this is fine from a legal perspective. However, a few things:

  • we need to make sure we remove any reference to not collecting telemetry before we do so (for example, here: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/start/adding-a-new-probe.html - we say "Every build of Firefox has Telemetry enabled. Local developer builds with no custom build flags will record all Telemetry data, but not send it out.").
  • we should probably provide some notice (blog post, snippet, etc.) that informs developers of this change before collection begins. This would be consistent with our Privacy Principle of "no surprises" and would alert them that they need to opt-out if they don't want the data sent.
  • if they opt-out, would that stop the sending of telemetry?
  • since telemetry could be collected, could studies be deployed on their builds?

It sounds like the larger question is a policy one, as Chutten described - i.e., Will Mozilla upset the developers with this change. I'll defer to you on whether this makes sense from a product perspective.

(In reply to Emily Litka from comment #11)

Hi, this is fine from a legal perspective. However, a few things:

Good catch!

  • we should probably provide some notice (blog post, snippet, etc.) that informs developers of this change before collection begins. This would be consistent with our Privacy Principle of "no surprises" and would alert them that they need to opt-out if they don't want the data sent.

Yes, most definitely.

  • if they opt-out, would that stop the sending of telemetry?

Yes. There's always a checkbox you can uncheck in the Preferences. But it would also make their tests stop working.

  • since telemetry could be collected, could studies be deployed on their builds?

Interestingly, in developer-built Firefox, Studies are enabled by default! (I was surprised too)

It sounds like the larger question is a policy one, as Chutten described - i.e., Will Mozilla upset the developers with this change. I'll defer to you on whether this makes sense from a product perspective.

Yes, and I'm beginning to think the better choice from that POV is indeed to find some room to configure it similarly to Telemetry for the comfort of familiarity. Definitely no matter which way we go on this it will require some careful investigation (like into why my optout pref is locked as already off in my dev build) and communication.

Thank you for your help!

Flags: needinfo?(emily)

Since we're nearly at the point where we'll start enabling builtin pings, we should sort this out now.

My intent is to, if MOZILLA_OFFICIAL isn't set, have FOG's uploader lie to the Glean SDK and tell it that its pings are being sent successfully, when really we're tossing them into the bin. I will document this in the Preferences documentation (which I may rename to "Preferences and Defines"). It will not be explicitly tested because compiler defines aren't that testable... but given it triggers the same codepath as the test-only pref telemetry.test.fog.localhost_port (when its value is negative), we at least have coverage on the path if not the condition. (that I'll manually test locally)

Assignee: nobody → chutten
Severity: -- → N/A
Status: NEW → ASSIGNED
Type: defect → task
Points: --- → 1
Priority: P3 → P1
Pushed by chutten@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f37a4a21c8e8
Disable FOG ping upload in developer builds r=janerik

https://hg.mozilla.org/mozilla-central/rev/f37a4a21c8e8

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
status-firefox87: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 87 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: