1651340 - Provide upload "off the main process" as a service (e.g. Glean's pingsender)

Status: NEW

(Data Platform and Tools :: Glean: SDK, enhancement, P4)

Description

[Alessio Placitelli [:Dexter]]

Our Python language bindings offer a way to upload pings using a separate process.

FOG will require a similar technique for reducing the latency of pings and to send them when Firefox shuts down.

FxR PC has a similar requirement: the initial runner quickly generates a ping and then quits the application. Ideally, it would spin up a separate process and make it do the uploading part. However, creating a new process on Windows requires an executable to be launched, since Windows does not have fork.

I believe that the Glean SDK should standardize the way off-the-main process uploading works, providing a consistent behaviour and a safe API to use it.

Requirements

• Build as a separate binary that's shipped along with the SDK, for all the supported platforms (Windows, Linux, MacOS, but not mobile platforms).
• Use the glean upload API.

Constraints

• Glean will be used in Firefox with project FOG, which means the ping sending binary should be:
  • very small (the current one is < 80kb on Windows x64);
  • should dynamically load libcurl on linux/mac and use Wininet on windows
• Glean should provide public APIs to customize the off-the-main-process upload experience (e.g. make it work like on Python, by having an always on child process started with Glean that communicates via IPC or have normal off the main thread upload and then trigger an off-the-main-process binary when shutting down, for best effort upload)
• Make it impossible (or hard!) to use the ping sending binary by external consumers, manually. The ping sending binary will be meant as a Glean SDK implementation detail to never be used directly by consumers.

Comment 1

[Jan-Erik Rediger [:janerik]]

Some quick notes so they don't get lost:

Build as a separate binary that's shipped along with the SDK, for all the supported platforms (Windows, Linux, MacOS, but not mobile platforms).

This is going to be tricky part and might differ per implementation (because of different build systems, delivery methods, ...)

Glean will be used in Firefox with project FOG, which means the ping sending binary should be:

• very small (the current one is < 80kb on Windows x64);
• should dynamically load libcurl on linux/mac and use Wininet on windows

That are the current constraints. These are not written in stone.
If the binary does more we can expect it to be larger.
It's still certainly something we should keep in mind though.

Comment 2

[Chris H-C :chutten]

I have Thoughts and Feelings:

• "At-shutdown" sending is a Dark Pattern we should avoid as much as possible. In Firefox Telemetry that has been the cause of grief in every way from Performance to Data Engineering to Analysis.
  • Telemetry's in-memory nature forced this pattern onto us, so there may be some institutional inertia behind this pattern that we'll have to fight. If this off-process sending mechanism makes this easy, expect someone to use it.
• I see this problem as having consumers where the instrumenting process cannot be the uploading process. This means we want to split the Glean SDK stack into two processes, which we can do at any layer between "just behind the API" (an IPC model a la Project FOG) to "just above the uploader" (an Uploader Binary model a la Pingsender). In Project FOG we determined that the best way to handle this was an IPC model, not a Pingsender model, but not every embedding application is Firefox Desktop so the calculus resulting in that decision might need to be rerun.
• Project FOG's requirements do not include at-shutdown sending. The design at present (and I hope into the future) assumes that pings will be sent eventually because the main process will live long enough to make network connections (it is a browser after all). If there is data captured late in the application that cannot be sent by the end of the session, then it'll be sent next session.
• I wholeheartedly approve of making any internal split difficult or impossible for outside consumers to use. In FOG we accomplish this by handing opaque byte arrays that are serialized private structs. (( Oh wait, I just noticed it isn't private at the moment. Gonna have to look into that. ))
• If we're getting into the "Glean SDK is process aware" game, how deep do we want to get? e.g. Is this a sign that we should revisit the "Glean SDK is process unaware" design decision?

Comment 3

[Michael Droettboom [:mdroettboom]]

:chutten -- any additional details to add from this week's discussion on this topic?

Comment 4

[Chris H-C :chutten]

My mind's a-whirling with a few contradictory ideas but nothing I can put down in text. For instance, I'm not sure whether FOG's cases of "crash" pings and "downgrade" pings are the same or are different. I'm going to let this stir a little while longer.