1521035 - Event pings are not being sent to the server on Mac OS

Status: RESOLVED WORKSFORME

(Toolkit :: Telemetry, defect)

Description

[Daniel Bodea [:danibodea]]

Note

• The event pings are not being sent to the server, can't be validated server-side.

Affected versions

• tested on Nightly v66.0a1

Affected platforms

• tested on Mac OS 10.14.2 (MacBook Pro 2017 13")

Steps to reproduce

1. Create a Normandy pref experiment recipe (or any other).
2. Open the browser with the right preconditions so that the recipe is fetched and executed.
3. Go to about:telemetry, Current ping data, Select Events tab, Select "dynamic" from the drop-down (instead of parent) and confirm that the event action is correctly displayed/recorded.
4. Quit the browser and relaunch.
5. Go to about:telemetry, Archived ping data, ping type: event, Raw Payload tab and confirm that the events are properly displayed/recorded in pings.
6. Quit the browser from the dock and wait for at lest a day.
7. Go to https://sql.telemetry.mozilla.org/queries/new and create a new query with the following code snippet format:

SELECT *
FROM EVENTS
WHERE event_category='normandy'
AND submission_date_s3 > '20190110'
AND client_id LIKE '824f1d6f-4616-264f-a066-c961e9afc241'

*With the correct date (a date previous to the date of the events being executed in the format YYYYMMDD;
*And the correct cliendID, the value of the pref "toolkit.telemetry.cachedClientID", taken from the profile that executed the recipe.
*And execute it.

Expected result

• The Normandy events should be displayed in the results section after the query is completed.

Actual result

• No results are being displayed in the results section after the query is completed.

Additional notes

• I could not yet find the cause of this issue so I logged this bug to cover the test objectives of the "Normandy - pref experiment as local action" feature, before the next sign-off. I can test any situation that developers might suggest in order to find the cause afterward. Please need info me.
• This issue is not caused by using the restart feature.

Comment 1

[Daniel Bodea [:danibodea]]

The following queries are saved with different clientIDs, taken from different profiles that have pings displayed locally (in about:telemetry, Archived ping data, ping type: event, Raw Payload tab):
https://sql.telemetry.mozilla.org/queries/60993/source
https://sql.telemetry.mozilla.org/queries/60995/source
https://sql.telemetry.mozilla.org/queries/60996/source
https://sql.telemetry.mozilla.org/queries/60997/source
https://sql.telemetry.mozilla.org/queries/60998/source
https://sql.telemetry.mozilla.org/queries/60999/source
https://sql.telemetry.mozilla.org/queries/61000/source

Hope this helps. Need info me to attempt new steps to reproduce or to ask any other information about the device/OS/procedure.

Comment 2

[Daniel Bodea [:danibodea]]

It appears that the pings were finally recorded on the server and the query shows them correctly, so this bug is definitely WORKSFORME after all, also because a normal user will use his browser with the same profile, so the pings will eventually be sent to the server in normal scenarios.

The pings were probably sent to the server only so late because these profiles were reopened (a few hours ago and/or about 24 hours ago). Some of these profiles were used about a week ago to run the feature's test cases.

Furthermore, We need to understand exactly how the pings get sent to the server, so we know what to avoid the next time we need to validate events server-side. @chutten, can you help us with that?

Comment 3

[Chris H-C :chutten]

The steps of a ping in Firefox Desktop are (broadly) these:

1. Assemble: Code, almost-exclusively JS, assembles together all the pieces of data for the ping's payload
2. Submit: The code calls TelemetryController.submitExternalPing. Telemetry code takes over, adding Telemetry-specific stuff (Environment, client id, common ping data).
3. Archive: If the Telemetry Archive is enabled (and there's no reason it shouldn't be) we save a copy of this ping document to disk. It can be found in the profile dir under datareporting/archived/, arranged by month.
4. Pend: We'll try sending this ping immediately, but in case something goes wrong we record that we intended to but didn't get a chance to send it. This usually means it ends up copied to the profile dir under saved-telemetry-pings/
5. Send: We almost always try to send the ping immediately using Firefox's network stack. Sometimes we use the pingsender instead, and some times we wait a little while before sending it.
6. Retry: If the send fails, we retry until we think we succeed (or until we evict it from pending due to size limitations or something).

You'll notice that Steps 5 and 6 require an active Firefox so we can use its network stack to send stuff. This means if the ping wasn't sent before Firefox shut down, we have to wait for Firefox to start back up again before we can retry.

To catch a ping exactly as it's sent (for the purposes of testing, for example) you could

• Rely on the ping being sent appropriately and simply read the archived ping from the disk. Telemetry has proven quite good at being a reliable data channel, so don't feel you need to test that (unless that's within the scope of the test, of course).
• Use a utility like gzipServer to run an ingest server locally. Configure toolkit.telemetry.server to point at it and the server will log a line and save a file each time it receives a ping.