1556418 - Some tagged pings get lost in the GCP pipeline

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Alessio Placitelli [:Dexter]]

Attachment: logcat_trace.txt

While performing tests using the GleanDebugActivity for the metrics ping, QA noticed that while pings were showing up in the logcat as sent, they were not displayed in the dashboard.

Here's the highlights of the attached logcat trace:

• The interesting part is ping 7d82ee9e-59db-484c-8f6c-053c5a6ec1b2
• On line 6936 it's being saved

2019-06-03 13:53:05.623 4271-4349/? D/glean/PingStorageEngine: Storing ping 7d82ee9e-59db-484c-8f6c-053c5a6ec1b2 at /data/user/0/org.mozilla.fenix/glean_data/pending_pings/7d82ee9e-59db-484c-8f6c-053c5a6ec1b2

• On line 7004-7058 it's being processed & attempted a send

2019-06-03 13:53:05.668 4271-4839/? D/glean/PingStorageEngine: Processing ping: 7d82ee9e-59db-484c-8f6c-053c5a6ec1b2
[...]
2019-06-03 13:53:05.670 4271-4839/? D/glean/HttpPingUploader: Submitting ping to: https://stage.ingestion.nonprod.dataops.mozgcp.net/submit/org-mozilla-fenix/metrics/1/7d82ee9e-59db-484c-8f6c-053c5a6ec1b2

• On line 7262 you can see that it timed out (it's sending to the GCP endpoint because it's tagged)

2019-06-03 13:53:35.801 4271-4839/? E/glean/PingStorageEngine: Error processing ping file: 7d82ee9e-59db-484c-8f6c-053c5a6ec1b2

And we know it's sending to GCP:

2019-06-03 13:53:05.670 4271-4839/? D/glean/HttpPingUploader: Submitting ping to: https://stage.ingestion.nonprod.dataops.mozgcp.net/submit/org-mozilla-fenix/metrics/1/7d82ee9e-59db-484c-8f6c-053c5a6ec1b2

• On line 7507 it's attempting again

2019-06-03 13:54:06.005 4271-4920/? D/glean/PingStorageEngine: Processing ping: 7d82ee9e-59db-484c-8f6c-053c5a6ec1b2

• And finally, on line 7561-7569 it's succeeding

2019-06-03 13:54:06.353 4271-4920/? D/glean/HttpPingUploader: Ping successfully sent (200)

Comment 1

[Arkadiusz Komarzewski [:akomar]]

We have some baseline and events pings from this client, however no metrics pings were stored in BigQuery.
As a next step, we should check the Landfill for any pings from this client - this might help to narrow down failing stages.

Comment 2

[Alessio Placitelli [:Dexter]]

(In reply to Arkadiusz Komarzewski [:akomar] from comment #1)

We have some baseline and events pings from this client, however no metrics pings were stored in BigQuery.
As a next step, we should check the Landfill for any pings from this client - this might help to narrow down failing stages.

There's one additional thing to note: there was no problem while sending other ping types, the networking error only happened on the metrics ping submission. I'm not sure if that's related, but might explain why we see the other types.

Comment 3

[Arkadiusz Komarzewski [:akomar]]

Yes, this might be related if we're crashing in some unfortunate place during deduplication (as this is two-step process spread out to two jobs).

Comment 4

[Arkadiusz Komarzewski [:akomar]]

This was caused by crashing Cloud Function that processes pings for DebugView. These pings weren't passing validation which explains why they're missing from BigQuery tables.

I have pushed a fix to DebugView function, metrics pings can now be seen in the view [1].

[1] https://debug-ping-preview.firebaseapp.com/pings/1dc73c15-4a6c-4624-a24b-d6e8936a9e82/aj-testing-101