1767511 - Many messaging-system/onboarding pings missing required "event" identifier

Status: RESOLVED FIXED

(Firefox :: Messaging System, defect)

Description

[Jeff Klukas [:klukas] (UTC-4)]

Starting on 2022-04-21, we started seeing a significant number (~100k/day) of onboarding_v1 pings rejected with error:

org.everit.json.schema.ValidationException: #: required key [event] not found 

The vast majority have user_agent ending in Firefox/100.0 so likely a new issue in Firefox 100.

?ni :nanj Do you think this could be a new code change?

Comment 1

[Nan Jiang [:nanj]]

Yeah, it looks like that's the case.

Looping Punam in for further investigation.

Comment 2

[Punam Dahiya [:pdahiya]]

@klukas Does onboarding_v1 pings rejected with an error means telemetry data isn't getting saved for these pings, trying to understand whats the possible fall out of these failing validations.

Including @barret to help confirm if its related to https://bugzilla.mozilla.org/show_bug.cgi?id=1752665 fix thanks!

Comment 3

[Jeff Klukas [:klukas] (UTC-4)]

The failing validation means that the pings don't show up in BigQuery for analysis (in table messaging_system_stable.onboarding_v1). Instead they get routed to an error output table in BigQuery that has highly restricted access. It is possible to backfill from errors to analysis tables, but this requires about a day of data engineering effort.

Comment 4

[Beth Rennie [:beth] (she/her)]

Bug 1752665 only sends experiment unenrollment pings and did not change how it computes the event object, only the reason extra value. It is unlikely to be related to this bug.

Comment 5

[Ed Lee :Mardak]

I think bug 1764064 didn't actually send the correct telemetry.

Showing a legacy spotlight results in…

{
	"experiments": {},
	"locale": "en-US",
	"version": "102.0a1",
	"release_channel": "default",
	"message_id": "TCP_SPOTLIGHT_MESSAGE_95",
	"event": "IMPRESSION",
	"addon_version": "20220503155751",
	"client_id": "bbe54017-2b41-4b2e-bf1e-637ca49fbf92",
	"browser_session_id": "de6abb25-c60f-4ecd-b311-5d3ada969d0d"
}

Triggering the Fx100 upgrade spotlight results in two pings neither have event at the top level…

{
	"experiments": {},
	"locale": "en-US",
	"version": "102.0a1",
	"release_channel": "default",
	"type": "SPOTLIGHT_TELEMETRY",
	"data": {
		"action": "spotlight_user_event",
		"event": "IMPRESSION"
	},
	"addon_version": "20220503155751",
	"client_id": "bbe54017-2b41-4b2e-bf1e-637ca49fbf92",
	"browser_session_id": "de6abb25-c60f-4ecd-b311-5d3ada969d0d"
}

{
	"experiments": {},
	"locale": "en-US",
	"version": "102.0a1",
	"release_channel": "default",
	"type": "IMPRESSION",
	"data": {
		"content": {
			"template": "multistage",
			"id": "FX_100_UPGRADE",
			"transitions": true,
			"screens": [{
				"id": "UPGRADE_PIN_FIREFOX",
				"content": {
… the rest of the message configuration
			}]
		}
	},
	"addon_version": "20220503155751",
	"client_id": "bbe54017-2b41-4b2e-bf1e-637ca49fbf92",
	"browser_session_id": "de6abb25-c60f-4ecd-b311-5d3ada969d0d"
}

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1764064

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

:mviar, since you are the author of the regressor, bug 1764064, could you take a look?
For more information, please visit auto_nag documentation.

Comment 8

[Meg Viar [:mviar]]

I'll put together a patch to restructure the Spotlight ping so that the event field is at the top level, not nested under data.

Comment 9

[Meg Viar [:mviar]]

Attachment: Bug 1767511 - Many messaging-system/onboarding pings missing required event identifier

Comment 10

[Ryan VanderMeulen [:RyanVM]]

What's the impact of this bug in the wild? Is this something we need on the radar for 100 still?

Comment 11

[Meg Viar [:mviar]]

:RyanVM the impact would be "DISMISS" events from uses of the multistage spotlight modal (like the 100 upgrade dialog) ending up in an error output table in BigQuery. To be clear, we are successfully capturing events for clicking the "X" dismiss button and the primary or secondary buttons (which can also dismiss the modal). We only rely on the "DISMISS" event mentioned here for cases where the user dismisses another way, like pressing ESC.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Sounds like we don't need to worry about this for 100 then. I'll leave it up to you to decide about nominating for Beta101 uplift eventually or letting it ride the trains.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/39beeea2f4cb
Many messaging-system/onboarding pings missing required event identifier r=Mardak

Comment 14

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/39beeea2f4cb