1612940 - Glean pings for `org-mozilla-fenix` contain unusual number of errors under `#/ping_info/experiments/web-render-performance-comparison-experiment/extra`

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Anthony Miyaguchi [:amiyaguchi]]

As of 2020-02-03, the schema errors by namespace and document shows that #/ping_info/experiments/web-render-performance-comparison-experiment/extra is causing rejection for pings under org-mozilla-fenix at rank 24, 29, 30, 32, and 34 with errors around the 50k mark over the last 28 days.

Comment 1

[Jeff Klukas [:klukas] (UTC-4)]

The specific error looks to be:

org.everit.json.schema.ValidationException: #/ping_info/experiments/webrender-performance-comparison-experiment/extra: expected type: JSONObject, found: Null

A relevant snippet of payload JSON is:

"experiments":{"webrender-performance-comparison-experiment":{"branch":"disable_webrender","extra":null}}

So the problem is that the client is explicitly sending "extra":null where the glean schema does not allow nulls for extra.

Comment 2

[Michael Droettboom [:mdroettboom]]

I'll look at this from the point of view of the Glean SDK...

Comment 3

[Michael Droettboom [:mdroettboom]]

I think this is definitely a bug in Glean SDK. It needs to be updated to leave out extra when there are no values. However, rolling this out into Fenix will take some time.

I would propose that we allow this in the schema, wait for the fit in Glean SDK to roll out, and then (optionally) disallow it in the schema again.

Comment 4

[Michael Droettboom [:mdroettboom]]

Attachment: GitHub Pull Request

Comment 5

[Michael Droettboom [:mdroettboom]]

Attachment: GitHub Pull Request

Comment 6

[Alessio Placitelli [:Dexter]]

(In reply to Michael Droettboom [:mdroettboom] from comment #3)

I think this is definitely a bug in Glean SDK. It needs to be updated to leave out extra when there are no values. However, rolling this out into Fenix will take some time.

I would propose that we allow this in the schema, wait for the fit in Glean SDK to roll out, and then (optionally) disallow it in the schema again.

Yes, good catch. I agree with the plan!

Comment 7

[Jeff Klukas [:klukas] (UTC-4)]

We are successfully ingesting some pings from the affected experiment now, as shown by querying the live table:

SELECT
  DATE(submission_timestamp) AS submission_date,
  COUNT(*) AS n
FROM
  `moz-fx-data-shar-nonprod-efed.org_mozilla_fenix_live.baseline_v1`,
  UNNEST(ping_info.experiments) AS experiment
WHERE
  DATE(submission_timestamp) >= "2020-02-01"
  AND experiment.key = 'webrender-performance-comparison-experiment'
GROUP BY
  1

Previous days return no rows, but we have 347 pings so far today.