1381555 - Validate environment.experiments during ingestion

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Wesley Dawson [:whd]]

We assume in https://github.com/mozilla-services/lua_sandbox_extensions/blob/master/moz_telemetry/sandboxes/heka/output/moz_telemetry_s3.lua#L193-L207 that environment.experiments has a certain structure if it is valid JSON. This can result in a type error when processing, as in the wild there has been at least one message for which the resultant "exp" was a number instead of a table.

We should update the output code or the schema, preferably the schema as this is why it exists.

Comment 1

[Mark Reid [:mreid]]

Let's update the schema.

Comment 2

[Mark Reid [:mreid]]

Attachment: 1381555.patch

Alessio, is this something you can take care of?

I was hoping to get to it before PTO, but I ran out of time to test it.

I think we need to add a section to the "environment" chunk like in the attached patch.

Comment 3

[Alessio Placitelli [:Dexter]]

(In reply to Mark Reid [:mreid] from comment #2)
> Created attachment 8888019 [details] [diff] [review]
> 1381555.patch
> 
> Alessio, is this something you can take care of?

Sure!

> I was hoping to get to it before PTO, but I ran out of time to test it.
> 
> I think we need to add a section to the "environment" chunk like in the
> attached patch.

Ok!

Comment 4

[Alessio Placitelli [:Dexter]]

Attachment: Schema update PR

Comment 5

[Alessio Placitelli [:Dexter]]

This was merged: https://github.com/mozilla-services/mozilla-pipeline-schemas/commit/84d84813f959d7dff0cf96253906dcaf5c1d3b1d