Validate environment.experiments during ingestion

RESOLVED FIXED

Status

Data Platform and Tools
Pipeline Ingestion
P1
normal
RESOLVED FIXED
5 months ago
5 months ago

People

(Reporter: whd, Assigned: Dexter)

Tracking

Details

(Whiteboard: [measurement:client])

Attachments

(2 attachments)

(Reporter)

Description

5 months ago
We assume in https://github.com/mozilla-services/lua_sandbox_extensions/blob/master/moz_telemetry/sandboxes/heka/output/moz_telemetry_s3.lua#L193-L207 that environment.experiments has a certain structure if it is valid JSON. This can result in a type error when processing, as in the wild there has been at least one message for which the resultant "exp" was a number instead of a table.

We should update the output code or the schema, preferably the schema as this is why it exists.

Comment 1

5 months ago
Let's update the schema.
Points: --- → 1
Priority: -- → P2

Comment 2

5 months ago
Created attachment 8888019 [details] [diff] [review]
1381555.patch

Alessio, is this something you can take care of?

I was hoping to get to it before PTO, but I ran out of time to test it.

I think we need to add a section to the "environment" chunk like in the attached patch.
Flags: needinfo?(alessio.placitelli)
(Assignee)

Comment 3

5 months ago
(In reply to Mark Reid [:mreid] from comment #2)
> Created attachment 8888019 [details] [diff] [review]
> 1381555.patch
> 
> Alessio, is this something you can take care of?

Sure!

> I was hoping to get to it before PTO, but I ran out of time to test it.
> 
> I think we need to add a section to the "environment" chunk like in the
> attached patch.

Ok!
Assignee: nobody → alessio.placitelli
Flags: needinfo?(alessio.placitelli)
Priority: P2 → P1
Whiteboard: [measurement:client]
(Assignee)

Comment 4

5 months ago
Created attachment 8888738 [details] [review]
Schema update PR
(Assignee)

Comment 5

5 months ago
This was merged: https://github.com/mozilla-services/mozilla-pipeline-schemas/commit/84d84813f959d7dff0cf96253906dcaf5c1d3b1d
Status: NEW → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.