1369743 - Create a lua_sandbox_extension decoder for the Pioneer data

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Mike Trinkala [:trink]]

Decoder for the Kafka input to handle the decryption and validation of the data.  The configuration will be provided to Ops through a secure channel as it contains the decryption key. A sample configuration will be provide in the decoder documentation.

Comment 1

[Mike Trinkala [:trink]]

https://github.com/mozilla-services/lua_sandbox_extensions/commit/861bcad09c9007b35659de75d03268b368935b59

Comment 2

[Mike Trinkala [:trink]]

The specification has been altered:

The telemetry end point will be used and the study data will be contained within a payload element [encryptedData] see: https://github.com/hannosch/mozilla-pipeline-schemas/blob/91e3cd8483f48dfd7040dda504264280f5d11ec4/schemas/telemetry/pioneer-study/pioneer-study.4.schema.json. 

The decoding steps are now: 
1) parse the telemetry JSON envelope
2) validate the envelope schema
3) move the metadata to top level Fields in the Heka message
4) decrypt the submission [encryptedData] using the key specified by encryptionKeyId
5) parse the decrypted JSON data
6) validate the decrypted data using the schema matching studyName/studyVersion
7) on success, the decrypted JSON is placed in the 'submission' field.
7a) on error, we discussed discarding all messages (i.e., no error stream) but if we leverage the standard telemetry error mechanism the error stream will only contain the original encrypted message plus any metadata allowing for standard error monitoring and alerting.  We will also need a message without the decrypted payload to track overall volume (it could be an aggregate summary message but for the time being I would prefer a one to one with full metadata). Note: the DWL will have to be configured to not forward any validated (decrypted) pioneer messages to the CEP.

Comment 3

[Mike Trinkala [:trink]]

https://github.com/mozilla-services/lua_sandbox_extensions/commit/e6d2887cb97f00a7de30a535a4661a36f3a6998e