1628539 - Create beam job for decrypting pioneer v2 documents

Status: RESOLVED FIXED

(Data Platform and Tools :: Pipeline Ingestion, task, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

Pioneer v2 requires an extra decoding step in the gcp-ingestion/ingestion-beam job. The transform will decrypt messages encrypted using JOSE and decompress the payload, before passing it to the rest of the ingestion stack.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/gcp-ingestion/pull/1239

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/gcp-ingestion/pull/1246

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Add script for benchmarking Pioneer decryption #1275

Comment 4

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Update Pioneer transform to handle `pioneer-study` envelope #1276

Comment 5

[Anthony Miyaguchi [:amiyaguchi]]

The decoder can be run using the relevant options. The benchmark script demonstrates the use of the new options and verifies that the decoder runs in a batch context on dataflow. The new options are --pioneerEnabled, --pioneerMetadataLocation, --pioneerKmsEnabled, and --pioneerDecompressPayload.

./bin/mvn compile exec:java -Dexec.mainClass=com.mozilla.telemetry.Decoder -Dexec.args="\
    --runner=Dataflow \
    --profilingAgentConfiguration='{\"APICurated\": true}'
    --project=$project \
    --autoscalingAlgorithm=NONE \
    --workerMachineType=n1-standard-1 \
    --gcpTempLocation=$bucket/tmp \
    --numWorkers=2 \
    --pioneerEnabled=true \
    --pioneerMetadataLocation=$bucket/$prefix/metadata/metadata.json \
    --pioneerKmsEnabled=false \
    --pioneerDecompressPayload=false \
    --geoCityDatabase=$bucket/$prefix/metadata/GeoLite2-City.mmdb \
    --geoCityFilter=$bucket/$prefix/metadata/cities15000.txt \
    --schemasLocation=$bucket/$prefix/metadata/schemas.tar.gz \
    --inputType=file \
    --input=$bucket/$prefix/input/ciphertext/'part-*' \
    --outputType=file \
    --output=$bucket/$prefix/output/ciphertext/ \
    --errorOutputType=file \
    --errorOutput=$bucket/$prefix/error/ciphertext/ \
"