Closed
Bug 1628539
Opened 5 years ago
Closed 5 years ago
Create beam job for decrypting pioneer v2 documents
Categories
(Data Platform and Tools :: General, task, P1)
Data Platform and Tools
General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: amiyaguchi, Assigned: amiyaguchi)
References
(Blocks 1 open bug)
Details
Attachments
(3 files, 1 obsolete file)
Pioneer v2 requires an extra decoding step in the gcp-ingestion/ingestion-beam job. The transform will decrypt messages encrypted using JOSE and decompress the payload, before passing it to the rest of the ingestion stack.
Comment 1•5 years ago
|
||
Assignee | ||
Updated•5 years ago
|
Assignee: nobody → amiyaguchi
Comment 2•5 years ago
|
||
Assignee | ||
Comment 3•5 years ago
|
||
Updated•5 years ago
|
Priority: -- → P1
Assignee | ||
Comment 4•5 years ago
|
||
Assignee | ||
Updated•5 years ago
|
Attachment #9139357 -
Attachment is obsolete: true
Assignee | ||
Comment 5•5 years ago
|
||
The decoder can be run using the relevant options. The benchmark script demonstrates the use of the new options and verifies that the decoder runs in a batch context on dataflow. The new options are --pioneerEnabled
, --pioneerMetadataLocation
, --pioneerKmsEnabled
, and --pioneerDecompressPayload
.
./bin/mvn compile exec:java -Dexec.mainClass=com.mozilla.telemetry.Decoder -Dexec.args="\
--runner=Dataflow \
--profilingAgentConfiguration='{\"APICurated\": true}'
--project=$project \
--autoscalingAlgorithm=NONE \
--workerMachineType=n1-standard-1 \
--gcpTempLocation=$bucket/tmp \
--numWorkers=2 \
--pioneerEnabled=true \
--pioneerMetadataLocation=$bucket/$prefix/metadata/metadata.json \
--pioneerKmsEnabled=false \
--pioneerDecompressPayload=false \
--geoCityDatabase=$bucket/$prefix/metadata/GeoLite2-City.mmdb \
--geoCityFilter=$bucket/$prefix/metadata/cities15000.txt \
--schemasLocation=$bucket/$prefix/metadata/schemas.tar.gz \
--inputType=file \
--input=$bucket/$prefix/input/ciphertext/'part-*' \
--outputType=file \
--output=$bucket/$prefix/output/ciphertext/ \
--errorOutputType=file \
--errorOutput=$bucket/$prefix/error/ciphertext/ \
"
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Updated•3 years ago
|
Component: Pipeline Ingestion → General
You need to log in
before you can comment on or make changes to this bug.
Description
•