1889185 - Write submitter as a processor-like service

Status: RESOLVED FIXED

(Socorro :: Antenna, task, P2)

Description

[Daniel Thorn [:relud]]

right now there is an AWS lambda function that sends prod crash reports from AWS prod to AWS stage. The plan for GCP the plan is to have a pubsub subscriber in kubernetes that submits crash reports from GCP prod to GCP stage.

:willkg suggested using antenna's testlib/mini_poster.py to simplify the submitting part, but put in a new stage_submitter/ dir in the root of the antenna repo. the python pubsub sdk has an interface for writing a subscriber that should make the "consume a pubsub subscription" part very simple.

putting this in antenna will mean that we can update antenna apis and the submitter together, and it also makes sense because the submitter is effectively a tool for testing antenna in stage.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I know I've been updating socorro-submitter regularly and I don't remember if I've updated mini_poster.py. It's likely that socorro-submitter does more of the things we need the new stage-submitter to do and thus we should rewrite the mini_poster.py code based on socorro-submitter.

https://github.com/mozilla-services/socorro-submitter/blob/main/src/submitter.py

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this from Relud per our conversation. I'll write up a rough shape for what it should do and get that reviewed by Relud. We'll put it in the socorro repo, structure it like the processor cache manager process-wise, reuse the load/process/save loop in the processor, and reuse the processor configuration where possible.

Relud suggested maybe we do the submitter as another mode of the processor, but I want to avoid dealing with all the task manager stuff. I'll look at that option, too.

Comment 3

[Daniel Thorn [:relud]]

maybe we do the submitter as another mode of the processor

that's me not understanding what does and doesn't exist. we don't need to look into this so long as we have any plan for how to run the submitter

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: [mozilla-services/socorro] bug-1889185: implement stage submitter and fake collector (#6617)

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR [mozilla-services/socorro]: bug-1889185: implement stage submitter and fake collector (#6617) in 00d064a.

This has no effect on the AWS infrastructure. This will only be used in the GCP infrastructure.