1405679 - adjust stage submitter to submit to multiple places

Status: VERIFIED FIXED

(Socorro :: General, task, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We have a "stage submitter" process which runs on a long-running node which pulls crash ids from the socorro.stagesubmitter RabbitMQ queue in -prod, downloads all the raw crash data from the -prod S3 bucket, assembles an HTTP POST payload, and then submits that payload to Antenna in the -stage infrastructure.

In this way, we populate -stage with 1/10 of what's in -prod (nb. the throttling is done by Pigeon in -prod).

We're in the process of redoing the infrastructure and that involves creating a parallel -stage universe. It would be helpful to have the stage submitter submit crashes to both -stage and -new-stage.

This bug covers looking at the stage submitter code and figuring out what is involved in getting it to submit to two places.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

If this turns out to be at all hard, then we should scrap the current stage submitter and write a new one based on stuff that exists in Antenna already.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm making this a P2 because it's not urgent (need to do it today) but we should do this soon since it will block progress on the new infrastructure.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Adding a blocker on new infrastructure bug.

Making it a P1 because we're ready for this now. Grabbing it to do soon.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4033 - remove class we don't use

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/2d9bc00eb9584ab197956327f5fc6663c5db8feb
bug 1405679 - Remove unused SubmitterFileSystemWalkerSource

https://github.com/mozilla-services/socorro/commit/68e22a64fc67ecc5229b6d6975e9b960e1767e6b
Merge pull request #4033 from willkg/1405679-submitter

bug 1405679 - Remove unused SubmitterFileSystemWalkerSource

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4034 - reify submitter

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

I cleaned up the code a bit, then converted it into a Docker service so we can run/test it locally.

Need to wait for all that to land. Then we do this:

1. Update -prod configuration adding a "urls" variable with the value of "url". (Needs review.)

2. Change the code to use "urls" instead of "url". "urls" will be a comma separated set of urls to post the crash to. (Needs review.)

3. Once that has been deployed to -prod, we need to update the submitter node.

Comment 8

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/46bce0611df6b475306841303379004f69deeace
bug 1405679 - add submitter as a docker service

This adds the submitter as a docker service with configuration and a run script.
This is different than other services because the configuration conflicts with
configuration for other services.

For now, I'm just slapping this together for the purposes of being able to test
changes to the submitter in the local development environment. If we decide to
use this for other purposes, we can redo it then.

https://github.com/mozilla-services/socorro/commit/8642feae54f5dee61b0d75f83f4a107c90d0cd30
Merge pull request #4034 from willkg/1405679-reify-submitter

bug 1405679 - add submitter as a docker service

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4049 - support multiple urls

Comment 10

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/1c473db79dd47ac95878cd81cfb68ac2bf4f7b7c
fixes bug 1405679 - add support for multiple urls to submitter

https://github.com/mozilla-services/socorro/commit/55a22b1664dd582b76c1da878e7f7e125e7e2a68
Merge pull request #4049 from willkg/1405679-submitter-urls

bug 1405679 - submitter urls

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

Next is to do the following (different order than comment #7, but that's fine):

1. Wait for the -prod deploy.

2. Update -prod configuration adding a "urls" variable with the value of "url".

   $ consulate kv backup > ~/consulate-backup-DATESTAMP.json
   $ consulate kv set socorro/submitter/destination.urls https://crash-reports.allizom.org/submit

3. Update the stage-submitter node (it's a long-running node that gets updated manually).

4. Remove the "url" configuration value.

   $ consulate kv rm socorro/submitter/destination.url

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

After the -prod deploy just now, I backed up the configuration in -prod, then added the destionation.urls variable.

The I updated the stage submitter node from 1:252.1478807329-1 to 1:293.1508943809-1 which is really the 294 tag, but we haven't done a -stage deploy, yet, so it has the wrong tag number in the rpm name.

Then I verified the stage submitter continued to work.

Turns out it was using bits of the collector (which no longer exists), so I also updated the destination.crashstorage_class:

$ consulate kv set socorro/submitter/destination.crashstorage_class socorro.submitter.breakpad_submitter_utilities.BreakpadPOSTDestination

After fixing that, the logs looked fine.

I watched the Antenna -stage dashboard to make sure crashes were coming in. You can see the dip when it was configured wrong, then it picks back up again. So that looks fine.

Then I removed the "url" configuration value.

Everything looks super duper! Marking this as VERIFIED.