910483 - push Socorro 57 release

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task, P3)

Description

[Lonnen :lonnen]

Build: https://ci.mozilla.org/job/socorro-release/481/artifact/socorro.tar.gz

Comment 1

[Brandon Burton [:solarce]]

This push is brought to you by the letters SQUIRREL and the number 481

Comment 2

[Brandon Burton [:solarce]]

Push completed. Will add details of push shortly.

Comment 3

[Brandon Burton [:solarce]]

Summary of Push related issues

1. Changes in https://bugzilla.mozilla.org/show_bug.cgi?id=907940 did not happen prior to code push, which caused the collector and mware tiers to begin erroring when the new code was pushed and Apache restarted. Fixed by manually fixing config files to return to service, then committing to Puppet to ensure changes weren't lost. 
 * (There is about a 10-12 minutes lag between committing to Puppet and it being able to be pushed out)

2. Multiple Collectors failed to start, had to SSH in and'rm -rf /var/lock/socorro-processor.lock' by hand, then start the service
 - Affected servers:

  - sp-processor01.phx1
  - sp-processor03.phx1
  - sp-processor04.phx1
  - sp-processor05.phx1
  - sp-processor06.phx1
  - sp-processor07.phx1
  - sp-processor08.phx1
  - sp-processor10.phx1

Comment 4

[Matt Brandt [:mbrandt]]

Late to update but the intent was there. Thank you all who participated, the release was one of the more difficult/bumpier ones that we've had in awhile but we were able to stand the env back up. Note's that follow are from last night post-release:
- QA Automation and manual tests pass
- /status/ page has normalized
- Spot checking new features verifies they are live
- B2G is in the product dropdown
- lars verified the processors and collectors are receiving and processing crashes

QA verified