937681 - MDN chief reports failing - stage & prod

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Luke Crouch [:groovecoder]]

1. Go to http://developeradm.private.scl3.mozilla.com/chief/developer.stage
2. enter user, pass, and commit hash
3. Push the Big Red button
4. Get:
Updating! revision: dd5228c53bb76a1c8ba89c8bc2f2ad06b0426a58
We have the new code!
Code has been updated locally!

(i.e., missing line about finished)

IRC reports:

10:40:14 - mdnstagepush: something terrible happened. check the logs (dd5228c53bb76a1c8ba89c8bc2f2ad06b0426a58 groovestager)

Checked the log at http://developeradm.private.scl3.mozilla.com/chief/developer.stage/logs/dd5228c53bb76a1c8ba89c8bc2f2ad06b0426a58.1384274292 - can't see what's broken; it says "Finished deploy"

http://developeradm.private.scl3.mozilla.com/chief/developer.stage/history is empty

https://developer.allizom.org/media/revision.txt reports the proper commit hash

Comment 1

[Chris Turra [:cturra]]

:groovecoder - when you don't see any deploy history on the cheif history page, that means someone/some process kicked the redis server on that node. you can still find a list of all the deploy logs in the following directory however:

  http://developeradm.private.scl3.mozilla.com/chief/developer.stage/logs/


it looks like the deploy is resulting in a "failure"  b/c of this was written out to stderr:

 [localhost] err: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
 [localhost] err: Dload  Upload   Total   Spent    Left  Speed
 [localhost] err: 
 [localhost] err: 0     0    0     0    0   128      0    725 --:--:-- --:--:-- --:--:--   725
 [localhost] err: 117  1398    0  1398    0   128   2903    265 --:--:-- --:--:-- --:--:--  4163

Comment 2

[Les Orchard [:lorchard]]

FWIW, I think :jakem recently added a curl command to ping New Relic on deploy. Not sure what that looks like because MDN devs don't control the chief update scripts, but it could probably be silenced - or even removed, if it's causing an issue

Comment 3

[Chris Turra [:cturra]]

(In reply to Les Orchard [:lorchard] from comment #2)
> Not sure what that looks like because MDN devs don't control the
> chief update scripts, but it could probably be silenced - or even removed,
> if it's causing an issue

:lorchard - i added --silent to the new relic curl command. can you test another deploy and see if this sorts things out for you?

Comment 4

[Les Orchard [:lorchard]]

Seems to have worked okay:

http://developeradm.private.scl3.mozilla.com/chief/developer.stage/logs/4fa0317cdfad51b28929decb438930c18c9f90cb.1384287635

Comment 5

[Chris Turra [:cturra]]

excellent. i have applied the same to the prod chief deploy script. going to mark this as r/fixed now :)