MDN chief reports failing - stage & prod

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: groovecoder, Assigned: cturra)

Tracking

Details

(Reporter)

Description

5 years ago
1. Go to http://developeradm.private.scl3.mozilla.com/chief/developer.stage
2. enter user, pass, and commit hash
3. Push the Big Red button
4. Get:
Updating! revision: dd5228c53bb76a1c8ba89c8bc2f2ad06b0426a58
We have the new code!
Code has been updated locally!

(i.e., missing line about finished)

IRC reports:

10:40:14 - mdnstagepush: something terrible happened. check the logs (dd5228c53bb76a1c8ba89c8bc2f2ad06b0426a58 groovestager)

Checked the log at http://developeradm.private.scl3.mozilla.com/chief/developer.stage/logs/dd5228c53bb76a1c8ba89c8bc2f2ad06b0426a58.1384274292 - can't see what's broken; it says "Finished deploy"

http://developeradm.private.scl3.mozilla.com/chief/developer.stage/history is empty

https://developer.allizom.org/media/revision.txt reports the proper commit hash
(Assignee)

Comment 1

5 years ago
:groovecoder - when you don't see any deploy history on the cheif history page, that means someone/some process kicked the redis server on that node. you can still find a list of all the deploy logs in the following directory however:

  http://developeradm.private.scl3.mozilla.com/chief/developer.stage/logs/


it looks like the deploy is resulting in a "failure"  b/c of this was written out to stderr:

 [localhost] err: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
 [localhost] err: Dload  Upload   Total   Spent    Left  Speed
 [localhost] err: 
 [localhost] err: 0     0    0     0    0   128      0    725 --:--:-- --:--:-- --:--:--   725
 [localhost] err: 117  1398    0  1398    0   128   2903    265 --:--:-- --:--:-- --:--:--  4163
OS: Mac OS X → All
Hardware: x86 → All
FWIW, I think :jakem recently added a curl command to ping New Relic on deploy. Not sure what that looks like because MDN devs don't control the chief update scripts, but it could probably be silenced - or even removed, if it's causing an issue
(Assignee)

Comment 3

5 years ago
(In reply to Les Orchard [:lorchard] from comment #2)
> Not sure what that looks like because MDN devs don't control the
> chief update scripts, but it could probably be silenced - or even removed,
> if it's causing an issue

:lorchard - i added --silent to the new relic curl command. can you test another deploy and see if this sorts things out for you?
Flags: needinfo?(lorchard)
(Assignee)

Comment 5

5 years ago
excellent. i have applied the same to the prod chief deploy script. going to mark this as r/fixed now :)
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Assignee)

Updated

5 years ago
Assignee: server-ops-webops → cturra
You need to log in before you can comment on or make changes to this bug.