[Mozillians][dev] Error when running chief in mozillians-dev

RESOLVED FIXED

Status

Infrastructure & Operations Graveyard
WebOps: Engagement
--
major
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: nemo, Assigned: w0ts0n)

Tracking

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/3044] )

(Reporter)

Description

2 years ago
We are getting this traceback in mozillians-dev chief:

[2016-04-07 09:48:04] [localhost] running: cd /data/python-dev/src/mozillians-dev.allizom.org/mozillians && virtualenv-2.7 --no-site-packages venv
[2016-04-07 09:48:04] [localhost] failed: cd /data/python-dev/src/mozillians-dev.allizom.org/mozillians && virtualenv-2.7 --no-site-packages venv (0.003s)
[localhost] err: /bin/sh: virtualenv-2.7: command not found

Last successful push was on "06-Apr-2016 15:04"
http://10.8.75.42/chief/mozillians.dev/logs/origin.master.1459954982
(Reporter)

Comment 1

2 years ago
Tried the same on stage and we are getting the same traceback.
(Reporter)

Updated

2 years ago
Severity: normal → major

Updated

2 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/2818]
<nemo-yiannis> Hey all! Chief is not working in mozillians-dev mozillians-stage and this issue is blocking a release.
<nemo-yiannis> https://bugzilla.mozilla.org/show_bug.cgi?id=1262778
<nemo-yiannis> Is there anyone around to help us with that?

Comment 3

2 years ago
I will fix this shortly, in a meeting but I know what's wrong.
Assignee: server-ops-webops → nmaul

Comment 4

2 years ago
This should be fixed now. We rebuilt genericadm yesterday for security reasons, and this package was missing from its puppet manifest so it didn't get installed automatically.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Reporter)

Comment 5

2 years ago
It looks like most of the functionality is fixed now (chief returns "success" on code updates). We are getting this error now:

[2016-04-07 18:18:35] [python1.dev.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh mozillians-dev.allizom.org
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/commander/commands.py", line 26, in func
    raise e
ValueError: identity_file should be a valid file
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 6

2 years ago
Also please make sure that the fixes in dev are also applied in stage/prod. I assume that all dev/stage/prod are going to encounter the same issues.
(Reporter)

Updated

2 years ago
Flags: needinfo?(nmaul)
(Reporter)

Updated

2 years ago
Flags: needinfo?(smani)

Comment 7

2 years ago
This should be fixed now as well... for all 3 environments. Let us know if you run into any more trouble. Sorry for the hassle.
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Flags: needinfo?(smani)
Flags: needinfo?(nmaul)
Resolution: --- → FIXED
(Reporter)

Comment 8

2 years ago
We are getting this error only on prod:

Post deployment to New Relic
Traceback (most recent call last):
  File "/usr/bin/commander", line 4, in <module>
    main()
  File "/usr/lib/python2.6/site-packages/commander/main.py", line 111, in main
    commands[cmd](*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/commander/deploy.py", line 165, in wrapper
    res = f(Context(), *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/commander/deploy.py", line 109, in wrapper
    return f(*args, **kwargs)
  File "/data/python/src/mozillians.org/mozillians/scripts/update/update.py", line 212, in deploy
    ping_newrelic()
  File "/usr/lib/python2.6/site-packages/commander/deploy.py", line 165, in wrapper
    res = f(Context(), *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/commander/deploy.py", line 109, in wrapper
    return f(*args, **kwargs)
  File "/data/python/src/mozillians.org/mozillians/scripts/update/update.py", line 92, in ping_newrelic
    except urllib.URLError as exp:
AttributeError: 'module' object has no attribute 'URLError'
Status: RESOLVED → REOPENED
Flags: needinfo?(nmaul)
Resolution: FIXED → ---
(Reporter)

Comment 9

2 years ago
There is no related change in our codebase, did you do any change in the New Relic settings?
Quick update: We just did another release and chief worked flawlessly.

Comment 11

2 years ago
relevant code in mozillians/scripts/update/update.py
(https://github.com/mozilla/mozillians/blob/master/scripts/update/update.py)

        try:
            request = urllib2.Request(NEW_RELIC_URL, data, headers)
            urllib2.urlopen(request)
        except urllib.URLError as exp:
            print 'Error notifying New Relic: {0}'.format(exp)

This is a bit out of my league, but should that be urllib2.URLError? I've never used urllib/urllib2 directly, only the "requests" module, so I'm not really sure what's going on here.

I'm guessing there was some sort of transient error on New Relic's side, and when that doesn't happen it never notices the "except" as being problematic.
Flags: needinfo?(nmaul)

Updated

2 years ago
Assignee: nmaul → server-ops-webops
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/2818]

Updated

2 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/3044]
(Assignee)

Updated

2 years ago
Assignee: server-ops-webops → rwatson
(Assignee)

Comment 12

2 years ago
I think this hasn't reared it's head for a few weeks. Going to close this out. Re-file if needed.
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.