987737 - Check mozilla bitbucket account is functioning correctly and that access key is still valid

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

At the moment our vcs sync process cannot push to bitbucket (bug 987613).

We need to check that our key is still valid for pushing content, and that our account is still valid/active etc.

If we discover an issue and resolve it, we will need to reenable commented out bitbucket jobs on our vcs mirroring infrastructure:

    * job03 on github-sync1.dmz.scl3
    * job01 and job04 on github-sync1-dev.dmz.scl3.mozilla.com

Thanks,
Pete

Comment 1

[Pete Moore [:pmoore][:pete]]

(and afterwards clean up the stale log files created in https://bugzilla.mozilla.org/show_bug.cgi?id=987613#c8)

Comment 2

[Hal Wine [:hwine] use NI!]

Certain hg operations are hanging when run from our vcs-sync production hosts. The problem operations include at least 'hg incomming' and 'hg outgoing' over ssh connections to bitbucket.org

I've replicated the issue with a simple test repo in my own bitbucket account from the production machine. And the problem may have just cleared just before 1000PT

Checking with netops about any known issue

Comment 3

[Hal Wine [:hwine] use NI!]

And the problem is back:

Tue Mar 25 18:38:40 UTC 2014
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
0   495 30851 10405  20   0 109904  3416 wait   Ss   pts/3      0:00 /bin/bash
0   495 12592 30851  20   0 177672 10064 pipe_w S+   pts/3      0:00  \_ /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg out
0   495 12593 12592  20   0  60036  3108 poll_s S+   pts/3      0:00      \_ ssh [e-mail] hg -R halwine/test_1 serve --stdio

Tue Mar 25 18:40:59 UTC 2014
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
0   495 30851 10405  20   0 109904  3416 wait   Ss   pts/3      0:00 /bin/bash
0   495 12592 30851  20   0 177672 10064 pipe_w S+   pts/3      0:00  \_ /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg out
0   495 12593 12592  20   0  60036  3108 poll_s S+   pts/3      0:00      \_ ssh [e-mail] hg -R halwine/test_1 serve --stdio

So, more glitches. Leaving disconnected for now. Need to revisit the utility.

Comment 4

[Hal Wine [:hwine] use NI!]

Looks like this sort of case could also be handled by "AUTOFIX" code:

ps flwwwt pts/3
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
0   495 30851 10405  20   0 109904  3416 wait   Ss   pts/3      0:00 /bin/bash
0   495 12897 30851  20   0 177672 10064 pipe_w S+   pts/3      0:00  \_ /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg out
0   495 12898 12897  20   0  60036  3108 poll_s S+   pts/3      0:00      \_ ssh [e-mail] hg -R halwine/test_1 serve --stdio
[[e-mail]3 ~]$ kill -hup 12898
[[e-mail]3 ~]$ !ps
ps flwwwt pts/3
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
0   495 30851 10405  20   0 109904  3416 n_tty_ Ss+  pts/3      0:00 /bin/bash
[[e-mail]3 ~]$

Comment 5

[Pete Moore [:pmoore][:pete]]

Hal,

Shall I close this now?

Thanks,
Pete