Closed Bug 829025 Opened 8 years ago Closed 5 years ago

tracker for all occurrences of hg socket hangs on vcs-sync system

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: hwine, Unassigned)

References

Details

(Whiteboard: [re-b2g])

We've seen a number of these (more to be added later). Want to start tracking now to see if a pattern is there.

Each one causes a page and requires manual intervention to restore operations of this production service.
Summary
========

Documented recovery procedure worked: http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/file/be390f8fcad2/docs/trouble_shooting.rst#l14

last good update: 2013-01-10T14:44:36+0000
next good update: 2013-01-10T15:33:47+0000
total delay: 49:11

Details
=======

Email alert:
------------

-------- Original Message --------
Subject: 	[vcs2vcs] process delays
Date: 	Thu, 10 Jan 2013 15:07:02 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	hwine@mozilla.com


check these 3 items:
Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-pt-BR-gaia pull
Unknown delay: wait S+ /usr/bin/time -f 2013-01-10T14:54:42+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-pt-BR-gaia pull
socket hang on 3032

Remote status scan verifies lack of update to log files
-------------------------------------------------------

All log file dates should be within 10 minutes of the display system date
gd3: Thu Jan 10 15:23:24 UTC 2013
gd3: -rw-r----- 1 vcs2vcs vcs2vcs 10234634 Jan 10 15:18 /opt/vcs2vcs/logs/job07/details.log
gd3: -rw-r----- 1 vcs2vcs vcs2vcs 11318095 Jan 10 15:18 /opt/vcs2vcs/logs/job07/update.log
gd3: -rw-r----- 1 vcs2vcs vcs2vcs  2362339 Jan 10 15:17 /opt/vcs2vcs/logs/job06/details.log
gd3: -rw-r----- 1 vcs2vcs vcs2vcs  2237911 Jan 10 15:17 /opt/vcs2vcs/logs/job06/update.log
gd3: -rw-r----- 1 vcs2vcs vcs2vcs   939429 Jan 10 14:54 /opt/vcs2vcs/logs/job08/details.log
gd3: -rw-r----- 1 vcs2vcs vcs2vcs   746043 Jan 10 14:54 /opt/vcs2vcs/logs/job08/update.log

Status on github-sync3.dmz.scl3.mozilla.com:
--------------------------------------------

[vcs2vcs@github-sync3.dmz.scl3 .git]$ ps -flwwwp 3032
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 S vcs2vcs   3032  3031  0  80   0 - 50200 sk_wai 14:54 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-pt-BR-gaia pull
[vcs2vcs@github-sync3.dmz.scl3 .git]$ ls -l /proc/!$/fd
ls -l /proc/3032/fd
total 0
lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 10 15:07 0 -> /dev/pts/9
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 10 15:07 1 -> /opt/vcs2vcs/logs/job08/details.log
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 10 14:55 2 -> /opt/vcs2vcs/logs/job08/details.log
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 10 15:07 3 -> /opt/vcs2vcs/logs/job08/update.log
lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 10 15:07 4 -> socket:[113644572]
[vcs2vcs@github-sync3.dmz.scl3 .git]$ 

socket:[113644572] no longer exists
Summary
========

Documented recovery procedure worked: http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/file/be390f8fcad2/docs/trouble_shooting.rst#l14

last good push: 2013-01-11T05:11:09+0000
next good push: 2013-01-11T06:22:22+0000
total delay: 1:11:13

Details
=======

Email alert:
------------


-------- Original Message --------
Subject: 	[vcs2vcs] process delays
Date: 	Fri, 11 Jan 2013 05:47:01 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


check these 2 items:
process not running for stale file '/opt/vcs2vcs/b2g/details.log'
process not running for stale file '/opt/vcs2vcs/b2g/update.log'

Note that error message incorrect, but problem real

Status on github-sync3.dmz.scl3.mozilla.com:
--------------------------------------------

[vcs2vcs@github-sync3.dmz.scl3 ~]$ cd b2g
[vcs2vcs@github-sync3.dmz.scl3 b2g]$ ll *log
-rw-r----- 1 vcs2vcs vcs2vcs 6401024 Jan 11 05:21 details.log
-rw-r----- 1 vcs2vcs vcs2vcs 8797607 Jan 11 05:21 update.log
[vcs2vcs@github-sync3.dmz.scl3 b2g]$ date
Fri Jan 11 05:54:41 UTC 2013

[vcs2vcs@github-sync3.dmz.scl3 b2g]$ pstree -p $USER
screen(21936)─┬─bash(5441)
              └─bash(25627)───bash(8142)───keep_updated.ne(28337)───timeit(30193)───time(30196)───hg(30197)

[vcs2vcs@github-sync3.dmz.scl3 b2g]$ ps -flwwwwp 30197
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 S vcs2vcs  30197 30196  0  80   0 - 51608 sk_wai 05:21 pts/2    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r fb011a7d8798 b2g18
[vcs2vcs@github-sync3.dmz.scl3 b2g]$ ls -l /proc/!$/fd
ls -l /proc/30197/fd
total 0
lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 11 05:47 0 -> /dev/pts/2
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 05:47 1 -> /opt/vcs2vcs/b2g/details.log
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 05:30 2 -> /opt/vcs2vcs/b2g/details.log
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 05:47 3 -> /opt/vcs2vcs/b2g/update.log
lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 11 05:47 4 -> socket:[115316066]

socket:[115316066] no longer exists
(In reply to Hal Wine [:hwine] from comment #2)
 Summary
 ========
 
 Documented recovery procedure worked:
 http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/file/
 be390f8fcad2/docs/trouble_shooting.rst#l14
 
 last good push: 2013-01-11T06:43:16+0000
 next good push: 2013-01-11T08:36:32+0000
 total delay: 1:54:16
 
 Details
 =======
 
 Email alert:
---------
Subject: Cron <vcs2vcs@github-sync3> $HOME/bin/check_process_delay
Date: Fri, 11 Jan 2013 07:07:01 +0000 (UTC)

Unknown delay: wait
S+
/bin/bash
/opt/vcs2vcs/bin/timeit
hg
--cwd
/opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko
pull
0 S vcs2vcs  11639  8844  0  80   0 - 26523 wait   06:43 pts/9    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
Unknown delay: wait
S+
/usr/bin/time
-f
2013-01-11T06:43:16+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C;
-a
-o
/opt/vcs2vcs/logs/job08/update.log
hg
--cwd
/opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko
pull
0 S vcs2vcs  11642 11639  0  80   0 -  1015 wait   06:43 pts/9    00:00:00 /usr/bin/time -f 2013-01-11T06:43:16+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
socket hang on pid 11643
0 S vcs2vcs  11643 11642  0  80   0 - 50341 sk_wai 06:43 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
check these 6 items:
Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
0 S vcs2vcs  11639  8844  0  80   0 - 26523 wait   06:43 pts/9    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
Unknown delay: wait S+ /usr/bin/time -f 2013-01-11T06:43:16+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
0 S vcs2vcs  11642 11639  0  80   0 -  1015 wait   06:43 pts/9    00:00:00 /usr/bin/time -f 2013-01-11T06:43:16+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
socket hang on pid 11643
0 S vcs2vcs  11643 11642  0  80   0 - 50341 sk_wai 06:43 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
 ------------


 
 Status on github-sync3.dmz.scl3.mozilla.com:
 --------------------------------------------

 
[vcs2vcs@github-sync3.dmz.scl3 etc]$ pstree -p $USER
screen(21936)─┬─bash(5441)───pstree(30641)
              └─bash(5931)───do_update(8844)───timeit(11639)───time(11642)───hg(11643)
              
[vcs2vcs@github-sync3.dmz.scl3 etc]$ ls -lR /proc/11643/fd
/proc/11643/fd:
total 0
lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 11 07:07 0 -> /dev/pts/9
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 07:07 1 -> /opt/vcs2vcs/logs/job08/details.log
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 06:50 2 -> /opt/vcs2vcs/logs/job08/details.log
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 07:07 3 -> /opt/vcs2vcs/logs/job08/update.log
lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 11 07:07 4 -> socket:[115484367]
> 
> socket:[115484367] no longer exists
Summary
=======

Documented recovery procedure worked: http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/file/be390f8fcad2/docs/trouble_shooting.rst#l14

last good push: 2013-01-13T01:24+0000
next good push: 2013-01-13T08:13+0000
total delay: 6:51:00

Details
=======

Email alert:
------------
Subject: Cron <vcs2vcs@github-sync3> $HOME/bin/check_process_delay
Date: Sun, 13 Jan 2013 01:47:01 +0000 (UTC)
From: root@github-sync3.dmz.scl3.mozilla.com (Cron Daemon)
To: vcs2vcs@github-sync3.dmz.scl3.mozilla.com

Unknown delay: wait
S+
/bin/bash
/opt/vcs2vcs/bin/timeit
hg
--cwd
/opt/vcs2vcs/b2g/mc-cvs-ma-update-test3
pull
-r
2bb8638b277e
beta
0 S vcs2vcs  14045 28337  0  80   0 - 26523 wait   01:35 pts/2    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta
Unknown delay: wait
S+
/usr/bin/time
-f
2013-01-13T01:35:01+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C;
-a
-o
/opt/vcs2vcs/b2g/update.log
hg
--cwd
/opt/vcs2vcs/b2g/mc-cvs-ma-update-test3
pull
-r
2bb8638b277e
beta
0 S vcs2vcs  14048 14045  0  80   0 -  1015 wait   01:35 pts/2    00:00:00 /usr/bin/time -f 2013-01-13T01:35:01+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/b2g/update.log hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta
socket hang on pid 14049
0 S vcs2vcs  14049 14048  0  80   0 - 51609 sk_wai 01:35 pts/2    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta
check these 6 items:
Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta
0 S vcs2vcs  14045 28337  0  80   0 - 26523 wait   01:35 pts/2    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta
Unknown delay: wait S+ /usr/bin/time -f 2013-01-13T01:35:01+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/b2g/update.log hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta
0 S vcs2vcs  14048 14045  0  80   0 -  1015 wait   01:35 pts/2    00:00:00 /usr/bin/time -f 2013-01-13T01:35:01+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/b2g/update.log hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta
socket hang on pid 14049
0 S vcs2vcs  14049 14048  0  80   0 - 51609 sk_wai 01:35 pts/2    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta

Status on github-sync3.dmz.scl3.mozilla.com:
--------------------------------------------
[vcs2vcs@github-sync3.dmz.scl3 ~]$ cd b2g/
[vcs2vcs@github-sync3.dmz.scl3 b2g]$ ll *log
-rw-r----- 1 vcs2vcs vcs2vcs 6618896 Jan 13 01:35 details.log
-rw-r----- 1 vcs2vcs vcs2vcs 9111767 Jan 13 01:25 update.log
[vcs2vcs@github-sync3.dmz.scl3 b2g]$ date
Sun Jan 13 08:08:28 UTC 2013

[vcs2vcs@github-sync3.dmz.scl3 b2g]$ pstree -p $USER
screen(21936)─┬─bash(5441)
              ├─bash(25627)───bash(8142)───keep_updated.ne(28337)─\
                   ──timeit(14045)───time(14048)───hg(14049)
 
[vcs2vcs@github-sync3.dmz.scl3 b2g]$ ps -flwwwwp 14049
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 S vcs2vcs  14049 14048  0  80   0 - 51609 sk_wai 01:35 pts/2    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta

$ ls -l /proc/!$/fd
ls -l /proc/14049/fd
total 0
lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 13 01:47 0 -> /dev/pts/2
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 13 01:47 1 -> /opt/vcs2vcs/b2g/details.log
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 13 01:47 2 -> /opt/vcs2vcs/b2g/details.log
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 13 01:47 3 -> /opt/vcs2vcs/b2g/update.log
lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 13 01:47 4 -> socket:[121537471]
Another - first email:


Subject: 	[vcs2vcs] process delays
Date: 	Thu, 17 Jan 2013 10:07:01 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


check these 6 items:
Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
0 S vcs2vcs   9629  6888  0  80   0 - 26523 wait   08:52 pts/9    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
Unknown delay: wait S+ /usr/bin/time -f 2013-01-17T08:52:42+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
0 S vcs2vcs   9632  9629  0  80   0 -  1015 wait   08:52 pts/9    00:00:00 /usr/bin/time -f 2013-01-17T08:52:42+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
socket hang on pid 9633
0 S vcs2vcs   9633  9632  0  80   0 - 50341 sk_wai 08:52 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
another: first email:



-------- Original Message --------
Subject: 	[vcs2vcs] process delays
Date: 	Sat, 19 Jan 2013 07:27:01 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


check these 6 items:
Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
0 S vcs2vcs   8629  8622  0  80   0 - 26523 wait   07:11 pts/9    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
Unknown delay: wait S+ /usr/bin/time -f 2013-01-19T07:11:23+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
0 S vcs2vcs   8632  8629  0  80   0 -  1015 wait   07:11 pts/9    00:00:00 /usr/bin/time -f 2013-01-19T07:11:23+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
socket hang on pid 8633
0 S vcs2vcs   8633  8632  0  80   0 - 49135 sk_wai 07:11 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
Another one:



-------- Original Message --------
Subject: 	[vcs2vcs] process delays
Date: 	Sat, 19 Jan 2013 17:47:01 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


check these 6 items:
Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
0 S vcs2vcs  13992 13985  0  80   0 - 26523 wait   17:34 pts/9    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
Unknown delay: wait S+ /usr/bin/time -f 2013-01-19T17:34:08+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
0 S vcs2vcs  13995 13992  0  80   0 -  1015 wait   17:34 pts/9    00:00:00 /usr/bin/time -f 2013-01-19T17:34:08+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
socket hang on pid 13996
0 S vcs2vcs  13996 13995  0  80   0 - 50200 sk_wai 17:34 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
check these 6 items:
Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
0 S vcs2vcs   1896  1428  0  80   0 - 26523 wait   04:07 pts/9    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
Unknown delay: wait S+ /usr/bin/time -f 2013-01-21T04:07:15+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
0 S vcs2vcs   1899  1896  0  80   0 -  1015 wait   04:07 pts/9    00:00:00 /usr/bin/time -f 2013-01-21T04:07:15+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
socket hang on pid 1900
0 S vcs2vcs   1900  1899  0  80   0 - 49134 sk_wai 04:07 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
Depends on: 833590
another one (auto workaround worked):

-------- Original Message --------
Subject: 	[vcs2vcs] process delays
Date: 	Wed, 23 Jan 2013 03:07:02 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


check these 6 items:
Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
0 S vcs2vcs   7651  4900  0  80   0 - 26523 wait   02:52 pts/9    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
Unknown delay: wait S+ /usr/bin/time -f 2013-01-23T02:52:13+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
0 S vcs2vcs   7654  7651  0  80   0 -  1015 wait   02:52 pts/9    00:00:00 /usr/bin/time -f 2013-01-23T02:52:13+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
socket hang on pid 7655
0 S vcs2vcs   7655  7654  0  80   0 - 50342 sk_wai 02:52 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
Date: Wed, 23 Jan 2013 08:07:01 +0000
To: release+vcs2vcs@mozilla.com
Subject: [vcs2vcs] process delays
From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com

check these 6 items:
Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r e3803e97aeff b2g18
0 S vcs2vcs  26021 31723  0  80   0 - 26523 wait   07:44 pts/2    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r e3803e97aeff b2g18
Unknown delay: wait S+ /usr/bin/time -f 2013-01-23T07:44:11+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/b2g/update.log hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r e3803e97aeff b2g18
0 S vcs2vcs  26024 26021  0  80   0 -  1015 wait   07:44 pts/2    00:00:00 /usr/bin/time -f 2013-01-23T07:44:11+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/b2g/update.log hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r e3803e97aeff b2g18
socket hang on pid 26025
0 S vcs2vcs  26025 26024  0  80   0 - 49136 sk_wai 07:44 pts/2    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r e3803e97aeff b2g18
Date: Wed, 23 Jan 2013 09:07:35 +0000
To: release+vcs2vcs@mozilla.com
Subject: [vcs2vcs] process delays
From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com

check these 6 items:
Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
0 S vcs2vcs    443 32673  0  80   0 - 26523 wait   08:50 pts/9    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
Unknown delay: wait S+ /usr/bin/time -f 2013-01-23T08:50:22+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
0 S vcs2vcs    446   443  0  80   0 -  1015 wait   08:50 pts/9    00:00:00 /usr/bin/time -f 2013-01-23T08:50:22+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
socket hang on pid 447
0 S vcs2vcs    447   446  0  80   0 - 50461 sk_wai 08:50 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
Another one, cleared automatically at 08:47



-------- Original Message --------
Subject: 	[vcs2vcs] process delays
Date: 	Sat, 26 Jan 2013 16:27:02 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


check these 6 items:
Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
0 S vcs2vcs  16013 13048  0  80   0 - 26523 wait   16:00 pts/9    00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
Unknown delay: wait S+ /usr/bin/time -f 2013-01-26T16:00:33+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
0 S vcs2vcs  16016 16013  0  80   0 -  1015 wait   16:00 pts/9    00:00:00 /usr/bin/time -f 2013-01-26T16:00:33+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
socket hang on pid 16017
0 S vcs2vcs  16017 16016  0  80   0 - 49136 sk_wai 16:00 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
another one:
(hwine-moz1)[vcs2vcs@github-sync3.dmz.scl3 tmp]$ lsof ~/b2g/[du]*log
(hwine-moz1)[vcs2vcs@github-sync3.dmz.scl3 tmp]$ ps -flwwwwwp 13010
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 S vcs2vcs  13010 14700  0  80   0 - 49138 sk_wai 00:02 pts/2    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg id http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0
(hwine-moz1)[vcs2vcs@github-sync3.dmz.scl3 tmp]$ ll /proc/13010/fd
total 0
lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 29 00:42 0 -> /dev/pts/2
l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 29 00:42 1 -> pipe:[175853002]
lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 29 00:39 2 -> /dev/pts/2
lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 29 00:42 3 -> socket:[175853010]
(hwine-moz1)[vcs2vcs@github-sync3.dmz.scl3 tmp]$
Another - autofix worked:



-------- Original Message --------
Subject: 	[vcs2vcs] AUTOFIX process delays
Date: 	Tue, 29 Jan 2013 02:47:03 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com



check these 2 items:
Attempting unstick of hung process 28330
0 S vcs2vcs  28330 28329  0  80   0 - 50201 sk_wai 02:26 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-pt-BR-gaia pull
autofix successful


-------- Original Message --------
Subject: 	[vcs2vcs] AUTOFIX process delays
Date: 	Thu, 31 Jan 2013 13:07:02 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com:
Attempting unstick of hung process 9324
0 S vcs2vcs   9324  9323  0  80   0 - 51685 sk_wai 12:47 pts/2    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 4d4116685156 beta

(if fix is successful, no further email will be sent)
(just add this email to https://bugzil.la/829025)
autofix successful


-------- Original Message --------
Subject: 	[vcs2vcs] AUTOFIX process delays
Date: 	Thu, 31 Jan 2013 21:47:02 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com:
Attempting unstick of hung process 9923
0 S vcs2vcs   9923  9922  0  80   0 - 51695 sk_wai 21:35 pts/2    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 0565fdda9c81 b2g18_v1_0_0

(if fix is successful, no further email will be sent)
(just add this email to https://bugzil.la/829025)
autofix successful


-------- Original Message --------
Subject: 	[vcs2vcs] AUTOFIX process delays
Date: 	Thu, 31 Jan 2013 22:47:02 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com:
Attempting unstick of hung process 17739
0 S vcs2vcs  17739 17738  0  80   0 - 50344 sk_wai 22:34 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull

(if fix is successful, no further email will be sent)
(just add this email to https://bugzil.la/829025)
autofix successful


-------- Original Message --------
Subject: 	[vcs2vcs] AUTOFIX process delays
Date: 	Sat, 02 Feb 2013 18:47:03 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com:
Attempting unstick of hung process 10137
0 S vcs2vcs  10137 10136  0  80   0 - 28342 sk_wai 18:39 pts/5    00:00:00 git --git-dir /opt/vcs2vcs/repos/b2g-android-device-crespo/.git fetch --force

(if fix is successful, no further email will be sent)
(just add this email to https://bugzil.la/829025)
autofix successful



-------- Original Message --------
Subject: 	[vcs2vcs] AUTOFIX process delays
Date: 	Mon, 11 Feb 2013 22:27:02 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com:
Attempting unstick of hung process 22645
0 S vcs2vcs  22645 22644  0  80   0 - 50409 sk_wai 22:10 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-release-sk-gecko pull

(if fix is successful, no further email will be sent)
(just add this email to https://bugzil.la/829025)
autofix successful:


-------- Original Message --------
Subject: 	[vcs2vcs] AUTOFIX process delays
Date: 	Tue, 12 Feb 2013 20:47:02 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com:
Attempting unstick of hung process 14069
0 S vcs2vcs  14069 14068  0  80   0 - 49172 sk_wai 20:24 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-aurora-nl-gecko pull

(if fix is successful, no further email will be sent)
(just add this email to https://bugzil.la/829025)
successful autofix



-------- Original Message --------
Subject: 	[vcs2vcs] AUTOFIX process delays
Date: 	Fri, 15 Feb 2013 07:27:02 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com:
Attempting unstick of hung process 26668
0 S vcs2vcs  26668 26667  0  80   0 - 50666 sk_wai 06:59 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-de-gecko pull

(if fix is successful, no further email will be sent)
(just add this email to https://bugzil.la/829025)
successful autofix



-------- Original Message --------
Subject: 	[vcs2vcs] AUTOFIX process delays
Date: 	Fri, 15 Feb 2013 16:27:03 +0000
From: 	vcs2vcs@github-sync3.dmz.scl3.mozilla.com
To: 	release+vcs2vcs@mozilla.com


report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com:
Attempting unstick of hung process 27216
0 S vcs2vcs  27216 27215  0  80   0 - 50686 sk_wai 16:04 pts/9    00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pl-gecko pull

(if fix is successful, no further email will be sent)
(just add this email to https://bugzil.la/829025)
Product: mozilla.org → Release Engineering
No updates for over 11 months; I think we're assuming autofixes work and filing new bugs if they aren't.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
agreed - hopefully when the new system starts pulling from hg, this won't be encountered.
report from /home/vcs2vcs/bin/check_process_delay on ip-172-31-26-60:
Attempting unstick of hung process 3584
0 S vcs2vcs   3584  3576  0  80   0 - 39467 poll_s 12:10 pts/11   00:00:00 git-remote-http origin http://sprdsource.spreadtrum.com:8085/b2g/android/platform/external/jack
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
report from /home/vcs2vcs/bin/check_process_delay on ip-172-31-26-60:
Attempting unstick of hung process 3198
0 S vcs2vcs   3198  3196  0  80   0 - 39467 poll_s 14:59 pts/11   00:00:00 git-remote-http origin http://sprdsource.spreadtrum.com:8085/b2g/android/device/sprd

(if fix fails, another email will come within 5 minutes)
(just add this email to https://bugzil.la/829025)
Old vcs-sync is dead.
Status: REOPENED → RESOLVED
Closed: 7 years ago5 years ago
Resolution: --- → WONTFIX
Component: Tools → General
You need to log in before you can comment on or make changes to this bug.