Closed Bug 829025 Opened 12 years ago Closed 8 years ago

tracker for all occurrences of hg socket hangs on vcs-sync system

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: u429623, Unassigned)

References

Details

(Whiteboard: [re-b2g])

We've seen a number of these (more to be added later). Want to start tracking now to see if a pattern is there. Each one causes a page and requires manual intervention to restore operations of this production service.
Summary ======== Documented recovery procedure worked: http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/file/be390f8fcad2/docs/trouble_shooting.rst#l14 last good update: 2013-01-10T14:44:36+0000 next good update: 2013-01-10T15:33:47+0000 total delay: 49:11 Details ======= Email alert: ------------ -------- Original Message -------- Subject: [vcs2vcs] process delays Date: Thu, 10 Jan 2013 15:07:02 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: hwine@mozilla.com check these 3 items: Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-pt-BR-gaia pull Unknown delay: wait S+ /usr/bin/time -f 2013-01-10T14:54:42+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-pt-BR-gaia pull socket hang on 3032 Remote status scan verifies lack of update to log files ------------------------------------------------------- All log file dates should be within 10 minutes of the display system date gd3: Thu Jan 10 15:23:24 UTC 2013 gd3: -rw-r----- 1 vcs2vcs vcs2vcs 10234634 Jan 10 15:18 /opt/vcs2vcs/logs/job07/details.log gd3: -rw-r----- 1 vcs2vcs vcs2vcs 11318095 Jan 10 15:18 /opt/vcs2vcs/logs/job07/update.log gd3: -rw-r----- 1 vcs2vcs vcs2vcs 2362339 Jan 10 15:17 /opt/vcs2vcs/logs/job06/details.log gd3: -rw-r----- 1 vcs2vcs vcs2vcs 2237911 Jan 10 15:17 /opt/vcs2vcs/logs/job06/update.log gd3: -rw-r----- 1 vcs2vcs vcs2vcs 939429 Jan 10 14:54 /opt/vcs2vcs/logs/job08/details.log gd3: -rw-r----- 1 vcs2vcs vcs2vcs 746043 Jan 10 14:54 /opt/vcs2vcs/logs/job08/update.log Status on github-sync3.dmz.scl3.mozilla.com: -------------------------------------------- [vcs2vcs@github-sync3.dmz.scl3 .git]$ ps -flwwwp 3032 F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 S vcs2vcs 3032 3031 0 80 0 - 50200 sk_wai 14:54 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-pt-BR-gaia pull [vcs2vcs@github-sync3.dmz.scl3 .git]$ ls -l /proc/!$/fd ls -l /proc/3032/fd total 0 lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 10 15:07 0 -> /dev/pts/9 l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 10 15:07 1 -> /opt/vcs2vcs/logs/job08/details.log l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 10 14:55 2 -> /opt/vcs2vcs/logs/job08/details.log l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 10 15:07 3 -> /opt/vcs2vcs/logs/job08/update.log lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 10 15:07 4 -> socket:[113644572] [vcs2vcs@github-sync3.dmz.scl3 .git]$ socket:[113644572] no longer exists
Summary ======== Documented recovery procedure worked: http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/file/be390f8fcad2/docs/trouble_shooting.rst#l14 last good push: 2013-01-11T05:11:09+0000 next good push: 2013-01-11T06:22:22+0000 total delay: 1:11:13 Details ======= Email alert: ------------ -------- Original Message -------- Subject: [vcs2vcs] process delays Date: Fri, 11 Jan 2013 05:47:01 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com check these 2 items: process not running for stale file '/opt/vcs2vcs/b2g/details.log' process not running for stale file '/opt/vcs2vcs/b2g/update.log' Note that error message incorrect, but problem real Status on github-sync3.dmz.scl3.mozilla.com: -------------------------------------------- [vcs2vcs@github-sync3.dmz.scl3 ~]$ cd b2g [vcs2vcs@github-sync3.dmz.scl3 b2g]$ ll *log -rw-r----- 1 vcs2vcs vcs2vcs 6401024 Jan 11 05:21 details.log -rw-r----- 1 vcs2vcs vcs2vcs 8797607 Jan 11 05:21 update.log [vcs2vcs@github-sync3.dmz.scl3 b2g]$ date Fri Jan 11 05:54:41 UTC 2013 [vcs2vcs@github-sync3.dmz.scl3 b2g]$ pstree -p $USER screen(21936)─┬─bash(5441) └─bash(25627)───bash(8142)───keep_updated.ne(28337)───timeit(30193)───time(30196)───hg(30197) [vcs2vcs@github-sync3.dmz.scl3 b2g]$ ps -flwwwwp 30197 F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 S vcs2vcs 30197 30196 0 80 0 - 51608 sk_wai 05:21 pts/2 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r fb011a7d8798 b2g18 [vcs2vcs@github-sync3.dmz.scl3 b2g]$ ls -l /proc/!$/fd ls -l /proc/30197/fd total 0 lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 11 05:47 0 -> /dev/pts/2 l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 05:47 1 -> /opt/vcs2vcs/b2g/details.log l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 05:30 2 -> /opt/vcs2vcs/b2g/details.log l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 05:47 3 -> /opt/vcs2vcs/b2g/update.log lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 11 05:47 4 -> socket:[115316066] socket:[115316066] no longer exists
(In reply to Hal Wine [:hwine] from comment #2) Summary ======== Documented recovery procedure worked: http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/file/ be390f8fcad2/docs/trouble_shooting.rst#l14 last good push: 2013-01-11T06:43:16+0000 next good push: 2013-01-11T08:36:32+0000 total delay: 1:54:16 Details ======= Email alert: --------- Subject: Cron <vcs2vcs@github-sync3> $HOME/bin/check_process_delay Date: Fri, 11 Jan 2013 07:07:01 +0000 (UTC) Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull 0 S vcs2vcs 11639 8844 0 80 0 - 26523 wait 06:43 pts/9 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull Unknown delay: wait S+ /usr/bin/time -f 2013-01-11T06:43:16+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull 0 S vcs2vcs 11642 11639 0 80 0 - 1015 wait 06:43 pts/9 00:00:00 /usr/bin/time -f 2013-01-11T06:43:16+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull socket hang on pid 11643 0 S vcs2vcs 11643 11642 0 80 0 - 50341 sk_wai 06:43 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull check these 6 items: Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull 0 S vcs2vcs 11639 8844 0 80 0 - 26523 wait 06:43 pts/9 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull Unknown delay: wait S+ /usr/bin/time -f 2013-01-11T06:43:16+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull 0 S vcs2vcs 11642 11639 0 80 0 - 1015 wait 06:43 pts/9 00:00:00 /usr/bin/time -f 2013-01-11T06:43:16+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull socket hang on pid 11643 0 S vcs2vcs 11643 11642 0 80 0 - 50341 sk_wai 06:43 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull ------------ Status on github-sync3.dmz.scl3.mozilla.com: -------------------------------------------- [vcs2vcs@github-sync3.dmz.scl3 etc]$ pstree -p $USER screen(21936)─┬─bash(5441)───pstree(30641) └─bash(5931)───do_update(8844)───timeit(11639)───time(11642)───hg(11643) [vcs2vcs@github-sync3.dmz.scl3 etc]$ ls -lR /proc/11643/fd /proc/11643/fd: total 0 lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 11 07:07 0 -> /dev/pts/9 l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 07:07 1 -> /opt/vcs2vcs/logs/job08/details.log l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 06:50 2 -> /opt/vcs2vcs/logs/job08/details.log l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 11 07:07 3 -> /opt/vcs2vcs/logs/job08/update.log lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 11 07:07 4 -> socket:[115484367] > > socket:[115484367] no longer exists
Summary ======= Documented recovery procedure worked: http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/file/be390f8fcad2/docs/trouble_shooting.rst#l14 last good push: 2013-01-13T01:24+0000 next good push: 2013-01-13T08:13+0000 total delay: 6:51:00 Details ======= Email alert: ------------ Subject: Cron <vcs2vcs@github-sync3> $HOME/bin/check_process_delay Date: Sun, 13 Jan 2013 01:47:01 +0000 (UTC) From: root@github-sync3.dmz.scl3.mozilla.com (Cron Daemon) To: vcs2vcs@github-sync3.dmz.scl3.mozilla.com Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta 0 S vcs2vcs 14045 28337 0 80 0 - 26523 wait 01:35 pts/2 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta Unknown delay: wait S+ /usr/bin/time -f 2013-01-13T01:35:01+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/b2g/update.log hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta 0 S vcs2vcs 14048 14045 0 80 0 - 1015 wait 01:35 pts/2 00:00:00 /usr/bin/time -f 2013-01-13T01:35:01+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/b2g/update.log hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta socket hang on pid 14049 0 S vcs2vcs 14049 14048 0 80 0 - 51609 sk_wai 01:35 pts/2 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta check these 6 items: Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta 0 S vcs2vcs 14045 28337 0 80 0 - 26523 wait 01:35 pts/2 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta Unknown delay: wait S+ /usr/bin/time -f 2013-01-13T01:35:01+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/b2g/update.log hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta 0 S vcs2vcs 14048 14045 0 80 0 - 1015 wait 01:35 pts/2 00:00:00 /usr/bin/time -f 2013-01-13T01:35:01+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/b2g/update.log hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta socket hang on pid 14049 0 S vcs2vcs 14049 14048 0 80 0 - 51609 sk_wai 01:35 pts/2 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta Status on github-sync3.dmz.scl3.mozilla.com: -------------------------------------------- [vcs2vcs@github-sync3.dmz.scl3 ~]$ cd b2g/ [vcs2vcs@github-sync3.dmz.scl3 b2g]$ ll *log -rw-r----- 1 vcs2vcs vcs2vcs 6618896 Jan 13 01:35 details.log -rw-r----- 1 vcs2vcs vcs2vcs 9111767 Jan 13 01:25 update.log [vcs2vcs@github-sync3.dmz.scl3 b2g]$ date Sun Jan 13 08:08:28 UTC 2013 [vcs2vcs@github-sync3.dmz.scl3 b2g]$ pstree -p $USER screen(21936)─┬─bash(5441) ├─bash(25627)───bash(8142)───keep_updated.ne(28337)─\ ──timeit(14045)───time(14048)───hg(14049) [vcs2vcs@github-sync3.dmz.scl3 b2g]$ ps -flwwwwp 14049 F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 S vcs2vcs 14049 14048 0 80 0 - 51609 sk_wai 01:35 pts/2 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 2bb8638b277e beta $ ls -l /proc/!$/fd ls -l /proc/14049/fd total 0 lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 13 01:47 0 -> /dev/pts/2 l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 13 01:47 1 -> /opt/vcs2vcs/b2g/details.log l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 13 01:47 2 -> /opt/vcs2vcs/b2g/details.log l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 13 01:47 3 -> /opt/vcs2vcs/b2g/update.log lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 13 01:47 4 -> socket:[121537471]
Another - first email: Subject: [vcs2vcs] process delays Date: Thu, 17 Jan 2013 10:07:01 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com check these 6 items: Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull 0 S vcs2vcs 9629 6888 0 80 0 - 26523 wait 08:52 pts/9 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull Unknown delay: wait S+ /usr/bin/time -f 2013-01-17T08:52:42+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull 0 S vcs2vcs 9632 9629 0 80 0 - 1015 wait 08:52 pts/9 00:00:00 /usr/bin/time -f 2013-01-17T08:52:42+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull socket hang on pid 9633 0 S vcs2vcs 9633 9632 0 80 0 - 50341 sk_wai 08:52 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
another: first email: -------- Original Message -------- Subject: [vcs2vcs] process delays Date: Sat, 19 Jan 2013 07:27:01 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com check these 6 items: Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull 0 S vcs2vcs 8629 8622 0 80 0 - 26523 wait 07:11 pts/9 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull Unknown delay: wait S+ /usr/bin/time -f 2013-01-19T07:11:23+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull 0 S vcs2vcs 8632 8629 0 80 0 - 1015 wait 07:11 pts/9 00:00:00 /usr/bin/time -f 2013-01-19T07:11:23+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull socket hang on pid 8633 0 S vcs2vcs 8633 8632 0 80 0 - 49135 sk_wai 07:11 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
Another one: -------- Original Message -------- Subject: [vcs2vcs] process delays Date: Sat, 19 Jan 2013 17:47:01 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com check these 6 items: Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull 0 S vcs2vcs 13992 13985 0 80 0 - 26523 wait 17:34 pts/9 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull Unknown delay: wait S+ /usr/bin/time -f 2013-01-19T17:34:08+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull 0 S vcs2vcs 13995 13992 0 80 0 - 1015 wait 17:34 pts/9 00:00:00 /usr/bin/time -f 2013-01-19T17:34:08+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull socket hang on pid 13996 0 S vcs2vcs 13996 13995 0 80 0 - 50200 sk_wai 17:34 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-ar-gaia pull
check these 6 items: Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull 0 S vcs2vcs 1896 1428 0 80 0 - 26523 wait 04:07 pts/9 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull Unknown delay: wait S+ /usr/bin/time -f 2013-01-21T04:07:15+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull 0 S vcs2vcs 1899 1896 0 80 0 - 1015 wait 04:07 pts/9 00:00:00 /usr/bin/time -f 2013-01-21T04:07:15+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull socket hang on pid 1900 0 S vcs2vcs 1900 1899 0 80 0 - 49134 sk_wai 04:07 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
Depends on: 833590
another one (auto workaround worked): -------- Original Message -------- Subject: [vcs2vcs] process delays Date: Wed, 23 Jan 2013 03:07:02 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com check these 6 items: Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull 0 S vcs2vcs 7651 4900 0 80 0 - 26523 wait 02:52 pts/9 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull Unknown delay: wait S+ /usr/bin/time -f 2013-01-23T02:52:13+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull 0 S vcs2vcs 7654 7651 0 80 0 - 1015 wait 02:52 pts/9 00:00:00 /usr/bin/time -f 2013-01-23T02:52:13+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull socket hang on pid 7655 0 S vcs2vcs 7655 7654 0 80 0 - 50342 sk_wai 02:52 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
Date: Wed, 23 Jan 2013 08:07:01 +0000 To: release+vcs2vcs@mozilla.com Subject: [vcs2vcs] process delays From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com check these 6 items: Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r e3803e97aeff b2g18 0 S vcs2vcs 26021 31723 0 80 0 - 26523 wait 07:44 pts/2 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r e3803e97aeff b2g18 Unknown delay: wait S+ /usr/bin/time -f 2013-01-23T07:44:11+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/b2g/update.log hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r e3803e97aeff b2g18 0 S vcs2vcs 26024 26021 0 80 0 - 1015 wait 07:44 pts/2 00:00:00 /usr/bin/time -f 2013-01-23T07:44:11+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/b2g/update.log hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r e3803e97aeff b2g18 socket hang on pid 26025 0 S vcs2vcs 26025 26024 0 80 0 - 49136 sk_wai 07:44 pts/2 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r e3803e97aeff b2g18
Date: Wed, 23 Jan 2013 09:07:35 +0000 To: release+vcs2vcs@mozilla.com Subject: [vcs2vcs] process delays From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com check these 6 items: Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull 0 S vcs2vcs 443 32673 0 80 0 - 26523 wait 08:50 pts/9 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull Unknown delay: wait S+ /usr/bin/time -f 2013-01-23T08:50:22+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull 0 S vcs2vcs 446 443 0 80 0 - 1015 wait 08:50 pts/9 00:00:00 /usr/bin/time -f 2013-01-23T08:50:22+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull socket hang on pid 447 0 S vcs2vcs 447 446 0 80 0 - 50461 sk_wai 08:50 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-es-ES-gecko pull
Another one, cleared automatically at 08:47 -------- Original Message -------- Subject: [vcs2vcs] process delays Date: Sat, 26 Jan 2013 16:27:02 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com check these 6 items: Unknown delay: wait S+ /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull 0 S vcs2vcs 16013 13048 0 80 0 - 26523 wait 16:00 pts/9 00:00:00 /bin/bash /opt/vcs2vcs/bin/timeit hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull Unknown delay: wait S+ /usr/bin/time -f 2013-01-26T16:00:33+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull 0 S vcs2vcs 16016 16013 0 80 0 - 1015 wait 16:00 pts/9 00:00:00 /usr/bin/time -f 2013-01-26T16:00:33+0000;%E;%e;%S;%U;%M;%c;%w;%I;%O;%r;%s;%x;%C; -a -o /opt/vcs2vcs/logs/job08/update.log hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull socket hang on pid 16017 0 S vcs2vcs 16017 16016 0 80 0 - 49136 sk_wai 16:00 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull
another one: (hwine-moz1)[vcs2vcs@github-sync3.dmz.scl3 tmp]$ lsof ~/b2g/[du]*log (hwine-moz1)[vcs2vcs@github-sync3.dmz.scl3 tmp]$ ps -flwwwwwp 13010 F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 S vcs2vcs 13010 14700 0 80 0 - 49138 sk_wai 00:02 pts/2 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg id http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0 (hwine-moz1)[vcs2vcs@github-sync3.dmz.scl3 tmp]$ ll /proc/13010/fd total 0 lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 29 00:42 0 -> /dev/pts/2 l-wx------ 1 vcs2vcs vcs2vcs 64 Jan 29 00:42 1 -> pipe:[175853002] lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 29 00:39 2 -> /dev/pts/2 lrwx------ 1 vcs2vcs vcs2vcs 64 Jan 29 00:42 3 -> socket:[175853010] (hwine-moz1)[vcs2vcs@github-sync3.dmz.scl3 tmp]$
Another - autofix worked: -------- Original Message -------- Subject: [vcs2vcs] AUTOFIX process delays Date: Tue, 29 Jan 2013 02:47:03 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com check these 2 items: Attempting unstick of hung process 28330 0 S vcs2vcs 28330 28329 0 80 0 - 50201 sk_wai 02:26 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-pt-BR-gaia pull
autofix successful -------- Original Message -------- Subject: [vcs2vcs] AUTOFIX process delays Date: Thu, 31 Jan 2013 13:07:02 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com: Attempting unstick of hung process 9324 0 S vcs2vcs 9324 9323 0 80 0 - 51685 sk_wai 12:47 pts/2 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 4d4116685156 beta (if fix is successful, no further email will be sent) (just add this email to https://bugzil.la/829025)
autofix successful -------- Original Message -------- Subject: [vcs2vcs] AUTOFIX process delays Date: Thu, 31 Jan 2013 21:47:02 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com: Attempting unstick of hung process 9923 0 S vcs2vcs 9923 9922 0 80 0 - 51695 sk_wai 21:35 pts/2 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/b2g/mc-cvs-ma-update-test3 pull -r 0565fdda9c81 b2g18_v1_0_0 (if fix is successful, no further email will be sent) (just add this email to https://bugzil.la/829025)
autofix successful -------- Original Message -------- Subject: [vcs2vcs] AUTOFIX process delays Date: Thu, 31 Jan 2013 22:47:02 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com: Attempting unstick of hung process 17739 0 S vcs2vcs 17739 17738 0 80 0 - 50344 sk_wai 22:34 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pt-BR-gecko pull (if fix is successful, no further email will be sent) (just add this email to https://bugzil.la/829025)
autofix successful -------- Original Message -------- Subject: [vcs2vcs] AUTOFIX process delays Date: Sat, 02 Feb 2013 18:47:03 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com: Attempting unstick of hung process 10137 0 S vcs2vcs 10137 10136 0 80 0 - 28342 sk_wai 18:39 pts/5 00:00:00 git --git-dir /opt/vcs2vcs/repos/b2g-android-device-crespo/.git fetch --force (if fix is successful, no further email will be sent) (just add this email to https://bugzil.la/829025)
autofix successful -------- Original Message -------- Subject: [vcs2vcs] AUTOFIX process delays Date: Mon, 11 Feb 2013 22:27:02 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com: Attempting unstick of hung process 22645 0 S vcs2vcs 22645 22644 0 80 0 - 50409 sk_wai 22:10 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-release-sk-gecko pull (if fix is successful, no further email will be sent) (just add this email to https://bugzil.la/829025)
autofix successful: -------- Original Message -------- Subject: [vcs2vcs] AUTOFIX process delays Date: Tue, 12 Feb 2013 20:47:02 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com: Attempting unstick of hung process 14069 0 S vcs2vcs 14069 14068 0 80 0 - 49172 sk_wai 20:24 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-aurora-nl-gecko pull (if fix is successful, no further email will be sent) (just add this email to https://bugzil.la/829025)
successful autofix -------- Original Message -------- Subject: [vcs2vcs] AUTOFIX process delays Date: Fri, 15 Feb 2013 07:27:02 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com: Attempting unstick of hung process 26668 0 S vcs2vcs 26668 26667 0 80 0 - 50666 sk_wai 06:59 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-de-gecko pull (if fix is successful, no further email will be sent) (just add this email to https://bugzil.la/829025)
successful autofix -------- Original Message -------- Subject: [vcs2vcs] AUTOFIX process delays Date: Fri, 15 Feb 2013 16:27:03 +0000 From: vcs2vcs@github-sync3.dmz.scl3.mozilla.com To: release+vcs2vcs@mozilla.com report from /opt/vcs2vcs/bin/check_process_delay on github-sync3.dmz.scl3.mozilla.com: Attempting unstick of hung process 27216 0 S vcs2vcs 27216 27215 0 80 0 - 50686 sk_wai 16:04 pts/9 00:00:00 /opt/vcs2vcs/venv/hg/bin/python /opt/vcs2vcs/bin/hg --cwd /opt/vcs2vcs/repos/releases-l10n-beta-pl-gecko pull (if fix is successful, no further email will be sent) (just add this email to https://bugzil.la/829025)
Product: mozilla.org → Release Engineering
No updates for over 11 months; I think we're assuming autofixes work and filing new bugs if they aren't.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
agreed - hopefully when the new system starts pulling from hg, this won't be encountered.
report from /home/vcs2vcs/bin/check_process_delay on ip-172-31-26-60: Attempting unstick of hung process 3584 0 S vcs2vcs 3584 3576 0 80 0 - 39467 poll_s 12:10 pts/11 00:00:00 git-remote-http origin http://sprdsource.spreadtrum.com:8085/b2g/android/platform/external/jack
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
report from /home/vcs2vcs/bin/check_process_delay on ip-172-31-26-60: Attempting unstick of hung process 3198 0 S vcs2vcs 3198 3196 0 80 0 - 39467 poll_s 14:59 pts/11 00:00:00 git-remote-http origin http://sprdsource.spreadtrum.com:8085/b2g/android/device/sprd (if fix fails, another email will come within 5 minutes) (just add this email to https://bugzil.la/829025)
Old vcs-sync is dead.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 8 years ago
Resolution: --- → WONTFIX
Component: Tools → General
You need to log in before you can comment on or make changes to this bug.