742233 - hg2.build.scl1:/integration/mozilla-inbound and /try are out of sync

Status: RESOLVED DUPLICATE of bug 733572

(Developer Services :: General, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

They lost sync about 22:20.

Comment 1

[Amy Rich [:arr] [:arich]]

I ran /root/sync_all.sh which should be keeping things in sync from cron.    It looks like there were some stuck process.  There were also a bunch of defunct sh processes and some defunct supervise processes.  I suspect there may be more stuff that needs to be cleaned out. Passing this over to bkero for further investigation.


root      8597  8596  0 Apr03 ?        00:00:00 /bin/sh -c /sbin/pidof bash sync_all.sh > /dev/null || /root/sync_all.sh > /dev/null
root      8599     1  0 Apr03 ?        00:00:00 /bin/bash /root/sync_all.sh
root      8701     1  0 Apr03 ?        00:00:00 python /usr/local/bin/check_hg_mirrorsync
hg        5337  5304  0 Apr03 ?        00:00:00 /usr/bin/python /usr/bin/hg pull --config hooks.pretxnchangegroup.z_linearhistory= --config hooks.pretxnchangegroup.z_loghistory=
root      6379     1  0 Mar20 ?        00:00:00 /usr/bin/python /usr/bin/hg pull
root      8692     1  0 Apr03 ?        00:00:00 /usr/bin/python /usr/bin/hg pull
root      8701     1  0 Apr03 ?        00:00:00 python /usr/local/bin/check_hg_mirrorsync
root      8932  8923  0 Apr03 ?        00:00:00 /usr/bin/python /usr/bin/hg pull --config hooks.pretxnchangegroup.z_linearhistory= --config hooks.pretxnchangegroup.z_loghistory=


pulling from http://hg.mozilla.org/mozilla-central
searching for changes
adding changesets
adding manifests
adding file changes
added 59 changesets with 245 changes to 187 files
(run 'hg update' to get a working copy)
~
/repo/hg/mozilla/releases/mozilla-aurora ~
pulling from http://hg.mozilla.org/releases/mozilla-aurora
searching for changes
no changes found
~
/repo/hg/mozilla/releases/mozilla-beta ~
pulling from http://hg.mozilla.org/releases/mozilla-beta
searching for changes
no changes found
~
/repo/hg/mozilla/try ~
pulling from http://hg.mozilla.org/try
searching for changes
adding changesets
adding manifests
adding file changes
added 128 changesets with 385 changes to 631 files (+23 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)
~
/repo/hg/mozilla/hgcustom/hghooks ~
pulling from http://hg.mozilla.org/hgcustom/hghooks
searching for changes
no changes found
~
/repo/hg/mozilla/hgcustom/hg_templates ~
pulling from http://hg.mozilla.org/hgcustom/hg_templates
searching for changes
no changes found
~
/repo/hg/mozilla/integration/mozilla-inbound ~
pulling from http://hg.mozilla.org/integration/mozilla-inbound
searching for changes
adding changesets
adding manifests
adding file changes
added 21 changesets with 209 changes to 174 files
(run 'hg update' to get a working copy)
~
/repo/hg/mozilla/build/tools ~
pulling from http://hg.mozilla.org/build/tools
searching for changes
no changes found
~
/repo/hg/mozilla/build/buildbot-configs ~
pulling from http://hg.mozilla.org/build/buildbot-configs
searching for changes
no changes found
~

Comment 2

[Amy Rich [:arr] [:arich]]

Additionally, it appears that the job to run /usr/local/bin/check_hg_mirrorsync is not functioning correctly.

root     18842 18840  0 Apr03 pts/0    00:00:00 supervise check-hg-mirrorsync
root     31675 18842  0 07:25 pts/0    00:00:00 /bin/sh /usr/local/bin/loop -i 180 /usr/local/bin/check_hg_mirrorsync


I've killed off the loop job and supervise has correctly respawned it.  Then the check happens once and never appears to again (it sits there and waits) and the check data becomes stale and makes nagios alert.

Comment 3

[Ben Kero [:bkero]]

I've disabled the mirrorsync script and gone with a local cronjob to update these.  Emails will be sent out if/when the repositories go out of sync again.  We're now investigating whether we actually need these scl1 mirrors too.  bug # 733572 deals with this, and is used as a tracking bug