631542 - Verify output from hg pull & update, and retry clone if required

Status: VERIFIED FIXED

(Release Engineering :: General, defect, P3)

Description

[Axel Hecht [:Pike]]

win32-slave05 took a handful of nightlies down with it, see http://l10n.mozilla.org/~axel/nightlies/?date=2011-02-04

Comment 1

[Chris Cooper [:coop] (he/him)]

The root cause here was a hg clone command of m-c that timed out. Because of the way we do our clones (http://hg.mozilla.org/build/buildbotcustom/file/default/process/factory.py#l2623), the checkout dir was created but not (yet) fully populated. Subsequent attempts *should* fail when trying to pull+update, but it looks like hg isn't returning an error code, or we're ignoring it. We're certainly ignoring the error message in the logs (from buildbot):

# hg -R mozilla-central pull
abort: repository default not found!
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
# hg -R mozilla-central up -C
abort: unknown revision 'default'!

This step, get_enUS_src, is marked as passing. :(

I've clobbered the build dir for this slave which is a quick workaround for this specific failure, but I think this warrants a code fix.

Comment 2

[[e-mail] (:bhearsum)]

If the try-only landing of bug 589885 continues to go well it would probably be pretty easy to start using hgtool.py for this part of the l10n repack.

Additionally, bug 608005 would fix this.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

The dupe bug 632418 added some extra detail.  The tl;dr is (1) the shell command used to run hg commands ignores exit values and (2) sometimes hg returns success inappropriately anyway.

Comment 5

[Chris AtLee [:catlee]]

Another example:

sh -c if [ -d mozilla-central ]; then hg -R mozilla-central pull ;hg -R mozilla-central up -C ;else hg clone http://hg.mozilla.org/mozilla-central mozilla-central ; fi && hg -R mozilla-central update -C -r default

pulling from http://hg.mozilla.org/mozilla-central
searching for changes
abort: abandoned transaction found - run hg recover!
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
program finished with exit code 0

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Clobbered win32-slave13 on 1.9.2 and m-c for hg timeouts that lead to more busted l10n builds.

Comment 7

[Chris Cooper [:coop] (he/him)]

Attachment: Use hgtool.py for l10n clones

Comment 8

[Armen [:armenzg]]

Comment on attachment 527009 [details] [diff] [review]
Use hgtool.py for l10n clones

Looks good.
I like it that we now have a lock :)

Comment 9

[Chris Cooper [:coop] (he/him)]

Comment on attachment 527009 [details] [diff] [review]
Use hgtool.py for l10n clones

http://hg.mozilla.org/build/buildbotcustom/rev/d466a526c5a1

Comment 10

[Chris Cooper [:coop] (he/him)]

Needs a merge to production-0.8 and then a reconfig.

Comment 11

[Chris Cooper [:coop] (he/him)]

(In reply to comment #10)
> Needs a merge to production-0.8 and then a reconfig.

jhford got this merged and deployed this afternoon. 

Looking forward to a greener l10n nightlies dashboard.

Comment 12

[Rail Aliiev [:rail]]

Looks like Python on Windows doesn't like the Unix style paths:

python /e/builds/moz2_slave/cen-w32-l10n-ntly/tools/buildfarm/utils/hgtool.py --rev=default http://hg.mozilla.org/mozilla-central mozilla-central
 in dir e:\builds\moz2_slave\cen-w32-l10n-ntly\build (timeout 1800 secs)
 watching logfiles {}
 argv: ['python', '/e/builds/moz2_slave/cen-w32-l10n-ntly/tools/buildfarm/utils/hgtool.py', u'--rev=default', 'http://hg.mozilla.org/mozilla-central', 'mozilla-central']
 environment:
..........
python: can't open file '/e/builds/moz2_slave/cen-w32-l10n-ntly/tools/buildfarm/utils/hgtool.py': [Errno 2] No such file or directory
program finished with exit code 2

Comment 13

[Chris Cooper [:coop] (he/him)]

The same command works on Try for Windows, but it ends up as:

python e:/builds/moz2_slave/try-w32/tools/buildfarm/utils/hgtool.py ...

...rather than:

python /e/builds/moz2_slave/cen-w32-l10n-ntly/tools/buildfarm/utils/hgtool.py ...

That happens despite having the toolsdir build prop set to |/e/builds/moz2_slave/*| in both cases.

Must be something in the env I'm missing that's causing the drivepath to be set wrong for l10n. Will investigate.

Comment 14

[Chris Cooper [:coop] (he/him)]

Attachment: Set toolsdir differently on Windows

This is how we handle it for MercurialBuildFactory.

Comment 15

[Rail Aliiev [:rail]]

Comment on attachment 527937 [details] [diff] [review]
Set toolsdir differently on Windows

Looks good to me. You may also want to retrigger win32 nightlies after landing, merging to production-0.8 and reconfiguring the masters.

Comment 16

[Chris Cooper [:coop] (he/him)]

Comment on attachment 527937 [details] [diff] [review]
Set toolsdir differently on Windows

http://hg.mozilla.org/build/buildbotcustom/rev/f1fd8e05d73c

Comment 17

[Chris Cooper [:coop] (he/him)]

A Windows l10n build is cloning successfully right now.

Comment 18

[Rail Aliiev [:rail]]

Most of the win32 l10n repacks are green today. Yay!