472779 - Mobile ARM builds failing when pulling buildbot-configs

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Not very many green builds recently, the error is:

/scratchbox/moz_scratchbox -p -d build/buildbot-configs hg pull -u
 in dir /builds/moz2_slave/mobile-linux-arm-dep/build (timeout 1200 secs)
eval eval /builds/scratchbox/sbin/chroot-uid /builds/scratchbox/users/cltbld /scratchbox/tools/bin/env - $SBENV /scratchbox/tools/bin/bash -l /scratchbox/tools/bin/exec.sh build/buildbot-configs hg pull -u
pulling from http://hg.mozilla.org/build/buildbot-configs
searching for changes
adding changesets
adding manifests
adding file changes
added 4 changesets with 4 changes to 1 files
abort: Is a directory - /home/cltbld/build/buildbot-configs/mozilla2-staging/macosx/mozilla-1.9.1
program finished with exit code 255

Probably need to blow away some checkouts so that the symlinks work

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

The problem is that the scratchbox includes Mercurial 0.9.1, while you need at least 0.9.4 to support symlinks (there are several symlink-related fixes in later releases too). As is, a fresh pull converts the symlinks to copies; a no change pulls work just fine, but any changes in the repo give the error in the initial report.

Options
* update hg to the latest version (the copy inside scratchbox)
* workaround by removing the buildbot-configs checkout then clone (on each run)

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Gavin suggests using the hg from the host environment, which sounds like a nice simplification of our setup so I'll work up a patch.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Use hg from host rather than scratchbox copy

This works in staging if you remove the old pull which doesn't have symlinks; the linux slave has hg v0.9.5. Also fixes minor things along the way (double / in mobile pushlog url, slave lists in staging & prod).

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Pushed into use on moz2-master, and removed existing tools clones.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

*sigh* Apparently slave07, 08, and 10 don't have scratchbox set up properly.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I've commented out slave07, 08 and 10 in mobile_master.cfg and pushed that up to hg.m.o - it'll be picked up on tomorrow's buildbot restart.
  http://hg.mozilla.org/build/buildbot-configs/rev/18d20ea288c1

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

The other slaves have been rehabilitated, just need to verify this fixed once the wonky slaves are out of the list of mobile-arm builders tomorrow.

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #5)
> *sigh* Apparently slave07, 08, and 10 don't have scratchbox set up properly.

...and slave09 also.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Builds have been green all day, with a couple of pushes to the buildbot-configs repo. Resolving FIXED.