601906 - l10n linux nightlies failed due to "tools" repo not being checked out properly

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Axel Hecht [:Pike]]

http://l10n.mozilla.org/~axel/nightlies/?date=2010-10-05&buildbot=true shows that the linux nightlies across branches are in deep slave desaster, apparently.

We're in high-noon for b7 localization, can we get the failed nightlies respun?

Also, not sure how wide we need to clobber stuff.

Comment 1

[Armen [:armenzg]]

This has to do that the "tools" repo was not checked out properly and the steps went forward. We deployed Mercurial 1.6 yesterday and it seems that .hgrc has to be absent so we won't hit this issue. Oddly enough this did not hit us with other types of builds.

> hg: parse error at /home/cltbld/.hgrc:2: merge: internal:other

Comment 2

[Armen [:armenzg]]

I have requested the machines to be clobbered for all linux branches.
I won't be triggering the l10n nightly repackages as I can't make sure that the machines have picked up the puppet changes that I have deployed as they have to reboot.

I will check in the morning to see what the state of affairs is and trigger whatever is left.

I hope this makes sense.

Comment 3

[Armen [:armenzg]]

The root of this bug has been fixed from looking at the dashboard.
It seems that there was some connection issues at around 4:30-4:40 to upload the packages but that should be another bug and I believe against Server Operations.

The problem with linux slaves also hit us in the try server and other places so it wasn't just L10n specific.
The problem when we deployed the Mercurial upgrade is that the slaves that took the jobs had not rebooted and were still running the older version of Mercurial (yay for not forcing reboot on idle machines!).

FIXING based on my comment. Let me know if there is anything left.

Thanks for filing!