757798 - hgtool.py doesn't work for thunderbird repacks

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[Mark Banner (:standard8)]

Bug 756351 worked around hgtool failing on the Thunderbird repacks by removing the hgtool argument that is passed to client.py. As a temporary work around, we need to add it back.

Sample from that bug:

command: START
command: hg pull -b default http://hg.mozilla.org/releases/mozilla-beta
command: cwd: e:\builds\hg-shared\releases\mozilla-beta
command: output:
pulling from http://hg.mozilla.org/releases/mozilla-beta
searching for changes
adding changesets
adding manifests
adding file changes
added 17 changesets with 40 changes to 39 files
(run 'hg update' to get a working copy)
command: END (14.83s elapsed)

Trying to share e:/builds/hg-shared\releases/mozilla-beta to e:\builds\moz2_slave\tb-rel-comm-beta-w32-rpk-1\comm-beta\mozilla
command: START
command: hg share -U e:/builds/hg-shared\\\\releases/mozilla-beta e:\\\\builds\\\\moz2_slave\\\\tb-rel-comm-beta-w32-rpk-1\\\\comm-beta\\\\mozilla
command: cwd: e:\builds\moz2_slave\tb-rel-comm-beta-w32-rpk-1\comm-beta
command: output:
command: END (0.19s elapsed)

command: START
command: hg branch
command: cwd: e:\builds\moz2_slave\tb-rel-comm-beta-w32-rpk-1\comm-beta\mozilla
command: output:
default

command: END (0.16 elapsed)

command: START
command: hg update -C
command: cwd: e:\builds\moz2_slave\tb-rel-comm-beta-w32-rpk-1\comm-beta\mozilla
command: output:
56767 files updated, 0 files merged, 0 files removed, 0 files unresolved
command: END (373.23s elapsed)

command: START
command: hg parent --template {node|short}
command: cwd: e:\builds\moz2_slave\tb-rel-comm-beta-w32-rpk-1\comm-beta\mozilla
command: output:
TinderboxPrint: <a href="http://hg.mozilla.org/releases/mozilla-beta/rev/5f412ea09aba">revision: 5f412ea09aba</a>
5f412ea09aba
command: END (0.69 elapsed)

abort: unknown revision 'THUNDERBIRD_13_0b2_RELEASE'!
Time: real 8.179 secs (user 6.672+0.000 sys 0.297+0.000)
Executing command: ['python', '../scripts/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-beta', 'e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta\\mozilla']
Executing command: ['hg', 'update', '-r', 'THUNDERBIRD_13_0b2_RELEASE', '-R', 'e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta\\mozilla', '--verbose', '--time']
Traceback (most recent call last):
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta/client.py", line 623, in <module>
    do_hg_pull('mozilla', options.mozilla_repo, options.hg, options.mozilla_rev, options.hgtool)
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta/client.py", line 343, in do_hg_pull
    check_call_noisy(cmd, retryMax=0)
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta/client.py", line 138, in check_call_noisy
    execute_check_call(cmd, *args, **kwargs)
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta/client.py", line 134, in execute_check_call
    check_call(cmd, *args, **kwargs)
  File "d:\mozilla-build\python25\Lib\subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['hg', 'update', '-r', 'THUNDERBIRD_13_0b2_RELEASE', '-R', 'e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta\\mozilla', '--verbose', '--time']' returned non-zero exit status -1
command: END (402.16s elapsed)

make: *** [run_client_py] Error 1
Traceback (most recent call last):
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/scripts/scripts/l10n/create-release-repacks.py", line 233, in <module>
    oldBuildNumber=releaseConfig["oldBuildNumber"],
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/scripts/scripts/l10n/create-release-repacks.py", line 69, in createRepacks
    localeSrcDir, env)
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/scripts/scripts/l10n\../../lib/python\build\l10n.py", line 50, in l10nRepackPrep
    env=env)
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/scripts/scripts/l10n\../../lib/python\util\commands.py", line 42, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "d:\mozilla-build\python25\Lib\subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-f', 'client.mk', 'configure']' returned non-zero exit status 2
program finished with exit code 1
elapsedTime=538.188000

Comment 1

[Mark Banner (:standard8)]

The basic issue here is that hgtool.py doesn't know that we're dealing with a release branch. Therefore it pulls on the default branch.

If the repository isn't up to date wrt the default branch, then we'll get the latest, probably including the branch changesets, however, if the default branch already contains the latest, then we won't pull, even if the non-default branches are out of date. It does make me ponder how the Firefox builders generally get away with this.

hgtool.py seems to know about being passed a branch argument, which may fix this, not sure how difficult it would be to pass in though.

Comment 2

[[e-mail] (:bhearsum)]

I'm not sure I understand what the issue here is....

Comment 3

[Mark Banner (:standard8)]

I'll try and expand:

1) On any given build, hg share will pull a repo and store a copy locally.

2) We then come along and push tags and/or patches to the relbranch.

3) hgtool.py gets passed a revision, e.g THUNDERBIRD_13_0_RELEASE, however it is not passed the branch argument.

4) So hgtool.py pulls the repo from the server to the share with hg pull -b default

=> This is the problem stage. The way hg works is if you've pushed something to a branch, but not to default, then when you update a copy of the repo with "-b default" hg won't necessarily pick up what's on the branches, e.g. your tags or patches are missing.

5) Hence, we then update the working copy, and then try to update to THUNDERBIRD_13_0_RELEASE, but that hasn't actually been pulled because of the -b default issue.

I'm pretty sure we hit this with the build 2 of either 10.0.5esr or 13, but I can't find the documentation for it at the moment.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

At a high level, this problem would go away if did 'hg pull' for all repos which don't end in 'try', while try still pulled by rev to minimize I/O.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

It also affects anything that uses client.py, which includes source, en-US builds, repacks.

Comment 6

[[e-mail] (:bhearsum)]

Ah, so this only happens for Thunderbird because the hgtool call that happens during configure isn't aware of the relbranch?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Could well be that. There is COMM_REVISION and MOZILLA_REVISION in the environment but they're the release tag rather than the relbranch.

Comment 8

[Mark Banner (:standard8)]

Ok, so now I see bug 737554, I realise why Firefox releases and the comm-* parts of Thunderbird releases aren't affected by this when I expected them to be - they aren't using hgtool yet.

Comment 9

[Armen [:armenzg]]

I think I might have hit this for TB beta2.
http://buildbot-master34.srv.releng.scl3.mozilla.com:8001/builders/release-comm-beta-linux64_build/builds/14/steps/compile/logs/stdio

command: START
command: hg update -C
command: cwd: /builds/slave/tb-rel-comm-beta-lnx64-bld/build/mozilla
command: output:
58052 files updated, 0 files merged, 0 files removed, 0 files unresolved
command: END (70.46s elapsed)

command: START
command: hg parent --template {node|short}
command: cwd: /builds/slave/tb-rel-comm-beta-lnx64-bld/build/mozilla
command: output:
271b0e7e0728
command: END (0.22 elapsed)

Got revision 271b0e7e0728
abort: unknown revision 'THUNDERBIRD_14_0b2_RELEASE'!
Time: real 7.330 secs (user 6.400+0.000 sys 0.120+0.000)
Executing command: ['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-beta', '/builds/slave/tb-rel-comm-beta-lnx64-bld/build/mozilla']
Executing command: ['hg', 'update', '-r', 'THUNDERBIRD_14_0b2_RELEASE', '-R', '/builds/slave/tb-rel-comm-beta-lnx64-bld/build/mozilla', '--verbose', '--time']
Traceback (most recent call last):
  File "/builds/slave/tb-rel-comm-beta-lnx64-bld/build/client.py", line 623, in <module>
    do_hg_pull('mozilla', options.mozilla_repo, options.hg, options.mozilla_rev, options.hgtool)
  File "/builds/slave/tb-rel-comm-beta-lnx64-bld/build/client.py", line 343, in do_hg_pull
    check_call_noisy(cmd, retryMax=0)
  File "/builds/slave/tb-rel-comm-beta-lnx64-bld/build/client.py", line 138, in check_call_noisy
    execute_check_call(cmd, *args, **kwargs)
  File "/builds/slave/tb-rel-comm-beta-lnx64-bld/build/client.py", line 134, in execute_check_call
    check_call(cmd, *args, **kwargs)
  File "/tools/python-2.5.1/lib/python2.5/subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['hg', 'update', '-r', 'THUNDERBIRD_14_0b2_RELEASE', '-R', '/builds/slave/tb-rel-comm-beta-lnx64-bld/build/mozilla', '--verbose', '--time']' returned non-zero exit status 255
make: *** [run_client_py] Error 1
program finished with exit code 2
elapsedTime=87.967177

Comment 10

[[e-mail] (:bhearsum)]

This seems like it should be a dupe of bug 764284......am I missing something?

Comment 11

[Mark Banner (:standard8)]

(In reply to Ben Hearsum [:bhearsum] from comment #10)
> This seems like it should be a dupe of bug 764284......am I missing
> something?

Hmm, my most recent thinking is that that bug would fix hgtool and this bug would put all the mozconfigs back.

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Mark Banner (:standard8) from comment #11)
> (In reply to Ben Hearsum [:bhearsum] from comment #10)
> > This seems like it should be a dupe of bug 764284......am I missing
> > something?
> 
> Hmm, my most recent thinking is that that bug would fix hgtool and this bug
> would put all the mozconfigs back.

ok, wfm.

Comment 13

[[e-mail] (:bhearsum)]

Oops, didn't mean to take this bug.

Comment 14

[[e-mail] (:bhearsum)]

Attachment: update buildbot-configs mozconfigs

Comment 15

[[e-mail] (:bhearsum)]

Looks like this is mine after all....

Comment 16

[Mark Banner (:standard8)]

Comment on attachment 634449 [details] [diff] [review]
update buildbot-configs mozconfigs

Looks good. Thanks.

Comment 17

[[e-mail] (:bhearsum)]

All done here, I think?

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

The 14.0b3 repacks are failing because the l10n mozconfigs should point to ../scripts instead of ../tools.

Comment 19

[[e-mail] (:bhearsum)]

Attachment: adjust path for l10n

Damn, nothing's ever simple :(. Sorry about that.

Comment 20

[[e-mail] (:bhearsum)]

Attachment: fix l10n, get rid of errant CXX change

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 634856 [details] [diff] [review]
fix l10n, get rid of errant CXX change

And retagged.

Comment 22

[[e-mail] (:bhearsum)]

Attachment: backout hgtool.py for l10n

Now we're hitting an issue where it's trying to update mozilla-beta to 'releases/comm-beta', because that's what 'branch' is in the properties file. I thought this was already working, modulo bug 764284, but it's apparent it's not for repacks =\. Need to back this out for them for now, production is not the place to debug.

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 634866 [details] [diff] [review]
backout hgtool.py for l10n

Retagged, too.

Comment 24

[[e-mail] (:bhearsum)]

I'm not going to have time to debug and test hgtool.py for l10n, back to the pool with this.

Comment 25

[Rail Aliiev [:rail]]

Merged to production today

Comment 26

[[e-mail] (:bhearsum)]

I really don't want to, but there's doesn't seem to be anybody able to take this on right now, so I'm P3'ing it.

Comment 27

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

I will fix the remaining issue in bug 764323, I believe this bug can be closed. Let's file new bugs if more issues arise.