hgtool.py doesn't work for thunderbird repacks

RESOLVED FIXED

Status

Release Engineering
Release Automation: Other
P3
normal
RESOLVED FIXED
6 years ago
3 years ago

People

(Reporter: standard8, Assigned: bhearsum)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 1 obsolete attachment)

(Reporter)

Description

6 years ago
Bug 756351 worked around hgtool failing on the Thunderbird repacks by removing the hgtool argument that is passed to client.py. As a temporary work around, we need to add it back.

Sample from that bug:

command: START
command: hg pull -b default http://hg.mozilla.org/releases/mozilla-beta
command: cwd: e:\builds\hg-shared\releases\mozilla-beta
command: output:
pulling from http://hg.mozilla.org/releases/mozilla-beta
searching for changes
adding changesets
adding manifests
adding file changes
added 17 changesets with 40 changes to 39 files
(run 'hg update' to get a working copy)
command: END (14.83s elapsed)

Trying to share e:/builds/hg-shared\releases/mozilla-beta to e:\builds\moz2_slave\tb-rel-comm-beta-w32-rpk-1\comm-beta\mozilla
command: START
command: hg share -U e:/builds/hg-shared\\\\releases/mozilla-beta e:\\\\builds\\\\moz2_slave\\\\tb-rel-comm-beta-w32-rpk-1\\\\comm-beta\\\\mozilla
command: cwd: e:\builds\moz2_slave\tb-rel-comm-beta-w32-rpk-1\comm-beta
command: output:
command: END (0.19s elapsed)

command: START
command: hg branch
command: cwd: e:\builds\moz2_slave\tb-rel-comm-beta-w32-rpk-1\comm-beta\mozilla
command: output:
default

command: END (0.16 elapsed)

command: START
command: hg update -C
command: cwd: e:\builds\moz2_slave\tb-rel-comm-beta-w32-rpk-1\comm-beta\mozilla
command: output:
56767 files updated, 0 files merged, 0 files removed, 0 files unresolved
command: END (373.23s elapsed)

command: START
command: hg parent --template {node|short}
command: cwd: e:\builds\moz2_slave\tb-rel-comm-beta-w32-rpk-1\comm-beta\mozilla
command: output:
TinderboxPrint: <a href="http://hg.mozilla.org/releases/mozilla-beta/rev/5f412ea09aba">revision: 5f412ea09aba</a>
5f412ea09aba
command: END (0.69 elapsed)

abort: unknown revision 'THUNDERBIRD_13_0b2_RELEASE'!
Time: real 8.179 secs (user 6.672+0.000 sys 0.297+0.000)
Executing command: ['python', '../scripts/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-beta', 'e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta\\mozilla']
Executing command: ['hg', 'update', '-r', 'THUNDERBIRD_13_0b2_RELEASE', '-R', 'e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta\\mozilla', '--verbose', '--time']
Traceback (most recent call last):
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta/client.py", line 623, in <module>
    do_hg_pull('mozilla', options.mozilla_repo, options.hg, options.mozilla_rev, options.hgtool)
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta/client.py", line 343, in do_hg_pull
    check_call_noisy(cmd, retryMax=0)
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta/client.py", line 138, in check_call_noisy
    execute_check_call(cmd, *args, **kwargs)
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta/client.py", line 134, in execute_check_call
    check_call(cmd, *args, **kwargs)
  File "d:\mozilla-build\python25\Lib\subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['hg', 'update', '-r', 'THUNDERBIRD_13_0b2_RELEASE', '-R', 'e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/comm-beta\\mozilla', '--verbose', '--time']' returned non-zero exit status -1
command: END (402.16s elapsed)

make: *** [run_client_py] Error 1
Traceback (most recent call last):
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/scripts/scripts/l10n/create-release-repacks.py", line 233, in <module>
    oldBuildNumber=releaseConfig["oldBuildNumber"],
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/scripts/scripts/l10n/create-release-repacks.py", line 69, in createRepacks
    localeSrcDir, env)
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/scripts/scripts/l10n\../../lib/python\build\l10n.py", line 50, in l10nRepackPrep
    env=env)
  File "e:/builds/moz2_slave/tb-rel-comm-beta-w32-rpk-1/scripts/scripts/l10n\../../lib/python\util\commands.py", line 42, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "d:\mozilla-build\python25\Lib\subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-f', 'client.mk', 'configure']' returned non-zero exit status 2
program finished with exit code 1
elapsedTime=538.188000
(Reporter)

Comment 1

6 years ago
The basic issue here is that hgtool.py doesn't know that we're dealing with a release branch. Therefore it pulls on the default branch.

If the repository isn't up to date wrt the default branch, then we'll get the latest, probably including the branch changesets, however, if the default branch already contains the latest, then we won't pull, even if the non-default branches are out of date. It does make me ponder how the Firefox builders generally get away with this.

hgtool.py seems to know about being passed a branch argument, which may fix this, not sure how difficult it would be to pass in though.
(Assignee)

Comment 2

6 years ago
I'm not sure I understand what the issue here is....
(Reporter)

Comment 3

6 years ago
I'll try and expand:

1) On any given build, hg share will pull a repo and store a copy locally.

2) We then come along and push tags and/or patches to the relbranch.

3) hgtool.py gets passed a revision, e.g THUNDERBIRD_13_0_RELEASE, however it is not passed the branch argument.

4) So hgtool.py pulls the repo from the server to the share with hg pull -b default

=> This is the problem stage. The way hg works is if you've pushed something to a branch, but not to default, then when you update a copy of the repo with "-b default" hg won't necessarily pick up what's on the branches, e.g. your tags or patches are missing.

5) Hence, we then update the working copy, and then try to update to THUNDERBIRD_13_0_RELEASE, but that hasn't actually been pulled because of the -b default issue.

I'm pretty sure we hit this with the build 2 of either 10.0.5esr or 13, but I can't find the documentation for it at the moment.
At a high level, this problem would go away if did 'hg pull' for all repos which don't end in 'try', while try still pulled by rev to minimize I/O.
It also affects anything that uses client.py, which includes source, en-US builds, repacks.
Summary: Re-enable hgtool for client.py cloning of mozilla-* in Thunderbird repacks → Re-enable hgtool for client.py cloning of mozilla-* in Thunderbird releases
(Assignee)

Comment 6

6 years ago
Ah, so this only happens for Thunderbird because the hgtool call that happens during configure isn't aware of the relbranch?
Could well be that. There is COMM_REVISION and MOZILLA_REVISION in the environment but they're the release tag rather than the relbranch.
(Reporter)

Comment 8

6 years ago
Ok, so now I see bug 737554, I realise why Firefox releases and the comm-* parts of Thunderbird releases aren't affected by this when I expected them to be - they aren't using hgtool yet.
Blocks: 737554

Comment 9

6 years ago
I think I might have hit this for TB beta2.
http://buildbot-master34.srv.releng.scl3.mozilla.com:8001/builders/release-comm-beta-linux64_build/builds/14/steps/compile/logs/stdio

command: START
command: hg update -C
command: cwd: /builds/slave/tb-rel-comm-beta-lnx64-bld/build/mozilla
command: output:
58052 files updated, 0 files merged, 0 files removed, 0 files unresolved
command: END (70.46s elapsed)

command: START
command: hg parent --template {node|short}
command: cwd: /builds/slave/tb-rel-comm-beta-lnx64-bld/build/mozilla
command: output:
271b0e7e0728
command: END (0.22 elapsed)

Got revision 271b0e7e0728
abort: unknown revision 'THUNDERBIRD_14_0b2_RELEASE'!
Time: real 7.330 secs (user 6.400+0.000 sys 0.120+0.000)
Executing command: ['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-beta', '/builds/slave/tb-rel-comm-beta-lnx64-bld/build/mozilla']
Executing command: ['hg', 'update', '-r', 'THUNDERBIRD_14_0b2_RELEASE', '-R', '/builds/slave/tb-rel-comm-beta-lnx64-bld/build/mozilla', '--verbose', '--time']
Traceback (most recent call last):
  File "/builds/slave/tb-rel-comm-beta-lnx64-bld/build/client.py", line 623, in <module>
    do_hg_pull('mozilla', options.mozilla_repo, options.hg, options.mozilla_rev, options.hgtool)
  File "/builds/slave/tb-rel-comm-beta-lnx64-bld/build/client.py", line 343, in do_hg_pull
    check_call_noisy(cmd, retryMax=0)
  File "/builds/slave/tb-rel-comm-beta-lnx64-bld/build/client.py", line 138, in check_call_noisy
    execute_check_call(cmd, *args, **kwargs)
  File "/builds/slave/tb-rel-comm-beta-lnx64-bld/build/client.py", line 134, in execute_check_call
    check_call(cmd, *args, **kwargs)
  File "/tools/python-2.5.1/lib/python2.5/subprocess.py", line 461, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['hg', 'update', '-r', 'THUNDERBIRD_14_0b2_RELEASE', '-R', '/builds/slave/tb-rel-comm-beta-lnx64-bld/build/mozilla', '--verbose', '--time']' returned non-zero exit status 255
make: *** [run_client_py] Error 1
program finished with exit code 2
elapsedTime=87.967177
(Reporter)

Updated

6 years ago
Depends on: 764284
(Reporter)

Updated

6 years ago
Depends on: 764323
(Reporter)

Updated

6 years ago
Depends on: 764325
(Assignee)

Comment 10

6 years ago
This seems like it should be a dupe of bug 764284......am I missing something?
(Reporter)

Comment 11

6 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #10)
> This seems like it should be a dupe of bug 764284......am I missing
> something?

Hmm, my most recent thinking is that that bug would fix hgtool and this bug would put all the mozconfigs back.
(Assignee)

Comment 12

6 years ago
(In reply to Mark Banner (:standard8) from comment #11)
> (In reply to Ben Hearsum [:bhearsum] from comment #10)
> > This seems like it should be a dupe of bug 764284......am I missing
> > something?
> 
> Hmm, my most recent thinking is that that bug would fix hgtool and this bug
> would put all the mozconfigs back.

ok, wfm.
(Assignee)

Updated

6 years ago
Assignee: nobody → bhearsum
(Assignee)

Comment 13

6 years ago
Oops, didn't mean to take this bug.
Assignee: bhearsum → nobody
(Reporter)

Updated

6 years ago
No longer depends on: 764323
(Assignee)

Comment 14

6 years ago
Created attachment 634449 [details] [diff] [review]
update buildbot-configs mozconfigs
Attachment #634449 - Flags: review?(mbanner)
(Assignee)

Comment 15

6 years ago
Looks like this is mine after all....
Assignee: nobody → bhearsum
(Reporter)

Comment 16

6 years ago
Comment on attachment 634449 [details] [diff] [review]
update buildbot-configs mozconfigs

Looks good. Thanks.
Attachment #634449 - Flags: review?(mbanner) → review+
(Assignee)

Updated

6 years ago
Attachment #634449 - Flags: checked-in+
(Assignee)

Comment 17

6 years ago
All done here, I think?
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
The 14.0b3 repacks are failing because the l10n mozconfigs should point to ../scripts instead of ../tools.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 19

6 years ago
Created attachment 634854 [details] [diff] [review]
adjust path for l10n

Damn, nothing's ever simple :(. Sorry about that.
Attachment #634854 - Flags: review?(rail)
(Assignee)

Comment 20

6 years ago
Created attachment 634856 [details] [diff] [review]
fix l10n, get rid of errant CXX change
Attachment #634856 - Flags: review?(rail)
(Assignee)

Updated

6 years ago
Attachment #634854 - Attachment is obsolete: true
Attachment #634854 - Flags: review?(rail)
Attachment #634856 - Flags: review?(rail) → review+
(Assignee)

Comment 21

6 years ago
Comment on attachment 634856 [details] [diff] [review]
fix l10n, get rid of errant CXX change

And retagged.
Attachment #634856 - Flags: checked-in+
(Assignee)

Comment 22

6 years ago
Created attachment 634866 [details] [diff] [review]
backout hgtool.py for l10n

Now we're hitting an issue where it's trying to update mozilla-beta to 'releases/comm-beta', because that's what 'branch' is in the properties file. I thought this was already working, modulo bug 764284, but it's apparent it's not for repacks =\. Need to back this out for them for now, production is not the place to debug.
Attachment #634866 - Flags: review?(mbanner)
(Reporter)

Updated

6 years ago
Attachment #634866 - Flags: review?(mbanner) → review+
(Assignee)

Comment 23

6 years ago
Comment on attachment 634866 [details] [diff] [review]
backout hgtool.py for l10n

Retagged, too.
Attachment #634866 - Flags: checked-in+
(Assignee)

Comment 24

6 years ago
I'm not going to have time to debug and test hgtool.py for l10n, back to the pool with this.
Summary: Re-enable hgtool for client.py cloning of mozilla-* in Thunderbird releases → hgtool.py doesn't work for thunderbird repacks
(Assignee)

Updated

6 years ago
Assignee: bhearsum → nobody
Merged to production today
(Assignee)

Comment 26

6 years ago
I really don't want to, but there's doesn't seem to be anybody able to take this on right now, so I'm P3'ing it.
Priority: -- → P3
Product: mozilla.org → Release Engineering
(Reporter)

Updated

3 years ago
Blocks: 1060195
I will fix the remaining issue in bug 764323, I believe this bug can be closed. Let's file new bugs if more issues arise.
Assignee: nobody → bhearsum
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.