798361 - Make release automation work with mock slaves [when 18 merges to beta]

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Comment 1

[John Hopkins (:jhopkins)]

Attachment: look for python in the PATH before using hard coded value

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 673232 [details] [diff] [review]
look for python in the PATH before using hard coded value

Review of attachment 673232 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/release/tagging.sh
@@ +5,5 @@
>  SCRIPTS_DIR="$MY_DIR/../../"
> +PYTHON=`which python`
> +if [ -z "$PYTHON" ]; then
> +    PYTHON=/tools/python/bin/python
> +fi

This would point to default python2.4 on the old machines. You need to test if /tools/python/bin/python exists, then use system python. Something like:

PYTHON=$(test -x /tools/python/bin/python || which python)

Comment 3

[John Hopkins (:jhopkins)]

Attachment: buildbot-configs patch

Comment 4

[John Hopkins (:jhopkins)]

Attachment: thunderbird buildbot-configs changes (wip, needs testing)

Comment 5

[John Hopkins (:jhopkins)]

Attachment: updated tools patch

Addressed feedback and kept '-x' addition because I think there's enough going on here that it would be useful log output to have.

Comment 6

[John Hopkins (:jhopkins)]

Attachment: buildbotcustom patch

(note: obsoleting old tools patch at the same time)

Comment 7

[John Hopkins (:jhopkins)]

Attachment: mozharness patch

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 673285 [details] [diff] [review]
buildbot-configs patch

Review of attachment 673285 [details] [diff] [review]:
-----------------------------------------------------------------

I was unable to pass even the first step of the release process (tagging). It was failing with the following:

remote: Warning: Identity file ~cltbld/.ssh/ffxbld_dsa not accessible: No such file or directory.
remote: Permission denied (publickey,keyboard-interactive).
abort: no suitable response from remote hg!

It comes from releaseConfig['hgSshKey']. cltbld doesn't exist under mock. Changing ~cltbld/.ssh/ to ~/.ssh/ may work.

However, I don't understand why you need to change HOME environment variable... Looks like setting it doesn't work at all. I believe that HOME is special.

::: mozilla/config.py
@@ +131,5 @@
>              'enable_shared_checkouts': True,
>              'env': {
>                  'DISPLAY': ':2',
>                  'HG_SHARE_BASE_DIR': '/builds/hg-shared',
> +                'HOME': '/home/mock_mozilla',

This one doesn't work as expected:

$ mock_mozilla  -r mozilla-centos6-i386 --cwd /builds/slave/rel-m-beta-firefox-tag/. --unpriv --shell '/usr/bin/env HOME="/home/mock_mozilla" echo "HOME: $HOME"' 
INFO: mock_mozilla.py version 1.0.3 starting...
State Changed: init plugins
INFO: selinux disabled
State Changed: start
State Changed: lock buildroot
State Changed: shell
HOME: /builds
State Changed: unlock buildroot

Additionally it may change a lot of other things. At least it may break old style releases.

@@ +175,5 @@
>              'mock_copyin_files': [
>                  ('/home/cltbld/.ssh', '/home/mock_mozilla/.ssh'),
> +                ('/home/cltbld/.android', '/home/mock_mozilla/.android'),
> +                ('/home/cltbld/.mozpass.cfg', '/home/mock_mozilla/.mozpass.cfg'),
> +                ('/home/cltbld/.hgrc', '/home/mock_mozilla/.hgrc'),

These changes break hg share extension, what means that at least .hgrc goes under a wrong directory. (hg: unknown command 'share' in the logs)

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 673287 [details] [diff] [review]
thunderbird buildbot-configs changes (wip, needs testing)

See my comments for config.py

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 673295 [details] [diff] [review]
buildbotcustom patch

Review of attachment 673295 [details] [diff] [review]:
-----------------------------------------------------------------

in overall it looks good, except this blocker:

::: process/release.py
@@ +965,5 @@
>                  partner_repack_factory = ScriptFactory(
>                      scriptRepo=mozharness_repo,
>                      scriptName=mh_cfg['script'],
>                      extra_args=extra_args,
> +                    use_mock=use_mock(platform),

Partner repack factory shouldn't use mock, because we can run it only on mack slaves. It didn't pass checkconfig when I enabled partner repacks in the configs.

Either remove mock_* parameter, or explicitly set use_mock to False here.

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 673299 [details] [diff] [review]
mozharness patch

the same issue as with config.py...

Comment 12

[John Hopkins (:jhopkins)]

Attachment: [buildbot-configs] updated patch

Comment 13

[John Hopkins (:jhopkins)]

Attachment: [buildbotcustom] updated patch

Comment 14

[John Hopkins (:jhopkins)]

Attachment: [buildbotcustom] updated patch

Comment 15

[Rail Aliiev [:rail]]

Comment on attachment 677825 [details] [diff] [review]
[buildbotcustom] updated patch

lgtm

Comment 16

[Chris AtLee [:catlee]]

Comment on attachment 677825 [details] [diff] [review]
[buildbotcustom] updated patch

Review of attachment 677825 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm too.

what diffs does this generate with the existing release configs?

Comment 17

[John Hopkins (:jhopkins)]

Comment on attachment 677825 [details] [diff] [review]
[buildbotcustom] updated patch

Landed in http://hg.mozilla.org/build/buildbotcustom/rev/bb086f2367c3

Comment 18

[John Hopkins (:jhopkins)]

Comment on attachment 673288 [details] [diff] [review]
updated tools patch

Landed in https://hg.mozilla.org/build/tools/rev/d062409a82bc

Comment 19

[Kim Moir [:kmoir] ET]

In production

Comment 20

[John Hopkins (:jhopkins)]

Attachment: [buildbot-configs] updated patch

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 680643 [details] [diff] [review]
[buildbot-configs] updated patch

lgtm!

Comment 22

[John Hopkins (:jhopkins)]

Comment on attachment 680643 [details] [diff] [review]
[buildbot-configs] updated patch

Landed in https://hg.mozilla.org/build/buildbot-configs/rev/4f68d76200ef

Comment 23

[John Hopkins (:jhopkins)]

Comment on attachment 680643 [details] [diff] [review]
[buildbot-configs] updated patch

in production

Comment 24

[John Hopkins (:jhopkins)]

The mozharness repo patches were neglected.  Those were landed today in:

 http://hg.mozilla.org/build/mozharness/rev/dac0d112a71f
 http://hg.mozilla.org/build/mozharness/rev/2f1d18c735af