1260415 - Schedule "partner repacks" builder

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Rail Aliiev [:rail]]

We need to schedule partner repacks as a part of release promotion.

Comment 1

[Rail Aliiev [:rail]]

Attachment: MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

Allow passing version and build_number via buildbot properties. This will allow
us add buildbot builders without hardcoded parameters depending on release
configs.

Review commit: https://reviewboard.mozilla.org/r/43219/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/43219/

Comment 2

[Rail Aliiev [:rail]]

Attachment: MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

Review commit: https://reviewboard.mozilla.org/r/43233/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/43233/

Comment 3

[Rail Aliiev [:rail]]

Attachment: MozReview Request: Bug 1260415 - buildboti-config: add "partner repacks" builder r=jlund

Review commit: https://reviewboard.mozilla.org/r/43237/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/43237/

Comment 4

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

https://reviewboard.mozilla.org/r/43219/#review39857

Comment 5

[Jordan Lund (:jlund)]

Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

https://reviewboard.mozilla.org/r/43233/#review39867

easy!

::: process/release.py:2016
(Diff revision 1)
> +            pf=branch_config["platforms"][platform],
> +            mh_cfg=mh_cfg)
> +
> +        builders.append({
> +            'name': buildername,
> +            'slavenames': branch_config['platforms']['macosx64']['slaves'],

my memory is bad. these partner builds use osx slaves every time regardless of platform it is based on?

Comment 6

[Jordan Lund (:jlund)]

Comment on attachment 8736365 [details]
MozReview Request: Bug 1260415 - buildboti-config: add "partner repacks" builder r=jlund

https://reviewboard.mozilla.org/r/43237/#review39865

Comment 7

[Rail Aliiev [:rail]]

(In reply to Jordan Lund (:jlund) from comment #5)
> my memory is bad. these partner builds use osx slaves every time regardless
> of platform it is based on?

Correct, http://hg.mozilla.org/build/buildbotcustom/file/d50064b10132/process/release.py#l915

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

https://hg.mozilla.org/build/buildbotcustom/rev/430fd7c357f7

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8736365 [details]
MozReview Request: Bug 1260415 - buildboti-config: add "partner repacks" builder r=jlund

https://hg.mozilla.org/build/buildbot-configs/rev/3e0f1271d6b1

Comment 10

[Chris Cooper [:coop] (he/him)]

(In reply to Jordan Lund (:jlund) from comment #5)
> my memory is bad. these partner builds use osx slaves every time regardless
> of platform it is based on?

Correct. Mac is the only platform that can do all of the required packaging.

Comment 11

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9e20e943e58b

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

https://hg.mozilla.org/integration/mozilla-inbound/rev/9e20e943e58b
https://hg.mozilla.org/releases/mozilla-aurora/rev/d4b132f5231d
https://hg.mozilla.org/releases/mozilla-beta/rev/e9c1a80d7c2e

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/9e20e943e58b

Comment 14

[Rail Aliiev [:rail]]

Attachment: MozReview Request: Bug 1260415 - tools: schedule partner repacks r=jlund

Review commit: https://reviewboard.mozilla.org/r/44267/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/44267/

Comment 15

[Rail Aliiev [:rail]]

Attachment: PR for releasetasks

Comment 16

[Jordan Lund (:jlund)]

Comment on attachment 8738028 [details]
MozReview Request: Bug 1260415 - tools: schedule partner repacks r=jlund

https://reviewboard.mozilla.org/r/44267/#review40935

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 8738028 [details]
MozReview Request: Bug 1260415 - tools: schedule partner repacks r=jlund

https://hg.mozilla.org/build/tools/rev/7e7af8aa260f

Comment 18

[Rail Aliiev [:rail]]

Comment on attachment 8738034 [details] [review]
PR for releasetasks

merged

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43219/diff/1-2/

Comment 20

[Jordan Lund (:jlund)]

Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

https://reviewboard.mozilla.org/r/43219/#review41763

hehe

Comment 21

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7b2a4edb55d9

Comment 22

[Rail Aliiev [:rail]]

Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

https://hg.mozilla.org/integration/mozilla-inbound/rev/7b2a4edb55d9
https://hg.mozilla.org/releases/mozilla-aurora/rev/274e8ba1cf09
https://hg.mozilla.org/releases/mozilla-beta/rev/4e4f75749488

Comment 23

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/7b2a4edb55d9

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

FYI, bug 1253926 asks for partner repacks to generate .checksums files for EME builds, so that they can be included in the SHA512SUMS file.

Comment 26

[Rail Aliiev [:rail]]

Windows repacks failed:

c:/mozilla-build/python27/python -u scripts/scripts/desktop_partner_repacks.py --cfg partner_repacks/release_mozilla-release_desktop.py --s3cfg /builds/partners-s3cfg --platform win64 --hgrepo releases/mozilla-beta

Need to explicitly set pf to macosx64 instead of platform.

Comment 27

[Rail Aliiev [:rail]]

Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43233/diff/1-2/

Comment 28

[Rail Aliiev [:rail]]

Attachment: set pf to macosx64

in case the reviewboard link doesn't work

Comment 29

[Jordan Lund (:jlund)]

https://reviewboard.mozilla.org/r/43233/#review42153

hmm, mozreview continues to throw me learning curveballs

Comment 30

[Rail Aliiev [:rail]]

Comment on attachment 8740180 [details] [diff] [review]
set pf to macosx64

https://hg.mozilla.org/build/buildbotcustom/rev/02f5c29c5b11

Comment 31

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/02f5c29c5b11

Comment 32

[Rail Aliiev [:rail]]

Windows got this:
.....
  PROPERTIES_FILE=/c/builds/moz2_slave/rel-m-beta-fx-w32_pner_repacks/buildprops.json
.....
 using PTY: False
11:38:36     INFO - MultiFileLogger online at 20160415 11:38:36 in /builds/slave/rel-m-beta-fx-w32_pner_repacks
Traceback (most recent call last):
  File "scripts/scripts/desktop_partner_repacks.py", line 201, in <module>
    partner_repacks = DesktopPartnerRepacks()
  File "scripts/scripts/desktop_partner_repacks.py", line 98, in __init__
    **buildscript_kwargs
  File "/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/mozharness/base/script.py", line 1563, in __init__
    self._pre_config_lock(rw_config)
  File "scripts/scripts/desktop_partner_repacks.py", line 107, in _pre_config_lock
    self.read_buildbot_config()
  File "/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/mozharness/mozilla/buildbot.py", line 64, in read_buildbot_config
    self.buildbot_config = parse_config_file(c['buildbot_json_path'])
  File "/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/mozharness/base/config.py", line 155, in parse_config_file
    raise IOError("Can't find %s in %s!" % (file_name, search_path))
IOError: Can't find /c/builds/moz2_slave/rel-m-beta-fx-w32_pner_repacks/buildprops.json in ['.', '/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/scripts/../configs', '/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/scripts/../../configs']!
program finished with exit code 1
elapsedTime=0.759153

Comment 33

[Rail Aliiev [:rail]]

I suspect https://dxr.mozilla.org/build-central/source/buildbotcustom/process/factory.py#4417 is too clever now and tries to guess the platform (mac in our case) based on target platform (windows per partner repacks platform)

Comment 34

[Rail Aliiev [:rail]]

Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43219/diff/2-3/

Comment 35

[Chris Cooper [:coop] (he/him)]

https://reviewboard.mozilla.org/r/43219/#review43425

Comment 36

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/79778dc48334

Comment 37

[Rail Aliiev [:rail]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/79778dc48334
https://hg.mozilla.org/releases/mozilla-aurora/rev/f5cbb01a27e5
https://hg.mozilla.org/releases/mozilla-beta/rev/6c4646c7a6d6

Comment 38

[Rail Aliiev [:rail]]

The push above fixed the props issue, but apparently the script uses tags to pull some stuff:

13:30:55     INFO -  2016-04-15 13:30:55,633 - INFO - retry: Calling <bound method StrictFancyURLopener.retrieve of <__main__.StrictFancyURLopener instance at 0x100778170>> with args: (), kwargs: {'url': 'https://hg.mozilla.org/releases/mozilla-beta/raw-file/FIREFOX_46_0b11_RELEASE/browser/installer/windows/app.tag', 'filename': 'app.tag'}, attempt #2

In release promotion we don't tag until we publish the release.

Comment 39

[Rail Aliiev [:rail]]

Probably we can pass --tag $revision to partner-repack.py to override the tag.

Comment 40

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/79778dc48334

Comment 41

[Rail Aliiev [:rail]]

Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43219/diff/3-4/

Comment 42

[Rail Aliiev [:rail]]

I ran parner-repacks.py ... --tag $revision on my laptop and it looks working:

2016-04-18 09:14:05,150 - INFO - Creating 4 Repack Workers
2016-04-18 09:14:05,151 - INFO - Starting repack process for partner: unitedinternet
2016-04-18 09:14:05,152 - INFO - Downloading from http://archive.mozilla.org/pub/firefox/candidates/46.0b11-candidates/build1/win32/de/Firefox%20Setup%2046.0b11.exe
2016-04-18 09:14:05,152 - INFO - To: Firefox Setup 46.0b11.exe
2016-04-18 09:14:05,153 - INFO - CWD: /home/rail/tmp/pr/scripts/original_builds/46.0b11/build1/win32/de
2016-04-18 09:14:28,163 - INFO - Repacking signed builds in /home/rail/tmp/pr/scripts/original_builds/46.0b11/build1/win32/de
2016-04-18 09:14:28,163 - INFO - Downloading from https://hg.mozilla.org/releases/mozilla-beta/raw-file/076bf6a0ac85ec6a4f3ee7c3efe653964d92b9f2/browser/installer/windows/app.tag
2016-04-18 09:14:28,163 - INFO - To: app.tag
2016-04-18 09:14:28,163 - INFO - CWD: /home/rail/tmp/pr/scripts
2016-04-18 09:14:29,198 - INFO - Downloading from https://hg.mozilla.org/releases/mozilla-beta/raw-file/076bf6a0ac85ec6a4f3ee7c3efe653964d92b9f2/other-licenses/7zstub/firefox/7zSD.sfx
2016-04-18 09:14:29,198 - INFO - To: 7zSD.sfx
2016-04-18 09:14:29,198 - INFO - CWD: /home/rail/tmp/pr/scripts
2016-04-18 09:14:30,586 - DEBUG - Executing upx --best -o "7zSD.sfx.compressed" "7zSD.sfx"

Comment 43

[Chris Cooper [:coop] (he/him)]

https://reviewboard.mozilla.org/r/43219/#review43705

Comment 44

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8cadd52cac62

Comment 45

[Rail Aliiev [:rail]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8cadd52cac62
https://hg.mozilla.org/releases/mozilla-aurora/rev/3e665a8f5cc0
https://hg.mozilla.org/releases/mozilla-release/rev/397fc30b2779

Comment 46

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/8cadd52cac62

Comment 47

[Rail Aliiev [:rail]]

Not it is failing with

01:30:25     INFO -  IOError: [Errno 2] No such file or directory: '/c/builds/moz2_slave/rel-m-rel-fx-w64_pner_repacks-/token'

Windows path won't die!

Comment 48

[Rail Aliiev [:rail]]

Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43233/diff/2-3/

Comment 49

[Rail Aliiev [:rail]]

Attachment: basedir.diff

Comment 50

[Rail Aliiev [:rail]]

Comment on attachment 8742764 [details] [diff] [review]
basedir.diff

https://hg.mozilla.org/build/buildbotcustom/rev/5557a28843c5

Comment 51

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/5557a28843c5

Comment 52

[Rail Aliiev [:rail]]

Grr, now it's:

06:08:22     INFO -  IOError: [Errno 2] No such file or directory: 'rel-m-rel-fx-w64_pner_repacks-/token'

Comment 53

[Rail Aliiev [:rail]]

Attachment: slave_platform.diff

This should just work (TM)

Comment 54

[Rail Aliiev [:rail]]

Comment on attachment 8742815 [details] [diff] [review]
slave_platform.diff

https://hg.mozilla.org/build/buildbotcustom/rev/901cbfd439c0

Comment 55

[Rail Aliiev [:rail]]

It worked this time! \o/

Comment 56

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.