Schedule "partner repacks" builder

RESOLVED FIXED

Status

Release Engineering
Release Automation
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: rail, Assigned: rail)

Tracking

(Blocks: 1 bug, {leave-open})

Firefox Tracking Flags

(Not tracked)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(8 attachments)

(Assignee)

Description

2 years ago
We need to schedule partner repacks as a part of release promotion.
(Assignee)

Updated

2 years ago
QA Contact: bhearsum → rail
(Assignee)

Comment 1

2 years ago
Created attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

Allow passing version and build_number via buildbot properties. This will allow
us add buildbot builders without hardcoded parameters depending on release
configs.

Review commit: https://reviewboard.mozilla.org/r/43219/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/43219/
Attachment #8736333 - Flags: review?(coop)
(Assignee)

Updated

2 years ago
Assignee: nobody → rail
(Assignee)

Comment 2

2 years ago
Created attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

Review commit: https://reviewboard.mozilla.org/r/43233/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/43233/
Attachment #8736364 - Flags: review?(jlund)
(Assignee)

Comment 3

2 years ago
Created attachment 8736365 [details]
MozReview Request: Bug 1260415 - buildboti-config: add "partner repacks" builder r=jlund

Review commit: https://reviewboard.mozilla.org/r/43237/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/43237/
Attachment #8736365 - Flags: review?(jlund)

Comment 4

2 years ago
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

https://reviewboard.mozilla.org/r/43219/#review39857
Attachment #8736333 - Flags: review?(coop) → review+

Comment 5

2 years ago
Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

https://reviewboard.mozilla.org/r/43233/#review39867

easy!

::: process/release.py:2016
(Diff revision 1)
> +            pf=branch_config["platforms"][platform],
> +            mh_cfg=mh_cfg)
> +
> +        builders.append({
> +            'name': buildername,
> +            'slavenames': branch_config['platforms']['macosx64']['slaves'],

my memory is bad. these partner builds use osx slaves every time regardless of platform it is based on?
Attachment #8736364 - Flags: review?(jlund) → review+

Comment 6

2 years ago
Comment on attachment 8736365 [details]
MozReview Request: Bug 1260415 - buildboti-config: add "partner repacks" builder r=jlund

https://reviewboard.mozilla.org/r/43237/#review39865
Attachment #8736365 - Flags: review?(jlund) → review+

Updated

2 years ago
Depends on: 1260826
(Assignee)

Comment 7

2 years ago
(In reply to Jordan Lund (:jlund) from comment #5)
> my memory is bad. these partner builds use osx slaves every time regardless
> of platform it is based on?

Correct, http://hg.mozilla.org/build/buildbotcustom/file/d50064b10132/process/release.py#l915
(Assignee)

Comment 8

2 years ago
Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

https://hg.mozilla.org/build/buildbotcustom/rev/430fd7c357f7
Attachment #8736364 - Flags: checked-in+
(Assignee)

Comment 9

2 years ago
Comment on attachment 8736365 [details]
MozReview Request: Bug 1260415 - buildboti-config: add "partner repacks" builder r=jlund

https://hg.mozilla.org/build/buildbot-configs/rev/3e0f1271d6b1
Attachment #8736365 - Flags: checked-in+
(Assignee)

Updated

2 years ago
Keywords: leave-open
(In reply to Jordan Lund (:jlund) from comment #5)
> my memory is bad. these partner builds use osx slaves every time regardless
> of platform it is based on?

Correct. Mac is the only platform that can do all of the required packaging.

Comment 11

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/9e20e943e58b
(Assignee)

Comment 12

2 years ago
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

https://hg.mozilla.org/integration/mozilla-inbound/rev/9e20e943e58b
https://hg.mozilla.org/releases/mozilla-aurora/rev/d4b132f5231d
https://hg.mozilla.org/releases/mozilla-beta/rev/e9c1a80d7c2e
Attachment #8736333 - Flags: checked-in+

Comment 13

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/9e20e943e58b
(Assignee)

Comment 14

2 years ago
Created attachment 8738028 [details]
MozReview Request: Bug 1260415 - tools: schedule partner repacks r=jlund

Review commit: https://reviewboard.mozilla.org/r/44267/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/44267/
Attachment #8738028 - Flags: review?(jlund)
(Assignee)

Comment 15

2 years ago
Created attachment 8738034 [details] [review]
PR for releasetasks
Attachment #8738034 - Flags: review?(jlund)
Comment on attachment 8738028 [details]
MozReview Request: Bug 1260415 - tools: schedule partner repacks r=jlund

https://reviewboard.mozilla.org/r/44267/#review40935
Attachment #8738028 - Flags: review?(jlund) → review+

Updated

2 years ago
Attachment #8738034 - Flags: review?(jlund) → review+
(Assignee)

Comment 17

2 years ago
Comment on attachment 8738028 [details]
MozReview Request: Bug 1260415 - tools: schedule partner repacks r=jlund

https://hg.mozilla.org/build/tools/rev/7e7af8aa260f
Attachment #8738028 - Flags: checked-in+
(Assignee)

Comment 18

2 years ago
Comment on attachment 8738034 [details] [review]
PR for releasetasks

merged
Attachment #8738034 - Flags: checked-in+
(Assignee)

Comment 19

2 years ago
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43219/diff/1-2/
Attachment #8736333 - Attachment description: MozReview Request: Bug 1260415 - Schedule "partner repacks" builder r=coop a=release DONTBUILD → MozReview Request: Bug 1260415 - Schedule "partner repacks" builder r=jlund a=release DONTBUILD
Attachment #8736333 - Flags: review?(jlund)
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

https://reviewboard.mozilla.org/r/43219/#review41763

hehe
Attachment #8736333 - Flags: review?(jlund) → review+

Comment 21

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/7b2a4edb55d9
(Assignee)

Comment 22

2 years ago
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

https://hg.mozilla.org/integration/mozilla-inbound/rev/7b2a4edb55d9
https://hg.mozilla.org/releases/mozilla-aurora/rev/274e8ba1cf09
https://hg.mozilla.org/releases/mozilla-beta/rev/4e4f75749488

Comment 23

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/7b2a4edb55d9
Duplicate of this bug: 1263435
FYI, bug 1253926 asks for partner repacks to generate .checksums files for EME builds, so that they can be included in the SHA512SUMS file.
(Assignee)

Comment 26

2 years ago
Windows repacks failed:

c:/mozilla-build/python27/python -u scripts/scripts/desktop_partner_repacks.py --cfg partner_repacks/release_mozilla-release_desktop.py --s3cfg /builds/partners-s3cfg --platform win64 --hgrepo releases/mozilla-beta

Need to explicitly set pf to macosx64 instead of platform.
(Assignee)

Comment 27

2 years ago
Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43233/diff/1-2/
Attachment #8736364 - Attachment description: MozReview Request: Bug 1260415 - buildbotcustom: add "partner repacks" builder r=jlund → MozReview Request: Bug 1260415 - Schedule "partner repacks" builder r=jlund
(Assignee)

Comment 28

2 years ago
Created attachment 8740180 [details] [diff] [review]
set pf to macosx64

in case the reviewboard link doesn't work
Attachment #8740180 - Flags: review?(jlund)
https://reviewboard.mozilla.org/r/43233/#review42153

hmm, mozreview continues to throw me learning curveballs

Updated

2 years ago
Attachment #8740180 - Flags: review?(jlund) → review+
(Assignee)

Comment 30

2 years ago
Comment on attachment 8740180 [details] [diff] [review]
set pf to macosx64

https://hg.mozilla.org/build/buildbotcustom/rev/02f5c29c5b11
Attachment #8740180 - Flags: checked-in+
In production: https://hg.mozilla.org/build/buildbotcustom/rev/02f5c29c5b11
(Assignee)

Comment 32

2 years ago
Windows got this:
.....
  PROPERTIES_FILE=/c/builds/moz2_slave/rel-m-beta-fx-w32_pner_repacks/buildprops.json
.....
 using PTY: False
11:38:36     INFO - MultiFileLogger online at 20160415 11:38:36 in /builds/slave/rel-m-beta-fx-w32_pner_repacks
Traceback (most recent call last):
  File "scripts/scripts/desktop_partner_repacks.py", line 201, in <module>
    partner_repacks = DesktopPartnerRepacks()
  File "scripts/scripts/desktop_partner_repacks.py", line 98, in __init__
    **buildscript_kwargs
  File "/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/mozharness/base/script.py", line 1563, in __init__
    self._pre_config_lock(rw_config)
  File "scripts/scripts/desktop_partner_repacks.py", line 107, in _pre_config_lock
    self.read_buildbot_config()
  File "/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/mozharness/mozilla/buildbot.py", line 64, in read_buildbot_config
    self.buildbot_config = parse_config_file(c['buildbot_json_path'])
  File "/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/mozharness/base/config.py", line 155, in parse_config_file
    raise IOError("Can't find %s in %s!" % (file_name, search_path))
IOError: Can't find /c/builds/moz2_slave/rel-m-beta-fx-w32_pner_repacks/buildprops.json in ['.', '/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/scripts/../configs', '/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/scripts/../../configs']!
program finished with exit code 1
elapsedTime=0.759153
(Assignee)

Comment 33

2 years ago
I suspect https://dxr.mozilla.org/build-central/source/buildbotcustom/process/factory.py#4417 is too clever now and tries to guess the platform (mac in our case) based on target platform (windows per partner repacks platform)
(Assignee)

Comment 34

2 years ago
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43219/diff/2-3/
Attachment #8736333 - Attachment description: MozReview Request: Bug 1260415 - Schedule "partner repacks" builder r=jlund a=release DONTBUILD → MozReview Request: Bug 1260415 - Use platform independent buildprops.json r=coop DONTBUILD a=release
https://reviewboard.mozilla.org/r/43219/#review43425

Comment 36

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/79778dc48334
(Assignee)

Comment 37

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/79778dc48334
https://hg.mozilla.org/releases/mozilla-aurora/rev/f5cbb01a27e5
https://hg.mozilla.org/releases/mozilla-beta/rev/6c4646c7a6d6
(Assignee)

Comment 38

2 years ago
The push above fixed the props issue, but apparently the script uses tags to pull some stuff:

13:30:55     INFO -  2016-04-15 13:30:55,633 - INFO - retry: Calling <bound method StrictFancyURLopener.retrieve of <__main__.StrictFancyURLopener instance at 0x100778170>> with args: (), kwargs: {'url': 'https://hg.mozilla.org/releases/mozilla-beta/raw-file/FIREFOX_46_0b11_RELEASE/browser/installer/windows/app.tag', 'filename': 'app.tag'}, attempt #2

In release promotion we don't tag until we publish the release.
(Assignee)

Comment 39

2 years ago
Probably we can pass --tag $revision to partner-repack.py to override the tag.

Comment 40

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/79778dc48334
(Assignee)

Comment 41

2 years ago
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43219/diff/3-4/
Attachment #8736333 - Attachment description: MozReview Request: Bug 1260415 - Use platform independent buildprops.json r=coop DONTBUILD a=release → MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD
(Assignee)

Comment 42

2 years ago
I ran parner-repacks.py ... --tag $revision on my laptop and it looks working:

2016-04-18 09:14:05,150 - INFO - Creating 4 Repack Workers
2016-04-18 09:14:05,151 - INFO - Starting repack process for partner: unitedinternet
2016-04-18 09:14:05,152 - INFO - Downloading from http://archive.mozilla.org/pub/firefox/candidates/46.0b11-candidates/build1/win32/de/Firefox%20Setup%2046.0b11.exe
2016-04-18 09:14:05,152 - INFO - To: Firefox Setup 46.0b11.exe
2016-04-18 09:14:05,153 - INFO - CWD: /home/rail/tmp/pr/scripts/original_builds/46.0b11/build1/win32/de
2016-04-18 09:14:28,163 - INFO - Repacking signed builds in /home/rail/tmp/pr/scripts/original_builds/46.0b11/build1/win32/de
2016-04-18 09:14:28,163 - INFO - Downloading from https://hg.mozilla.org/releases/mozilla-beta/raw-file/076bf6a0ac85ec6a4f3ee7c3efe653964d92b9f2/browser/installer/windows/app.tag
2016-04-18 09:14:28,163 - INFO - To: app.tag
2016-04-18 09:14:28,163 - INFO - CWD: /home/rail/tmp/pr/scripts
2016-04-18 09:14:29,198 - INFO - Downloading from https://hg.mozilla.org/releases/mozilla-beta/raw-file/076bf6a0ac85ec6a4f3ee7c3efe653964d92b9f2/other-licenses/7zstub/firefox/7zSD.sfx
2016-04-18 09:14:29,198 - INFO - To: 7zSD.sfx
2016-04-18 09:14:29,198 - INFO - CWD: /home/rail/tmp/pr/scripts
2016-04-18 09:14:30,586 - DEBUG - Executing upx --best -o "7zSD.sfx.compressed" "7zSD.sfx"
https://reviewboard.mozilla.org/r/43219/#review43705

Comment 44

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/8cadd52cac62
(Assignee)

Comment 45

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/8cadd52cac62
https://hg.mozilla.org/releases/mozilla-aurora/rev/3e665a8f5cc0
https://hg.mozilla.org/releases/mozilla-release/rev/397fc30b2779

Comment 46

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/8cadd52cac62
(Assignee)

Comment 47

2 years ago
Not it is failing with

01:30:25     INFO -  IOError: [Errno 2] No such file or directory: '/c/builds/moz2_slave/rel-m-rel-fx-w64_pner_repacks-/token'

Windows path won't die!
(Assignee)

Comment 48

2 years ago
Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43233/diff/2-3/
Attachment #8736364 - Attachment description: MozReview Request: Bug 1260415 - Schedule "partner repacks" builder r=jlund → MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee
Attachment #8736364 - Flags: review?(catlee)
(Assignee)

Updated

2 years ago
Attachment #8736364 - Flags: review?(catlee)
(Assignee)

Comment 49

2 years ago
Created attachment 8742764 [details] [diff] [review]
basedir.diff
Attachment #8742764 - Flags: review?(catlee)

Updated

2 years ago
Attachment #8742764 - Flags: review?(catlee) → review+
(Assignee)

Comment 50

2 years ago
Comment on attachment 8742764 [details] [diff] [review]
basedir.diff

https://hg.mozilla.org/build/buildbotcustom/rev/5557a28843c5
Attachment #8742764 - Flags: checked-in+
In production: https://hg.mozilla.org/build/buildbotcustom/rev/5557a28843c5
(Assignee)

Comment 52

2 years ago
Grr, now it's:

06:08:22     INFO -  IOError: [Errno 2] No such file or directory: 'rel-m-rel-fx-w64_pner_repacks-/token'
(Assignee)

Comment 53

2 years ago
Created attachment 8742815 [details] [diff] [review]
slave_platform.diff

This should just work (TM)
Attachment #8742815 - Flags: review?(catlee)

Updated

2 years ago
Attachment #8742815 - Flags: review?(catlee) → review+
(Assignee)

Comment 54

2 years ago
Comment on attachment 8742815 [details] [diff] [review]
slave_platform.diff

https://hg.mozilla.org/build/buildbotcustom/rev/901cbfd439c0
Attachment #8742815 - Flags: checked-in+
(Assignee)

Comment 55

2 years ago
It worked this time! \o/
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.