Closed Bug 1260415 Opened 8 years ago Closed 8 years ago

Schedule "partner repacks" builder

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rail, Assigned: rail)

References

Details

Attachments

(8 files)

We need to schedule partner repacks as a part of release promotion.
QA Contact: bhearsum → rail
Allow passing version and build_number via buildbot properties. This will allow
us add buildbot builders without hardcoded parameters depending on release
configs.

Review commit: https://reviewboard.mozilla.org/r/43219/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/43219/
Attachment #8736333 - Flags: review?(coop)
Assignee: nobody → rail
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

https://reviewboard.mozilla.org/r/43219/#review39857
Attachment #8736333 - Flags: review?(coop) → review+
Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

https://reviewboard.mozilla.org/r/43233/#review39867

easy!

::: process/release.py:2016
(Diff revision 1)
> +            pf=branch_config["platforms"][platform],
> +            mh_cfg=mh_cfg)
> +
> +        builders.append({
> +            'name': buildername,
> +            'slavenames': branch_config['platforms']['macosx64']['slaves'],

my memory is bad. these partner builds use osx slaves every time regardless of platform it is based on?
Attachment #8736364 - Flags: review?(jlund) → review+
Comment on attachment 8736365 [details]
MozReview Request: Bug 1260415 - buildboti-config: add "partner repacks" builder r=jlund

https://reviewboard.mozilla.org/r/43237/#review39865
Attachment #8736365 - Flags: review?(jlund) → review+
Depends on: 1260826
(In reply to Jordan Lund (:jlund) from comment #5)
> my memory is bad. these partner builds use osx slaves every time regardless
> of platform it is based on?

Correct, http://hg.mozilla.org/build/buildbotcustom/file/d50064b10132/process/release.py#l915
Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

https://hg.mozilla.org/build/buildbotcustom/rev/430fd7c357f7
Attachment #8736364 - Flags: checked-in+
Comment on attachment 8736365 [details]
MozReview Request: Bug 1260415 - buildboti-config: add "partner repacks" builder r=jlund

https://hg.mozilla.org/build/buildbot-configs/rev/3e0f1271d6b1
Attachment #8736365 - Flags: checked-in+
Keywords: leave-open
(In reply to Jordan Lund (:jlund) from comment #5)
> my memory is bad. these partner builds use osx slaves every time regardless
> of platform it is based on?

Correct. Mac is the only platform that can do all of the required packaging.
Attached file PR for releasetasks
Attachment #8738034 - Flags: review?(jlund)
Comment on attachment 8738028 [details]
MozReview Request: Bug 1260415 - tools: schedule partner repacks r=jlund

https://reviewboard.mozilla.org/r/44267/#review40935
Attachment #8738028 - Flags: review?(jlund) → review+
Attachment #8738034 - Flags: review?(jlund) → review+
Comment on attachment 8738028 [details]
MozReview Request: Bug 1260415 - tools: schedule partner repacks r=jlund

https://hg.mozilla.org/build/tools/rev/7e7af8aa260f
Attachment #8738028 - Flags: checked-in+
Comment on attachment 8738034 [details] [review]
PR for releasetasks

merged
Attachment #8738034 - Flags: checked-in+
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43219/diff/1-2/
Attachment #8736333 - Attachment description: MozReview Request: Bug 1260415 - Schedule "partner repacks" builder r=coop a=release DONTBUILD → MozReview Request: Bug 1260415 - Schedule "partner repacks" builder r=jlund a=release DONTBUILD
Attachment #8736333 - Flags: review?(jlund)
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

https://reviewboard.mozilla.org/r/43219/#review41763

hehe
Attachment #8736333 - Flags: review?(jlund) → review+
FYI, bug 1253926 asks for partner repacks to generate .checksums files for EME builds, so that they can be included in the SHA512SUMS file.
Windows repacks failed:

c:/mozilla-build/python27/python -u scripts/scripts/desktop_partner_repacks.py --cfg partner_repacks/release_mozilla-release_desktop.py --s3cfg /builds/partners-s3cfg --platform win64 --hgrepo releases/mozilla-beta

Need to explicitly set pf to macosx64 instead of platform.
Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43233/diff/1-2/
Attachment #8736364 - Attachment description: MozReview Request: Bug 1260415 - buildbotcustom: add "partner repacks" builder r=jlund → MozReview Request: Bug 1260415 - Schedule "partner repacks" builder r=jlund
in case the reviewboard link doesn't work
Attachment #8740180 - Flags: review?(jlund)
https://reviewboard.mozilla.org/r/43233/#review42153

hmm, mozreview continues to throw me learning curveballs
Attachment #8740180 - Flags: review?(jlund) → review+
Windows got this:
.....
  PROPERTIES_FILE=/c/builds/moz2_slave/rel-m-beta-fx-w32_pner_repacks/buildprops.json
.....
 using PTY: False
11:38:36     INFO - MultiFileLogger online at 20160415 11:38:36 in /builds/slave/rel-m-beta-fx-w32_pner_repacks
Traceback (most recent call last):
  File "scripts/scripts/desktop_partner_repacks.py", line 201, in <module>
    partner_repacks = DesktopPartnerRepacks()
  File "scripts/scripts/desktop_partner_repacks.py", line 98, in __init__
    **buildscript_kwargs
  File "/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/mozharness/base/script.py", line 1563, in __init__
    self._pre_config_lock(rw_config)
  File "scripts/scripts/desktop_partner_repacks.py", line 107, in _pre_config_lock
    self.read_buildbot_config()
  File "/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/mozharness/mozilla/buildbot.py", line 64, in read_buildbot_config
    self.buildbot_config = parse_config_file(c['buildbot_json_path'])
  File "/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/mozharness/base/config.py", line 155, in parse_config_file
    raise IOError("Can't find %s in %s!" % (file_name, search_path))
IOError: Can't find /c/builds/moz2_slave/rel-m-beta-fx-w32_pner_repacks/buildprops.json in ['.', '/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/scripts/../configs', '/builds/slave/rel-m-beta-fx-w32_pner_repacks/scripts/scripts/../../configs']!
program finished with exit code 1
elapsedTime=0.759153
I suspect https://dxr.mozilla.org/build-central/source/buildbotcustom/process/factory.py#4417 is too clever now and tries to guess the platform (mac in our case) based on target platform (windows per partner repacks platform)
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43219/diff/2-3/
Attachment #8736333 - Attachment description: MozReview Request: Bug 1260415 - Schedule "partner repacks" builder r=jlund a=release DONTBUILD → MozReview Request: Bug 1260415 - Use platform independent buildprops.json r=coop DONTBUILD a=release
The push above fixed the props issue, but apparently the script uses tags to pull some stuff:

13:30:55     INFO -  2016-04-15 13:30:55,633 - INFO - retry: Calling <bound method StrictFancyURLopener.retrieve of <__main__.StrictFancyURLopener instance at 0x100778170>> with args: (), kwargs: {'url': 'https://hg.mozilla.org/releases/mozilla-beta/raw-file/FIREFOX_46_0b11_RELEASE/browser/installer/windows/app.tag', 'filename': 'app.tag'}, attempt #2

In release promotion we don't tag until we publish the release.
Probably we can pass --tag $revision to partner-repack.py to override the tag.
Comment on attachment 8736333 [details]
MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43219/diff/3-4/
Attachment #8736333 - Attachment description: MozReview Request: Bug 1260415 - Use platform independent buildprops.json r=coop DONTBUILD a=release → MozReview Request: Bug 1260415 - Use revision as tag r=coop a=release DONTBUILD
I ran parner-repacks.py ... --tag $revision on my laptop and it looks working:

2016-04-18 09:14:05,150 - INFO - Creating 4 Repack Workers
2016-04-18 09:14:05,151 - INFO - Starting repack process for partner: unitedinternet
2016-04-18 09:14:05,152 - INFO - Downloading from http://archive.mozilla.org/pub/firefox/candidates/46.0b11-candidates/build1/win32/de/Firefox%20Setup%2046.0b11.exe
2016-04-18 09:14:05,152 - INFO - To: Firefox Setup 46.0b11.exe
2016-04-18 09:14:05,153 - INFO - CWD: /home/rail/tmp/pr/scripts/original_builds/46.0b11/build1/win32/de
2016-04-18 09:14:28,163 - INFO - Repacking signed builds in /home/rail/tmp/pr/scripts/original_builds/46.0b11/build1/win32/de
2016-04-18 09:14:28,163 - INFO - Downloading from https://hg.mozilla.org/releases/mozilla-beta/raw-file/076bf6a0ac85ec6a4f3ee7c3efe653964d92b9f2/browser/installer/windows/app.tag
2016-04-18 09:14:28,163 - INFO - To: app.tag
2016-04-18 09:14:28,163 - INFO - CWD: /home/rail/tmp/pr/scripts
2016-04-18 09:14:29,198 - INFO - Downloading from https://hg.mozilla.org/releases/mozilla-beta/raw-file/076bf6a0ac85ec6a4f3ee7c3efe653964d92b9f2/other-licenses/7zstub/firefox/7zSD.sfx
2016-04-18 09:14:29,198 - INFO - To: 7zSD.sfx
2016-04-18 09:14:29,198 - INFO - CWD: /home/rail/tmp/pr/scripts
2016-04-18 09:14:30,586 - DEBUG - Executing upx --best -o "7zSD.sfx.compressed" "7zSD.sfx"
Not it is failing with

01:30:25     INFO -  IOError: [Errno 2] No such file or directory: '/c/builds/moz2_slave/rel-m-rel-fx-w64_pner_repacks-/token'

Windows path won't die!
Comment on attachment 8736364 [details]
MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43233/diff/2-3/
Attachment #8736364 - Attachment description: MozReview Request: Bug 1260415 - Schedule "partner repacks" builder r=jlund → MozReview Request: Bug 1260415 - Explicitly set basedir for all platforms. r=catlee
Attachment #8736364 - Flags: review?(catlee)
Attachment #8736364 - Flags: review?(catlee)
Attached patch basedir.diffSplinter Review
Attachment #8742764 - Flags: review?(catlee)
Attachment #8742764 - Flags: review?(catlee) → review+
Grr, now it's:

06:08:22     INFO -  IOError: [Errno 2] No such file or directory: 'rel-m-rel-fx-w64_pner_repacks-/token'
This should just work (TM)
Attachment #8742815 - Flags: review?(catlee)
Attachment #8742815 - Flags: review?(catlee) → review+
It worked this time! \o/
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.