Closed
Bug 1216907
Opened 9 years ago
Closed 9 years ago
Uploads are broken on try for desktop and mobile builds since S3 migration
Categories
(Release Engineering :: Applications: MozharnessCore, defect)
Release Engineering
Applications: MozharnessCore
Tracking
(firefox44 fixed)
RESOLVED
FIXED
Tracking | Status | |
---|---|---|
firefox44 | --- | fixed |
People
(Reporter: nthomas, Unassigned)
References
Details
Attachments
(1 file, 1 obsolete file)
7.43 KB,
patch
|
nthomas
:
checked-in+
|
Details | Diff | Splinter Review |
Not all jobs show this, because some people are based on older code, but this is a fair example:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=dc9691794015
So busted buildbot builds for firefox and fennec, hit this when starting 'make upload':
subprocess.CalledProcessError: Command '['ssh', '-o', 'IdentityFile=/home/mock_mozilla/.ssh/trybld_dsa', 'trybld@upload.ffxbld.productdelivery.prod.mozaws.net', 'mktemp -d']' returned non-zero exit status 255
Reporter | ||
Comment 1•9 years ago
|
||
The problem is that the upload host should be upload.trybld.productdelivery.prod.mozaws.net (instead of upload.ffxbld.pr...). This is specific to try with its level 1 access, and all the other branches use upload.ffxbld.
Attached is a work-around for developers to use until we fix this. It should not land in m-c or friends!
Reporter | ||
Comment 2•9 years ago
|
||
I has hoping I could fix this with a patch like so:
diff --git a/testing/mozharness/configs/builds/branch_specifics.py b/testing/mozharness/configs/builds/branch_specifics.py
--- a/testing/mozharness/configs/builds/branch_specifics.py
+++ b/testing/mozharness/configs/builds/branch_specifics.py
@@ -66,20 +66,21 @@ config = {
},
'try': {
'repo_path': 'try',
'clone_by_revision': True,
'clone_with_purge': True,
'tinderbox_build_dir': '%(who)s-%(got_revision)s',
'to_tinderbox_dated': False,
'include_post_upload_builddir': True,
'release_to_try_builds': True,
'use_branch_in_symbols_extra_buildid': False,
+ 'stage_server': 'upload.trybld.productdelivery.prod.mozaws.net',
'stage_username': 'trybld',
'stage_ssh_key': 'trybld_dsa',
'branch_supports_uploadsymbols': False,
...
However the precedence order at https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/building/buildbase.py#225 means the production config will override the special branch value, so we still get upload.fffbld. Making branch trump prod/staging would fix the issue here, but potentially bust other parameters which relying on the current order.
Reporter | ||
Comment 4•9 years ago
|
||
Here's a hack to fix this, landed as bustage fix at
https://hg.mozilla.org/integration/mozilla-inbound/rev/0ee21e8d5ca6
Devs push your changes on top of this (or a descendent, obvs).
Attachment #8676719 -
Flags: checked-in+
Reporter | ||
Comment 5•9 years ago
|
||
Jordan, how could we do this better ?
Severity: critical → normal
Flags: needinfo?(jlund)
Reporter | ||
Comment 6•9 years ago
|
||
Try push to verify this actually works - https://treeherder.mozilla.org/#/jobs?repo=try&revision=c7073e6ceb85 - and stage_server is set properly. Also checked inbound push is not changed.
Reporter | ||
Updated•9 years ago
|
Attachment #8676719 -
Attachment is obsolete: true
Reporter | ||
Updated•9 years ago
|
Attachment #8676719 -
Attachment is obsolete: false
Reporter | ||
Updated•9 years ago
|
Attachment #8676707 -
Attachment is obsolete: true
I'm hitting conflicts trying to get this onto anything beta or older.
Flags: needinfo?(nthomas)
Comment 10•9 years ago
|
||
(In reply to Nick Thomas [:nthomas] from comment #5)
> Jordan, how could we do this better ?
hrm, how about a 'try' pool in build_pool_specifics that is a dupe of 'production' but has its own stage_server.
benefits
1) wouldn't require change in hierarchy configs
2) wouldn't require 'stage_server' explicitly set in each branch
downsides:
1) we would also have to patch all the script calls for our builds in Buildbot and Taskcluster for try specifically.
Flags: needinfo?(jlund)
Reporter | ||
Comment 12•9 years ago
|
||
(In reply to Wes Kocher (:KWierso) from comment #9)
> I'm hitting conflicts trying to get this onto anything beta or older.
I actually need to merge attachment 8672532 [details] [diff] [review] (bug 1213721) and the patch here to beta/release/esr38. It's not super urgent because we're not using mozharness for the builds there, just in case someone pushes to try.
Flags: needinfo?(nthomas)
Reporter | ||
Comment 13•9 years ago
|
||
(In reply to Jordan Lund (:jlund) from comment #10)
> hrm, how about a 'try' pool in build_pool_specifics that is a dupe of
> 'production' but has its own stage_server.
I like it. Maybe go a step further to allow for staging try as well, ie
staging, production, staging-try, production-try
Reporter | ||
Comment 14•9 years ago
|
||
Spun that out to bug 1217271.
Reporter | ||
Comment 15•9 years ago
|
||
Comment on attachment 8676719 [details] [diff] [review]
[gecko] Hack to fix try
Merged to release branches:
https://hg.mozilla.org/releases/mozilla-beta/rev/569a9bdffe0a
https://hg.mozilla.org/releases/mozilla-release/rev/473d17bfdd8e
http://hg.mozilla.org/releases/mozilla-esr38/rev/a942a18f103a (default)
http://hg.mozilla.org/releases/mozilla-esr38/rev/9a453c7474a2 (tb verbranch)
Comment hidden (typo) |
You need to log in
before you can comment on or make changes to this bug.
Description
•