Closed Bug 799654 Opened 10 years ago Closed 10 years ago

symbol upload should retry upload upon name resolution failure

Categories

(Release Engineering :: Release Automation: Other, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kmoir, Assigned: kmoir)

Details

Attachments

(1 file)

xulrunner_linux64_build failed with ssh: symbols1.dmz.phx1.mozilla.com: Temporary failure in name resolution

The build should retry when this error occurs
Summary: symbol upload should retry → symbol upload should retry upon name resolution failure
(In reply to Kim Moir [:kmoir] from comment #0)
> xulrunner_linux64_build failed with ssh: symbols1.dmz.phx1.mozilla.com:
> Temporary failure in name resolution
> 
> The build should retry when this error occurs

Just to be clear, if at all possible, we should retry the symbol upload itself, rather than waste time rebuilding the entire build.
Updated bug title to clarify, that's what I meant :-)
Summary: symbol upload should retry upon name resolution failure → symbol upload should retry upload upon name resolution failure
s/MockCommand/RetryingMockCommand/ here should do the trick: https://github.com/mozilla/buildbotcustom/blob/master/process/factory.py#L1934.
Attached patch patchSplinter Review
Assignee: nobody → kmoir
Status: NEW → ASSIGNED
Attachment #670890 - Flags: review?(bhearsum)
Comment on attachment 670890 [details] [diff] [review]
patch

Review of attachment 670890 [details] [diff] [review]:
-----------------------------------------------------------------

Did you test this in staging?
Attachment #670890 - Flags: review?(bhearsum) → review+
My dev master was in a very broken state yesterday and I had to rebuild it.  Anyways, is there a way to trigger a xulrunner_linux64_build from the release branch to run in staging, I tried several sendchanges without success.
You can just use "force build" on the builder.
I tried that earlier and it failed for various reasons, I just played with the properties passed but now it fails with download token exception 
http://dev-master01.build.scl1.mozilla.com:8039/builders/release-mozilla-release-xulrunner_linux64_build/builds/3

Hmmm not sure how to overcome this...how can I skip signing?

If I don't pass any parameters at all, it fails running hg update.
Huh, I'm surprised that it fails when properties aren't set. The signing stuff is easy to fix - ping me on IRC so we can get you a copy of passwords.py with the correct signing server credentials.
Looks like the symbol upload retries now but it fails due to permission issues 
which I think would be expected since it's running from a dev master?

http://dev-master01.build.scl1.mozilla.com:8039/builders/release-mozilla-release-xulrunner_linux64_build/builds/4/steps/make_uploadsymbols/logs/stdio
I think your release configs are out of date and using the wrong keys. In any case, retry is working so as far as I'm concerned you're good to land!
Attachment #670890 - Flags: checked-in+
This is live.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.