799654 - symbol upload should retry upload upon name resolution failure

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Kim Moir [:kmoir] ET]

xulrunner_linux64_build failed with ssh: symbols1.dmz.phx1.mozilla.com: Temporary failure in name resolution

The build should retry when this error occurs

Comment 1

[Justin Wood (:Callek)]

(In reply to Kim Moir [:kmoir] from comment #0)
> xulrunner_linux64_build failed with ssh: symbols1.dmz.phx1.mozilla.com:
> Temporary failure in name resolution
> 
> The build should retry when this error occurs

Just to be clear, if at all possible, we should retry the symbol upload itself, rather than waste time rebuilding the entire build.

Comment 2

[Kim Moir [:kmoir] ET]

Updated bug title to clarify, that's what I meant :-)

Comment 3

[[e-mail] (:bhearsum)]

s/MockCommand/RetryingMockCommand/ here should do the trick: https://github.com/mozilla/buildbotcustom/blob/master/process/factory.py#L1934.

Comment 4

[Kim Moir [:kmoir] ET]

Attachment: patch

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 670890 [details] [diff] [review]
patch

Review of attachment 670890 [details] [diff] [review]:
-----------------------------------------------------------------

Did you test this in staging?

Comment 6

[Kim Moir [:kmoir] ET]

My dev master was in a very broken state yesterday and I had to rebuild it.  Anyways, is there a way to trigger a xulrunner_linux64_build from the release branch to run in staging, I tried several sendchanges without success.

Comment 7

[[e-mail] (:bhearsum)]

You can just use "force build" on the builder.

Comment 8

[Kim Moir [:kmoir] ET]

I tried that earlier and it failed for various reasons, I just played with the properties passed but now it fails with download token exception 
http://dev-master01.build.scl1.mozilla.com:8039/builders/release-mozilla-release-xulrunner_linux64_build/builds/3

Hmmm not sure how to overcome this...how can I skip signing?

If I don't pass any parameters at all, it fails running hg update.

Comment 9

[[e-mail] (:bhearsum)]

Huh, I'm surprised that it fails when properties aren't set. The signing stuff is easy to fix - ping me on IRC so we can get you a copy of passwords.py with the correct signing server credentials.

Comment 10

[Kim Moir [:kmoir] ET]

Looks like the symbol upload retries now but it fails due to permission issues 
which I think would be expected since it's running from a dev master?

http://dev-master01.build.scl1.mozilla.com:8039/builders/release-mozilla-release-xulrunner_linux64_build/builds/4/steps/make_uploadsymbols/logs/stdio

Comment 11

[[e-mail] (:bhearsum)]

I think your release configs are out of date and using the wrong keys. In any case, retry is working so as far as I'm concerned you're good to land!

Comment 12

[Aki Sasaki (not active)]

This is live.