885331 - partner repacks need to retry fetching from hg

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P3)

Description

[Chris AtLee [:catlee]]

The win32 partner repack for Firefox 22 failed with this error:

2013-06-18 11:50:13,452 - Warning: use of --signed is deprecated. It is now the default.
2013-06-18 11:50:13,453 - ##################################################
2013-06-18 11:50:13,453 - Starting repack process for partner: 1und1
2013-06-18 11:50:13,454 - Downloading from http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/22.0-candidates/build1/win32/de/Firefox%20Setup%2022.0.exe
2013-06-18 11:50:13,454 - To: Firefox Setup 22.0.exe
2013-06-18 11:50:13,454 - CWD: /builds/slave/rel-m-rel-w32_partner_rpk-0000/partner-repacks/scripts/original_builds/22.0/build1/win32/de
2013-06-18 11:50:14,201 - Repacking signed builds in /builds/slave/rel-m-rel-w32_partner_rpk-0000/partner-repacks/scripts/original_builds/22.0/build1/win32/de
2013-06-18 11:50:14,201 - Downloading from http://hg.mozilla.org/releases/mozilla-release/raw-file/FIREFOX_22_0_RELEASE/browser/installer/windows/app.tag
2013-06-18 11:50:14,201 - To: app.tag
2013-06-18 11:50:14,201 - CWD: /builds/slave/rel-m-rel-w32_partner_rpk-0000/partner-repacks/scripts
2013-06-18 11:50:14,237 - Error downloading http://hg.mozilla.org/releases/mozilla-release/raw-file/FIREFOX_22_0_RELEASE/browser/installer/windows/app.tag
Traceback (most recent call last):
  File "./partner-repacks.py", line 494, in retrieveFile
    StrictFancyURLopener().retrieve(url, file_path)
  File "/tools/python-2.7.3/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 239, in retrieve
    fp = self.open(url, data)
  File "/tools/python-2.7.3/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 207, in open
    return getattr(self, name)(url)
  File "/tools/python-2.7.3/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 358, in open_http
    return self.http_error(url, fp, errcode, errmsg, headers)
  File "/tools/python-2.7.3/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 375, in http_error
    return self.http_error_default(url, fp, errcode, errmsg, headers)
  File "./partner-repacks.py", line 48, in http_error_default
    headers)
  File "/tools/python-2.7.3/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 380, in http_error_default
    raise IOError, ('http error', errcode, errmsg, headers)
IOError: ('http error', 500, 'Internal Server Error', <httplib.HTTPMessage instance at 0x10129ecf8>)
2013-06-18 11:50:14,313 - Cannot remove app.tag
Traceback (most recent call last):
  File "./partner-repacks.py", line 499, in retrieveFile
    os.remove(file_path)
OSError: [Errno 2] No such file or directory: 'app.tag'
2013-06-18 11:50:14,313 - Error: Unable to retrieve app.tag


We should retry a few times before giving up.

Comment 1

[Massimo Gervasini [:massimo]]

Attachment: [partner-repacks] Bug 885331 - partner repacks need to retry fetching from hg.patch

Adds a retries for retrieveFile()

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

partner repack jobs have the tools repo available, so they can use lib/python/util/retry.py. Did you avoid that for some reason ?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8498185 [details] [diff] [review]
[partner-repacks] Bug 885331 - partner repacks need to retry fetching from hg.patch

Clearing review pending response to comment #2.

Comment 4

[Massimo Gervasini [:massimo]]

Attachment: [partner-repacks] Bug 885331 - partner repacks need to retry fetching from hg.patch

Adding retries using lib/python/util/retry.py (and pep8 fixes)

Comment 5

[Massimo Gervasini [:massimo]]

Attachment: [buildbotcustom] Bug 885331 - added PYTHONPATH in doPartnerRepacks.patch

Adding PYTHONPATH to doPartnerRepacks() environment

Comment 6

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

Checked in code landed in production

Comment 7

[Rail Aliiev [:rail]]

This broke https://intranet.mozilla.org/Build:Partners:Repacks:Creation :/

Comment 8

[Rail Aliiev [:rail]]

Traceback (most recent call last):
  File "./partner-repacks.py", line 10, in <module>
    from util.retry import retry
ImportError: No module named util.retry

Comment 9

[Rail Aliiev [:rail]]

Updated the docs to use PYTHONPATH
https://intranet.mozilla.org/index.php?title=Build%2FPartners%2FRepacks%2FCreation&action=historysubmit&diff=177705&oldid=177163