Closed Bug 1303668 Opened 9 years ago Closed 9 years ago

Intermittent AttributeError: 'NoneType' object has no attribute 'find' when fallback for not finding symbols URL doesn't work

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

Product:
Release Engineering
Component:
Applications: MozharnessCore
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox50 wontfix, firefox51 fixed, firefox52 fixed, firefox53 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox50 --- wontfix
firefox51 --- fixed
firefox52 --- fixed
firefox53 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: gbrown)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files, 1 obsolete file)

Bug 1303668 - Retry downloading symbols if we're expecting them.
58 bytes, text/x-review-board-request
mozilla
: review+
Details
fix retry of query_symbols_url
2.35 KB, patch
mozilla
: review+
Details | Diff | Splinter Review
Filed by: philringnalda [at] gmail.com https://treeherder.mozilla.org/logviewer.html#?job_id=3671062&repo=autoland https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-linux-debug/1474251137/autoland_ubuntu32_vm-debug_test-mochitest-4-bm08-tests1-linux32-build68.txt.gz 21:34:06 WARNING - Can't figure out symbols_url from installer_url: https://queue.taskcluster.net/v1/task/c5zgKXI2QQmKduigjtiIcg/artifacts/public/build/firefox-51.0a1.en-US.linux-i686.tar.bz2! (traceback from a connection reset while querying the symbols_url) WARNING - No symbols_url found. Let minidump_stackwalk query for symbols. 21:34:06 INFO - Setting buildbot property symbols_url to None 21:34:06 INFO - Writing buildbot properties ['symbols_url'] to /builds/slave/test/properties/symbols_url 21:34:06 INFO - Writing to file /builds/slave/test/properties/symbols_url 21:34:06 INFO - Contents: 21:34:06 INFO - symbols_url:None 21:34:06 INFO - Downloading and extracting to /builds/slave/test/build/symbols these dirs * from None 21:34:06 INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': None}, attempt #1 21:34:06 INFO - Fetch None into memory 21:34:06 INFO - Running post-action listener: _resource_record_post_action 21:34:06 INFO - Running post-action listener: set_extra_try_arguments 21:34:06 INFO - [mozharness: 2016-09-19 04:34:06.136733Z] Finished download-and-extract step (failed) 21:34:06 FATAL - Uncaught exception: Traceback (most recent call last): 21:34:06 FATAL - File "/builds/slave/test/scripts/mozharness/base/script.py", line 2001, in run 21:34:06 FATAL - self.run_action(action) 21:34:06 FATAL - File "/builds/slave/test/scripts/mozharness/base/script.py", line 1940, in run_action 21:34:06 FATAL - self._possibly_run_method(method_name, error_if_missing=True) 21:34:06 FATAL - File "/builds/slave/test/scripts/mozharness/base/script.py", line 1880, in _possibly_run_method 21:34:06 FATAL - return getattr(self, method_name)() 21:34:06 FATAL - File "scripts/scripts/desktop_unittest.py", line 505, in download_and_extract 21:34:06 FATAL - suite_categories=target_categories) 21:34:06 FATAL - File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 557, in download_and_extract 21:34:06 FATAL - self._download_and_extract_symbols() 21:34:06 FATAL - File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 518, in _download_and_extract_symbols 21:34:06 FATAL - self.download_unpack(self.symbols_url, self.symbols_path) 21:34:06 FATAL - File "/builds/slave/test/scripts/mozharness/base/script.py", line 675, in download_unpack 21:34:06 FATAL - **retry_args 21:34:06 FATAL - File "/builds/slave/test/scripts/mozharness/base/script.py", line 1099, in retry 21:34:06 FATAL - status = action(*args, **kwargs) 21:34:06 FATAL - File "/builds/slave/test/scripts/mozharness/base/script.py", line 366, in fetch_url_into_memory 21:34:06 FATAL - parsed_url = urlparse.urlparse(url) 21:34:06 FATAL - File "/usr/lib/python2.7/urlparse.py", line 135, in urlparse 21:34:06 FATAL - tuple = urlsplit(url, scheme, allow_fragments) 21:34:06 FATAL - File "/usr/lib/python2.7/urlparse.py", line 174, in urlsplit 21:34:06 FATAL - i = url.find(':') 21:34:06 FATAL - AttributeError: 'NoneType' object has no attribute 'find' 21:34:06 FATAL - Running post_fatal callback... 21:34:06 FATAL - Exiting -1 hahahaha, "Fetch None into memory"
Depends on: 1303759
Assignee: nobody → armenzg
Attachment #8794331 - Flags: review?(jlund)
Comment on attachment 8794331 [details] Bug 1303668 - Failing to find a symbols url should be a warning instead of a exception. https://reviewboard.mozilla.org/r/80836/#review79652 ::: testing/mozharness/mozharness/mozilla/testing/testbase.py (Diff revision 1) > return self.symbols_url > > elif self.installer_url: > - symbols_url = self.query_prefixed_build_dir_url('.crashreporter-symbols.zip') > + self.symbols_url = self.query_prefixed_build_dir_url('.crashreporter-symbols.zip') > > - # Check if the URL exists. If not, use none to allow mozcrash to auto-check for symbols We should not remove that part. As the comment below states minidump-stackwalk can only auto-detect the symbols for Nightly and release builds but not for CI builds. I would suggest to maybe reduce the failure from exception to warning.
Comment on attachment 8794331 [details] Bug 1303668 - Failing to find a symbols url should be a warning instead of a exception. https://reviewboard.mozilla.org/r/80836/#review79744 I am less familiar with current state than you and whimboo are. as this is not part of core mh(it's just a subclass of our test scripts), I'll defer to whimboo to give the final r+ once his conerns are addressed.
Attachment #8794331 - Flags: review?(jlund)
Comment on attachment 8794331 [details] Bug 1303668 - Failing to find a symbols url should be a warning instead of a exception. https://reviewboard.mozilla.org/r/80836/#review80420
Attachment #8794331 - Flags: review?(hskupin) → review+
Pushed by armenzg@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/1418ec10eda2 Failing to find a symbols url should be a warning instead of a exception. r=whimboo
Backout by gszorc@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/a008ff0d08b4 Backed out changeset 1418ec10eda2 for mozharness bustage
Pushed by armenzg@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/a7c3db5394f2 Failing to find a symbols url should be a warning instead of a exception. r=whimboo
https://hg.mozilla.org/mozilla-central/rev/a7c3db5394f2
Status: NEW → RESOLVED
Closed: 9 years ago
status-firefox52: --- → fixed
Resolution: --- → FIXED
If I understand https://treeherder.mozilla.org/logviewer.html#?job_id=11788227&repo=fx-team correctly, that's not enough, because the FATAL problem is that after failing to find the symbols URL, we just merrily go ahead and "INFO - Fetch None into memory" which doesn't go well for us.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
For that log, we obsiously have symbols avalaible: https://queue.taskcluster.net/v1/task/TeCsg2htQYeMN5JVX8RkEQ/runs/0/artifacts/public/build/target.crashreporter-symbols.zip I believe if we retry when we expect symbols (e.g. debug tests) we would be able to reduce this.
Attachment #8794331 - Attachment is obsolete: true
Attachment #8797171 - Flags: review?(hskupin)
Comment on attachment 8797171 [details] Bug 1303668 - Retry downloading symbols if we're expecting them. https://reviewboard.mozilla.org/r/82772/#review82794 I'm not an official peer of mozharness. So you might want to run this review request through Jordan first.
Attachment #8797171 - Flags: review?(jlund) → review?(aki)
Attachment #8797171 - Flags: review?(hskupin)
Comment on attachment 8797171 [details] Bug 1303668 - Retry downloading symbols if we're expecting them. https://reviewboard.mozilla.org/r/82772/#review82978
Attachment #8797171 - Flags: review?(aki) → review+
Pushed by armenzg@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/df471131e1b6 Retry downloading symbols if we're expecting them. r=aki
https://hg.mozilla.org/mozilla-central/rev/df471131e1b6
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Assignee: armenzg → nobody
The retry() added in comment 20 generally doesn't work (doesn't retry) because query_symbols_url() catches exceptions -- it may return None, but it never fails from the retry() perspective. I think the retry might work without the change landed in comment 12 (change the warning() back to exception()). I would also like to report the exception causing query_symbols_url() to fail. For the failures that I checked, the symbols artifact is present, so I am hopeful that the retry will work. Still, I wonder why the failed jobs are not finding the symbols when they check.
Assignee: nobody → gbrown
With this patch, query_symbols_url() takes a new parameter, raise_on_failure, which is set to true when called from Armen's retry(); if raise_on_failure is set and the url lookup fails, query_symbols_url() will raise an exception, triggering a retry. Other callers of query_symbols_url(), which don't retry and appear to be more tolerant of query_symbols_url() failures do not set retry_on_failure; I anticipate no change in behavior for those callers. I also updated the query_symbols_url() warning wording to give us more information. Finally, _download_and_extract_symbols() now does not call download_unpack() if the symbols_url is not set; this should not be necessary since retry() should be fatal on failure, but is logical and provides further protection from the exception in this bug. Normal run without failure: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7312358584ce878a5b4f2e4de5fd07ad649ee86b Forced permanent error: https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa02a72deb8d42b63dc3c70f297dc099c5a06cf1 https://public-artifacts.taskcluster.net/TF_RZWR1RjKr0H6CY3ZYBw/0/public/logs/live_backing.log [task 2017-01-12T22:06:09.713180Z] 22:06:09 INFO - retry: Calling query_symbols_url with args: (True,), kwargs: {}, attempt #1 [task 2017-01-12T22:06:15.353281Z] 22:06:15 WARNING - Cannot open symbols url https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.crashreporter-symbols.zip (installer url: https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.tar.bz2): 42 [task 2017-01-12T22:06:15.354712Z] 22:06:15 INFO - retry: attempt #1 caught exception: 42 [task 2017-01-12T22:06:15.355387Z] 22:06:15 INFO - retry: Failed, sleeping 20 seconds before retrying [task 2017-01-12T22:06:35.377218Z] 22:06:35 INFO - retry: Calling query_symbols_url with args: (True,), kwargs: {}, attempt #2 [task 2017-01-12T22:06:35.824206Z] 22:06:35 WARNING - Cannot open symbols url https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.crashreporter-symbols.zip (installer url: https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.tar.bz2): 42 [task 2017-01-12T22:06:35.826373Z] 22:06:35 INFO - retry: attempt #2 caught exception: 42 [task 2017-01-12T22:06:35.827786Z] 22:06:35 INFO - retry: Failed, sleeping 40 seconds before retrying [task 2017-01-12T22:07:15.859127Z] 22:07:15 INFO - retry: Calling query_symbols_url with args: (True,), kwargs: {}, attempt #3 [task 2017-01-12T22:07:16.255247Z] 22:07:16 WARNING - Cannot open symbols url https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.crashreporter-symbols.zip (installer url: https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.tar.bz2): 42 [task 2017-01-12T22:07:16.257462Z] 22:07:16 INFO - retry: attempt #3 caught exception: 42 [task 2017-01-12T22:07:16.258832Z] 22:07:16 INFO - retry: Failed, sleeping 80 seconds before retrying [task 2017-01-12T22:08:36.340457Z] 22:08:36 INFO - retry: Calling query_symbols_url with args: (True,), kwargs: {}, attempt #4 [task 2017-01-12T22:08:36.785815Z] 22:08:36 WARNING - Cannot open symbols url https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.crashreporter-symbols.zip (installer url: https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.tar.bz2): 42 [task 2017-01-12T22:08:36.788011Z] 22:08:36 INFO - retry: attempt #4 caught exception: 42 [task 2017-01-12T22:08:36.789372Z] 22:08:36 INFO - retry: Failed, sleeping 160 seconds before retrying [task 2017-01-12T22:11:16.857755Z] 22:11:16 INFO - retry: Calling query_symbols_url with args: (True,), kwargs: {}, attempt #5 [task 2017-01-12T22:11:17.219013Z] 22:11:17 WARNING - Cannot open symbols url https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.crashreporter-symbols.zip (installer url: https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.tar.bz2): 42 [task 2017-01-12T22:11:17.221217Z] 22:11:17 INFO - retry: attempt #5 caught exception: 42 [task 2017-01-12T22:11:17.222591Z] 22:11:17 FATAL - We can't proceed without downloading symbols. [task 2017-01-12T22:11:17.224379Z] 22:11:17 FATAL - Caught exception: 42 [task 2017-01-12T22:11:17.225981Z] 22:11:17 FATAL - Caught exception: 42 [task 2017-01-12T22:11:17.227562Z] 22:11:17 FATAL - Caught exception: 42 [task 2017-01-12T22:11:17.229180Z] 22:11:17 FATAL - Caught exception: 42 [task 2017-01-12T22:11:17.230765Z] 22:11:17 FATAL - Caught exception: 42 [task 2017-01-12T22:11:17.232422Z] 22:11:17 FATAL - Running post_fatal callback... [task 2017-01-12T22:11:17.233995Z] 22:11:17 FATAL - Exiting -1
Attachment #8826411 - Flags: review?(aki)
Comment on attachment 8826411 [details] [diff] [review] fix retry of query_symbols_url >- def query_symbols_url(self): >+ def query_symbols_url(self, raise_on_failure=False): > if self.symbols_url: > return self.symbols_url > >@@ -206,8 +206,11 @@ class TestingMixin(VirtualenvMixin, Buil > if symbols_url: > self._urlopen(symbols_url, timeout=120) > self.symbols_url = symbols_url >- except (urllib2.HTTPError, urllib2.URLError, socket.error, socket.timeout): >- self.warning("Can't figure out symbols_url from installer_url: %s!" % self.installer_url) >+ except (urllib2.HTTPError, urllib2.URLError, socket.error, socket.timeout) as ex: >+ self.warning("Cannot open symbols url %s (installer url: %s): %s" % >+ (symbols_url, self.installer_url, ex)) >+ if raise_on_failure: >+ raise Good catch! We retry on exceptions. > > # If no symbols URL can be determined let minidump_stackwalk query the symbols. > # As of now this only works for Nightly and release builds. >@@ -524,6 +527,7 @@ 2. running via buildbot and running the > # before being unable to proceed (e.g. debug tests need symbols) > self.symbols_url = self.retry( > action=self.query_symbols_url, >+ args=(True,), This will work because raise_on_failure is the first kwarg for query_symbols_url. Relying on that order is more implicit than explicit, and can break if someone introduces another kwarg in the wrong place. kwargs={'raise_on_failure': True} would be more explicit.
Attachment #8826411 - Flags: review?(aki) → review+
(In reply to Aki Sasaki [:aki] from comment #36) > kwargs={'raise_on_failure': True} would be more explicit. Good idea - will do. Thanks!
Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/40e0595028bf Actually retry when mozharness query_symbols_url() fails; r=aki
https://hg.mozilla.org/mozilla-central/rev/40e0595028bf
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
status-firefox53: --- → fixed
Resolution: --- → FIXED
See Also: → 1301807
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: