Closed Bug 1303668 Opened 8 years ago Closed 7 years ago

Intermittent AttributeError: 'NoneType' object has no attribute 'find' when fallback for not finding symbols URL doesn't work

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

Product:
Release Engineering
Component:
Applications: MozharnessCore
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox50 wontfix, firefox51 fixed, firefox52 fixed, firefox53 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox50 --- wontfix
firefox51 --- fixed
firefox52 --- fixed
firefox53 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: gbrown)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files, 1 obsolete file)

Bug 1303668 - Retry downloading symbols if we're expecting them.
58 bytes, text/x-review-board-request
mozilla
: review+
Details
fix retry of query_symbols_url
2.35 KB, patch
mozilla
: review+
Details | Diff | Splinter Review 
Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=3671062&repo=autoland

https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-linux-debug/1474251137/autoland_ubuntu32_vm-debug_test-mochitest-4-bm08-tests1-linux32-build68.txt.gz

21:34:06  WARNING - Can't figure out symbols_url from installer_url: https://queue.taskcluster.net/v1/task/c5zgKXI2QQmKduigjtiIcg/artifacts/public/build/firefox-51.0a1.en-US.linux-i686.tar.bz2!
(traceback from a connection reset while querying the symbols_url)
WARNING - No symbols_url found. Let minidump_stackwalk query for symbols.
 21:34:06     INFO - Setting buildbot property symbols_url to None
 21:34:06     INFO - Writing buildbot properties ['symbols_url'] to /builds/slave/test/properties/symbols_url
 21:34:06     INFO - Writing to file /builds/slave/test/properties/symbols_url
 21:34:06     INFO - Contents:
 21:34:06     INFO -  symbols_url:None
 21:34:06     INFO - Downloading and extracting to /builds/slave/test/build/symbols these dirs * from None
 21:34:06     INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': None}, attempt #1
 21:34:06     INFO - Fetch None into memory
 21:34:06     INFO - Running post-action listener: _resource_record_post_action
 21:34:06     INFO - Running post-action listener: set_extra_try_arguments
 21:34:06     INFO - [mozharness: 2016-09-19 04:34:06.136733Z] Finished download-and-extract step (failed)
 21:34:06    FATAL - Uncaught exception: Traceback (most recent call last):
 21:34:06    FATAL -   File "/builds/slave/test/scripts/mozharness/base/script.py", line 2001, in run
 21:34:06    FATAL -     self.run_action(action)
 21:34:06    FATAL -   File "/builds/slave/test/scripts/mozharness/base/script.py", line 1940, in run_action
 21:34:06    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
 21:34:06    FATAL -   File "/builds/slave/test/scripts/mozharness/base/script.py", line 1880, in _possibly_run_method
 21:34:06    FATAL -     return getattr(self, method_name)()
 21:34:06    FATAL -   File "scripts/scripts/desktop_unittest.py", line 505, in download_and_extract
 21:34:06    FATAL -     suite_categories=target_categories)
 21:34:06    FATAL -   File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 557, in download_and_extract
 21:34:06    FATAL -     self._download_and_extract_symbols()
 21:34:06    FATAL -   File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 518, in _download_and_extract_symbols
 21:34:06    FATAL -     self.download_unpack(self.symbols_url, self.symbols_path)
 21:34:06    FATAL -   File "/builds/slave/test/scripts/mozharness/base/script.py", line 675, in download_unpack
 21:34:06    FATAL -     **retry_args
 21:34:06    FATAL -   File "/builds/slave/test/scripts/mozharness/base/script.py", line 1099, in retry
 21:34:06    FATAL -     status = action(*args, **kwargs)
 21:34:06    FATAL -   File "/builds/slave/test/scripts/mozharness/base/script.py", line 366, in fetch_url_into_memory
 21:34:06    FATAL -     parsed_url = urlparse.urlparse(url)
 21:34:06    FATAL -   File "/usr/lib/python2.7/urlparse.py", line 135, in urlparse
 21:34:06    FATAL -     tuple = urlsplit(url, scheme, allow_fragments)
 21:34:06    FATAL -   File "/usr/lib/python2.7/urlparse.py", line 174, in urlsplit
 21:34:06    FATAL -     i = url.find(':')
 21:34:06    FATAL - AttributeError: 'NoneType' object has no attribute 'find'
 21:34:06    FATAL - Running post_fatal callback...
 21:34:06    FATAL - Exiting -1

hahahaha, "Fetch None into memory"
Depends on: 1303759
Assignee: nobody → armenzg
Attachment #8794331 - Flags: review?(jlund)
Comment on attachment 8794331 [details]
Bug 1303668 - Failing to find a symbols url should be a warning instead of a exception.

https://reviewboard.mozilla.org/r/80836/#review79652

::: testing/mozharness/mozharness/mozilla/testing/testbase.py
(Diff revision 1)
>              return self.symbols_url
>  
>          elif self.installer_url:
> -            symbols_url = self.query_prefixed_build_dir_url('.crashreporter-symbols.zip')
> +            self.symbols_url = self.query_prefixed_build_dir_url('.crashreporter-symbols.zip')
>  
> -            # Check if the URL exists. If not, use none to allow mozcrash to auto-check for symbols

We should not remove that part. As the comment below states minidump-stackwalk can only auto-detect the symbols for Nightly and release builds but not for CI builds.

I would suggest to maybe reduce the failure from exception to warning.
Comment on attachment 8794331 [details]
Bug 1303668 - Failing to find a symbols url should be a warning instead of a exception.

https://reviewboard.mozilla.org/r/80836/#review79744

I am less familiar with current state than you and whimboo are. as this is not part of core mh(it's just a subclass of our test scripts), I'll defer to whimboo to give the final r+ once his conerns are addressed.
Attachment #8794331 - Flags: review?(jlund)
Comment on attachment 8794331 [details]
Bug 1303668 - Failing to find a symbols url should be a warning instead of a exception.

https://reviewboard.mozilla.org/r/80836/#review80420
Attachment #8794331 - Flags: review?(hskupin) → review+
Pushed by armenzg@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1418ec10eda2
Failing to find a symbols url should be a warning instead of a exception. r=whimboo
Backout by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a008ff0d08b4
Backed out changeset 1418ec10eda2 for mozharness bustage
Pushed by armenzg@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a7c3db5394f2
Failing to find a symbols url should be a warning instead of a exception. r=whimboo
https://hg.mozilla.org/mozilla-central/rev/a7c3db5394f2
Status: NEW → RESOLVED
Closed: 8 years ago
status-firefox52: --- → fixed
Resolution: --- → FIXED
If I understand https://treeherder.mozilla.org/logviewer.html#?job_id=11788227&repo=fx-team correctly, that's not enough, because the FATAL problem is that after failing to find the symbols URL, we just merrily go ahead and "INFO - Fetch None into memory" which doesn't go well for us.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
For that log, we obsiously have symbols avalaible:
https://queue.taskcluster.net/v1/task/TeCsg2htQYeMN5JVX8RkEQ/runs/0/artifacts/public/build/target.crashreporter-symbols.zip

I believe if we retry when we expect symbols (e.g. debug tests) we would be able to reduce this.
Attachment #8794331 - Attachment is obsolete: true
Attachment #8797171 - Flags: review?(hskupin)
Comment on attachment 8797171 [details]
Bug 1303668 - Retry downloading symbols if we're expecting them.

https://reviewboard.mozilla.org/r/82772/#review82794

I'm not an official peer of mozharness. So you might want to run this review request through Jordan first.
Attachment #8797171 - Flags: review?(jlund) → review?(aki)
Attachment #8797171 - Flags: review?(hskupin)
Comment on attachment 8797171 [details]
Bug 1303668 - Retry downloading symbols if we're expecting them.

https://reviewboard.mozilla.org/r/82772/#review82978
Attachment #8797171 - Flags: review?(aki) → review+
Pushed by armenzg@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/df471131e1b6
Retry downloading symbols if we're expecting them. r=aki
https://hg.mozilla.org/mozilla-central/rev/df471131e1b6
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → FIXED
Assignee: armenzg → nobody
The retry() added in comment 20 generally doesn't work (doesn't retry) because query_symbols_url() catches exceptions -- it may return None, but it never fails from the retry() perspective. 

I think the retry might work without the change landed in comment 12 (change the warning() back to exception()).

I would also like to report the exception causing query_symbols_url() to fail.

For the failures that I checked, the symbols artifact is present, so I am hopeful that the retry will work. Still, I wonder why the failed jobs are not finding the symbols when they check.
Assignee: nobody → gbrown
With this patch, query_symbols_url() takes a new parameter, raise_on_failure, which is set to true when called from Armen's retry(); if raise_on_failure is set and the url lookup fails, query_symbols_url() will raise an exception, triggering a retry.

Other callers of query_symbols_url(), which don't retry and appear to be more tolerant of query_symbols_url() failures do not set retry_on_failure; I anticipate no change in behavior for those callers.

I also updated the query_symbols_url() warning wording to give us more information.

Finally, _download_and_extract_symbols() now does not call download_unpack() if the symbols_url is not set; this should not be necessary since retry() should be fatal on failure, but is logical and provides further protection from the exception in this bug.

Normal run without failure: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7312358584ce878a5b4f2e4de5fd07ad649ee86b

Forced permanent error: https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa02a72deb8d42b63dc3c70f297dc099c5a06cf1

https://public-artifacts.taskcluster.net/TF_RZWR1RjKr0H6CY3ZYBw/0/public/logs/live_backing.log

[task 2017-01-12T22:06:09.713180Z] 22:06:09     INFO - retry: Calling query_symbols_url with args: (True,), kwargs: {}, attempt #1
[task 2017-01-12T22:06:15.353281Z] 22:06:15  WARNING - Cannot open symbols url https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.crashreporter-symbols.zip (installer url: https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.tar.bz2): 42
[task 2017-01-12T22:06:15.354712Z] 22:06:15     INFO - retry: attempt #1 caught exception: 42
[task 2017-01-12T22:06:15.355387Z] 22:06:15     INFO - retry: Failed, sleeping 20 seconds before retrying
[task 2017-01-12T22:06:35.377218Z] 22:06:35     INFO - retry: Calling query_symbols_url with args: (True,), kwargs: {}, attempt #2
[task 2017-01-12T22:06:35.824206Z] 22:06:35  WARNING - Cannot open symbols url https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.crashreporter-symbols.zip (installer url: https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.tar.bz2): 42
[task 2017-01-12T22:06:35.826373Z] 22:06:35     INFO - retry: attempt #2 caught exception: 42
[task 2017-01-12T22:06:35.827786Z] 22:06:35     INFO - retry: Failed, sleeping 40 seconds before retrying
[task 2017-01-12T22:07:15.859127Z] 22:07:15     INFO - retry: Calling query_symbols_url with args: (True,), kwargs: {}, attempt #3
[task 2017-01-12T22:07:16.255247Z] 22:07:16  WARNING - Cannot open symbols url https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.crashreporter-symbols.zip (installer url: https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.tar.bz2): 42
[task 2017-01-12T22:07:16.257462Z] 22:07:16     INFO - retry: attempt #3 caught exception: 42
[task 2017-01-12T22:07:16.258832Z] 22:07:16     INFO - retry: Failed, sleeping 80 seconds before retrying
[task 2017-01-12T22:08:36.340457Z] 22:08:36     INFO - retry: Calling query_symbols_url with args: (True,), kwargs: {}, attempt #4
[task 2017-01-12T22:08:36.785815Z] 22:08:36  WARNING - Cannot open symbols url https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.crashreporter-symbols.zip (installer url: https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.tar.bz2): 42
[task 2017-01-12T22:08:36.788011Z] 22:08:36     INFO - retry: attempt #4 caught exception: 42
[task 2017-01-12T22:08:36.789372Z] 22:08:36     INFO - retry: Failed, sleeping 160 seconds before retrying
[task 2017-01-12T22:11:16.857755Z] 22:11:16     INFO - retry: Calling query_symbols_url with args: (True,), kwargs: {}, attempt #5
[task 2017-01-12T22:11:17.219013Z] 22:11:17  WARNING - Cannot open symbols url https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.crashreporter-symbols.zip (installer url: https://queue.taskcluster.net/v1/task/cAlOB_FTRDeRDBsxGBYO4g/artifacts/public/build/target.tar.bz2): 42
[task 2017-01-12T22:11:17.221217Z] 22:11:17     INFO - retry: attempt #5 caught exception: 42
[task 2017-01-12T22:11:17.222591Z] 22:11:17    FATAL - We can't proceed without downloading symbols.
[task 2017-01-12T22:11:17.224379Z] 22:11:17    FATAL - Caught exception: 42
[task 2017-01-12T22:11:17.225981Z] 22:11:17    FATAL - Caught exception: 42
[task 2017-01-12T22:11:17.227562Z] 22:11:17    FATAL - Caught exception: 42
[task 2017-01-12T22:11:17.229180Z] 22:11:17    FATAL - Caught exception: 42
[task 2017-01-12T22:11:17.230765Z] 22:11:17    FATAL - Caught exception: 42
[task 2017-01-12T22:11:17.232422Z] 22:11:17    FATAL - Running post_fatal callback...
[task 2017-01-12T22:11:17.233995Z] 22:11:17    FATAL - Exiting -1
Attachment #8826411 - Flags: review?(aki)
Comment on attachment 8826411 [details] [diff] [review]
fix retry of  query_symbols_url

>-    def query_symbols_url(self):
>+    def query_symbols_url(self, raise_on_failure=False):
>         if self.symbols_url:
>             return self.symbols_url
> 
>@@ -206,8 +206,11 @@ class TestingMixin(VirtualenvMixin, Buil
>                 if symbols_url:
>                     self._urlopen(symbols_url, timeout=120)
>                     self.symbols_url = symbols_url
>-            except (urllib2.HTTPError, urllib2.URLError, socket.error, socket.timeout):
>-                self.warning("Can't figure out symbols_url from installer_url: %s!" % self.installer_url)
>+            except (urllib2.HTTPError, urllib2.URLError, socket.error, socket.timeout) as ex:
>+                self.warning("Cannot open symbols url %s (installer url: %s): %s" %
>+                    (symbols_url, self.installer_url, ex))
>+                if raise_on_failure:
>+                    raise

Good catch! We retry on exceptions.

> 
>         # If no symbols URL can be determined let minidump_stackwalk query the symbols.
>         # As of now this only works for Nightly and release builds.
>@@ -524,6 +527,7 @@ 2. running via buildbot and running the 
>             # before being unable to proceed (e.g. debug tests need symbols)
>             self.symbols_url = self.retry(
>                 action=self.query_symbols_url,
>+                args=(True,),

This will work because raise_on_failure is the first kwarg for query_symbols_url.  Relying on that order is more implicit than explicit, and can break if someone introduces another kwarg in the wrong place.

kwargs={'raise_on_failure': True} would be more explicit.
Attachment #8826411 - Flags: review?(aki) → review+
(In reply to Aki Sasaki [:aki] from comment #36)
> kwargs={'raise_on_failure': True} would be more explicit.

Good idea - will do. Thanks!
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/40e0595028bf
Actually retry when mozharness query_symbols_url() fails; r=aki
https://hg.mozilla.org/mozilla-central/rev/40e0595028bf
Status: REOPENED → RESOLVED
Closed: 8 years ago → 7 years ago
status-firefox53: --- → fixed
Resolution: --- → FIXED
See Also: → 1301807
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: