1303759 - Improve symbols fetching

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Armen [:armenzg]]

It seems that there are various bugs where downloading the symbols seems to fail for a reason or another.

Let's track it here and fix them.

We need some retry logic when calling self._urlopen() [1]

For Asan tests we should not call query_symbols_url() since urlopen() fails 100% of the time.

See also in bug 1283879 we ensured that we stop calling --download-symbols (I think) for Asan tests.

[1] https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/testbase.py#190

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

ASAN builds do not produce any symbols so this failure would be expected. That's why we do not pass in the symbols option for those builds to the mozharness scripts.

Comment 2

[Armen [:armenzg]]

We tell them to be downloaded ondemand.
We should set self.symbols_path and self.symbols_url to None.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #2)
> We tell them to be downloaded ondemand.

This is not what I see when I check the patch as landed:
https://bug1283879.bmoattachments.org/attachment.cgi?id=8779109

Where have you seen this?

> We should set self.symbols_path and self.symbols_url to None.

We do not set any of those for ASAN builds. 

So on which assumptions or failures has this bug been created of? I believe that the main issue here really is that if AWS fails, we do not retry and an exception gets bubbled up.

Comment 4

[Armen [:armenzg]]

(In reply to Henrik Skupin (:whimboo) from comment #3)
> (In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #2)
> > We tell them to be downloaded ondemand.
> 
> This is not what I see when I check the patch as landed:
> https://bug1283879.bmoattachments.org/attachment.cgi?id=8779109
> 
> Where have you seen this?
> 

Bah. My apologies. I must have got confused looking at the wrong logs.

IIUC by deleting 'download_symbols' as one of the config values we got through the 'true' path:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/testbase.py#
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/testbase.py#513-518

> > We should set self.symbols_path and self.symbols_url to None.
> 
> We do not set any of those for ASAN builds. 
> 
> So on which assumptions or failures has this bug been created of? I believe
> that the main issue here really is that if AWS fails, we do not retry and an
> exception gets bubbled up.

Yes, we do not retry and the exception get bubbled up.
When an orange asan test job appears then the http errors also suggested to be tagged with a bug.
Even though the failure to find symbols is irrelevant to the test failure.

Comment 5

[Armen [:armenzg]]

I ended up putting the fix on bug 1303668.