The default bug view has changed. See this FAQ.

Linux Mulet nightlies failing during the uploadsymbols step of the build process

RESOLVED FIXED

Status

Release Engineering
General Automation
--
critical
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: RyanVM, Assigned: jgriffin)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
These were enabled today and are failing on trunk and Aurora.

https://tbpl.mozilla.org/php/getParsedLog.php?id=48423150&tree=Mozilla-Central

Linux x86-64 Mulet mozilla-central nightly on 2014-09-18 17:09:06 PDT for push c8e325eee9e1
slave: b-linux64-ix-0008

INFO: mock_mozilla.py version 1.0.3 starting...
State Changed: init plugins
INFO: selinux disabled
State Changed: start
State Changed: lock buildroot
State Changed: shell
retry: Calling <function run_with_timeout at 0x1d5e668> with args: (['make', 'uploadsymbols'], 2460, None, None, False, True), kwargs: {}, attempt #1
Executing: ['make', 'uploadsymbols']
/bin/sh /builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh firefox-35.0a1-Linux-20140918160202-x86_64-symbols.txt 'dist/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip'
/builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh: line 26: SYMBOL_SERVER_HOST: parameter null or not set
make: *** [uploadsymbols] Error 1
retry: Failed, sleeping 1 seconds before retrying
retry: Calling <function run_with_timeout at 0x1d5e668> with args: (['make', 'uploadsymbols'], 2460, None, None, False, True), kwargs: {}, attempt #2
Executing: ['make', 'uploadsymbols']
/bin/sh /builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh firefox-35.0a1-Linux-20140918160202-x86_64-symbols.txt 'dist/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip'
/builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh: line 26: SYMBOL_SERVER_HOST: parameter null or not set
make: *** [uploadsymbols] Error 1
retry: Failed, sleeping 2 seconds before retrying
retry: Calling <function run_with_timeout at 0x1d5e668> with args: (['make', 'uploadsymbols'], 2460, None, None, False, True), kwargs: {}, attempt #3
Executing: ['make', 'uploadsymbols']
/bin/sh /builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh firefox-35.0a1-Linux-20140918160202-x86_64-symbols.txt 'dist/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip'
/builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh: line 26: SYMBOL_SERVER_HOST: parameter null or not set
make: *** [uploadsymbols] Error 1
retry: Failed, sleeping 4 seconds before retrying
retry: Calling <function run_with_timeout at 0x1d5e668> with args: (['make', 'uploadsymbols'], 2460, None, None, False, True), kwargs: {}, attempt #4
Executing: ['make', 'uploadsymbols']
/bin/sh /builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh firefox-35.0a1-Linux-20140918160202-x86_64-symbols.txt 'dist/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip'
/builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh: line 26: SYMBOL_SERVER_HOST: parameter null or not set
make: *** [uploadsymbols] Error 1
retry: Failed, sleeping 8 seconds before retrying
retry: Calling <function run_with_timeout at 0x1d5e668> with args: (['make', 'uploadsymbols'], 2460, None, None, False, True), kwargs: {}, attempt #5
Executing: ['make', 'uploadsymbols']
/bin/sh /builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh firefox-35.0a1-Linux-20140918160202-x86_64-symbols.txt 'dist/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip'
/builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh: line 26: SYMBOL_SERVER_HOST: parameter null or not set
make: *** [uploadsymbols] Error 1
retry: Giving up on <function run_with_timeout at 0x1d5e668>
Unable to successfully run ['make', 'uploadsymbols'] after 5 attempts
The failure above is uploading to Socorro, which seems of little utility to me but I don't really how widely these builds are, or will be, used. Perhaps we only need SYMBOL_SERVER_HOST files for tests ?

The relevant code is at  http://hg.mozilla.org/build/buildbotcustom/file/default/process/factory.py#l1077:
  1077         if self.uploadSymbols or (not self.disableSymbols and self.packageTests):
  1078             self.addBuildSymbolsStep()
  1079         if self.uploadSymbols:
  1080             self.addUploadSymbolsStep()

The linux-mulet platform has upload_symbols set to True in b2g_config.py, and self.uploadSymbols ends up True. We already have settings to make the 2nd half of the or true, so the way forward is

* if we don't need Socorro upload, change http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/b2g_config.py#l406 to False
* if we do want Socorro, then see http://hg.mozilla.org/mozilla-central/file/default/toolkit/crashreporter/tools/upload_symbols.sh for what need to be set in the env (SYMBOL_SERVER_HOST), and figure out how to modify what we have in config.py to do that. We should use a different SYMBOL_SERVER_PATH at least.
Component: Buildduty → General Automation
QA Contact: bugspam.Callek → catlee
Summary: Mulet nightlies failing during the uploadsymbols step of the build process → Linux Mulet nightlies failing during the uploadsymbols step of the build process
(In reply to Nick Thomas [:nthomas] from comment #1)
> Perhaps we only need SYMBOL_SERVER_HOST files for tests ?

Oopsie, meant files like firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip.
(Assignee)

Comment 3

3 years ago
Currently, I'm thinking we don't need soccoro, so I'll do the simplest thing and turn the upload off.  We can revisit later if we decide we want this.
(Assignee)

Comment 4

3 years ago
Created attachment 8492254 [details] [diff] [review]
Turn off symbol uploads,
Attachment #8492254 - Flags: review?(nthomas)
(Assignee)

Updated

3 years ago
Assignee: nobody → jgriffin
Comment on attachment 8492254 [details] [diff] [review]
Turn off symbol uploads,

Works as expected in a master dump.
Attachment #8492254 - Flags: review?(nthomas) → review+
(Assignee)

Comment 6

3 years ago
Comment on attachment 8492254 [details] [diff] [review]
Turn off symbol uploads,

Thanks for verifying.

https://hg.mozilla.org/build/buildbot-configs/rev/00a5bfaa2bc5
Attachment #8492254 - Flags: checked-in+
I'm seeing mulet symbolupload errors on try -  i assume we should skip it there as well?
E.g. https://tbpl.mozilla.org/?tree=Try&rev=a6cf446146c0
(Assignee)

Comment 8

3 years ago
(In reply to Georg Fritzsche [:gfritzsche] from comment #7)
> I'm seeing mulet symbolupload errors on try -  i assume we should skip it
> there as well?
> E.g. https://tbpl.mozilla.org/?tree=Try&rev=a6cf446146c0

Yes, this will be fixed as soon as the patch for this bug goes into production.
(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> Currently, I'm thinking we don't need soccoro, so I'll do the simplest thing
> and turn the upload off.  We can revisit later if we decide we want this.

We'd only need uploadsymbols run if we either:
a) Intend to have users running these nightlies and want to accept crash reports or
b) Want to allow users to debug (with a native debugger) the nightlies we're shipping using the symbol server
Something here landed in production today: https://wiki.mozilla.org/ReleaseEngineering/Maintenance#Reconfigs_.2F_Deployments
(Assignee)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.