1069702 - Linux Mulet nightlies failing during the uploadsymbols step of the build process

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

These were enabled today and are failing on trunk and Aurora.

https://tbpl.mozilla.org/php/getParsedLog.php?id=48423150&tree=Mozilla-Central

Linux x86-64 Mulet mozilla-central nightly on 2014-09-18 17:09:06 PDT for push c8e325eee9e1
slave: b-linux64-ix-0008

INFO: mock_mozilla.py version 1.0.3 starting...
State Changed: init plugins
INFO: selinux disabled
State Changed: start
State Changed: lock buildroot
State Changed: shell
retry: Calling <function run_with_timeout at 0x1d5e668> with args: (['make', 'uploadsymbols'], 2460, None, None, False, True), kwargs: {}, attempt #1
Executing: ['make', 'uploadsymbols']
/bin/sh /builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh firefox-35.0a1-Linux-20140918160202-x86_64-symbols.txt 'dist/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip'
/builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh: line 26: SYMBOL_SERVER_HOST: parameter null or not set
make: *** [uploadsymbols] Error 1
retry: Failed, sleeping 1 seconds before retrying
retry: Calling <function run_with_timeout at 0x1d5e668> with args: (['make', 'uploadsymbols'], 2460, None, None, False, True), kwargs: {}, attempt #2
Executing: ['make', 'uploadsymbols']
/bin/sh /builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh firefox-35.0a1-Linux-20140918160202-x86_64-symbols.txt 'dist/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip'
/builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh: line 26: SYMBOL_SERVER_HOST: parameter null or not set
make: *** [uploadsymbols] Error 1
retry: Failed, sleeping 2 seconds before retrying
retry: Calling <function run_with_timeout at 0x1d5e668> with args: (['make', 'uploadsymbols'], 2460, None, None, False, True), kwargs: {}, attempt #3
Executing: ['make', 'uploadsymbols']
/bin/sh /builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh firefox-35.0a1-Linux-20140918160202-x86_64-symbols.txt 'dist/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip'
/builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh: line 26: SYMBOL_SERVER_HOST: parameter null or not set
make: *** [uploadsymbols] Error 1
retry: Failed, sleeping 4 seconds before retrying
retry: Calling <function run_with_timeout at 0x1d5e668> with args: (['make', 'uploadsymbols'], 2460, None, None, False, True), kwargs: {}, attempt #4
Executing: ['make', 'uploadsymbols']
/bin/sh /builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh firefox-35.0a1-Linux-20140918160202-x86_64-symbols.txt 'dist/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip'
/builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh: line 26: SYMBOL_SERVER_HOST: parameter null or not set
make: *** [uploadsymbols] Error 1
retry: Failed, sleeping 8 seconds before retrying
retry: Calling <function run_with_timeout at 0x1d5e668> with args: (['make', 'uploadsymbols'], 2460, None, None, False, True), kwargs: {}, attempt #5
Executing: ['make', 'uploadsymbols']
/bin/sh /builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh firefox-35.0a1-Linux-20140918160202-x86_64-symbols.txt 'dist/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols-full.zip'
/builds/slave/m-cen-l64-mulet-ntly-000000000/build/toolkit/crashreporter/tools/upload_symbols.sh: line 26: SYMBOL_SERVER_HOST: parameter null or not set
make: *** [uploadsymbols] Error 1
retry: Giving up on <function run_with_timeout at 0x1d5e668>
Unable to successfully run ['make', 'uploadsymbols'] after 5 attempts

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

The failure above is uploading to Socorro, which seems of little utility to me but I don't really how widely these builds are, or will be, used. Perhaps we only need SYMBOL_SERVER_HOST files for tests ?

The relevant code is at  http://hg.mozilla.org/build/buildbotcustom/file/default/process/factory.py#l1077:
  1077         if self.uploadSymbols or (not self.disableSymbols and self.packageTests):
  1078             self.addBuildSymbolsStep()
  1079         if self.uploadSymbols:
  1080             self.addUploadSymbolsStep()

The linux-mulet platform has upload_symbols set to True in b2g_config.py, and self.uploadSymbols ends up True. We already have settings to make the 2nd half of the or true, so the way forward is

* if we don't need Socorro upload, change http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/b2g_config.py#l406 to False
* if we do want Socorro, then see http://hg.mozilla.org/mozilla-central/file/default/toolkit/crashreporter/tools/upload_symbols.sh for what need to be set in the env (SYMBOL_SERVER_HOST), and figure out how to modify what we have in config.py to do that. We should use a different SYMBOL_SERVER_PATH at least.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] from comment #1)
> Perhaps we only need SYMBOL_SERVER_HOST files for tests ?

Oopsie, meant files like firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip.

Comment 3

[Jonathan Griffin (:jgriffin)]

Currently, I'm thinking we don't need soccoro, so I'll do the simplest thing and turn the upload off.  We can revisit later if we decide we want this.

Comment 4

[Jonathan Griffin (:jgriffin)]

Attachment: Turn off symbol uploads,

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8492254 [details] [diff] [review]
Turn off symbol uploads,

Works as expected in a master dump.

Comment 6

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8492254 [details] [diff] [review]
Turn off symbol uploads,

Thanks for verifying.

https://hg.mozilla.org/build/buildbot-configs/rev/00a5bfaa2bc5

Comment 7

[Georg Fritzsche [:gfritzsche]]

I'm seeing mulet symbolupload errors on try -  i assume we should skip it there as well?
E.g. https://tbpl.mozilla.org/?tree=Try&rev=a6cf446146c0

Comment 8

[Jonathan Griffin (:jgriffin)]

(In reply to Georg Fritzsche [:gfritzsche] from comment #7)
> I'm seeing mulet symbolupload errors on try -  i assume we should skip it
> there as well?
> E.g. https://tbpl.mozilla.org/?tree=Try&rev=a6cf446146c0

Yes, this will be fixed as soon as the patch for this bug goes into production.

Comment 9

[(not currently active) Ted Mielczarek]

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> Currently, I'm thinking we don't need soccoro, so I'll do the simplest thing
> and turn the upload off.  We can revisit later if we decide we want this.

We'd only need uploadsymbols run if we either:
a) Intend to have users running these nightlies and want to accept crash reports or
b) Want to allow users to debug (with a native debugger) the nightlies we're shipping using the symbol server

Comment 10

[Pete Moore [:pmoore][:pete]]

Something here landed in production today: https://wiki.mozilla.org/ReleaseEngineering/Maintenance#Reconfigs_.2F_Deployments