1278445 - mozilla-beta build failure on linux32 (and others): Error: SOCORRO_UPLOAD_TOKEN_FILE "/builds/crash-stats-api.token" does not exist

Status: RESOLVED FIXED

(Taskcluster :: Operations and Service Requests, task)

Description

[Selena Deckelmann :selenamarie :selena]

Some Linux builds on mozilla-beta are failing with:

INFO - Error: SOCORRO_UPLOAD_TOKEN_FILE "/builds/crash-stats-api.token" does not exist

Example: https://tools.taskcluster.net/task-inspector/#OXhKAaZAS9iWKfrIX_Hg5g/0

I looked at the scopes and both mozilla-beta and mozilla-aurora have:

docker-worker:image:taskclusterprivate/upload_symbols:*
queue:create-task:aws-provisioner-v1/symbol-upload

I'm sure I'm missing something.

Comment 1

[Selena Deckelmann :selenamarie :selena]

Affected builds: 

Linux64 opt
Linux32 opt

Also, wasn't able to find upload_symbols.py mentioned in the TC linux64 debug build logs.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, that's not the error I was thinking it was.

it sounds like something in the mozharness config on this branch is causing it to attempt to actually upload the symbols, rather than just storing them as an artifact for the symbol-upload task to process.  Maybe ted can see why that's the case?

Comment 3

[(not currently active) Ted Mielczarek]

So that error comes from here:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/toolkit/crashreporter/tools/upload_symbols.py#66

which gets invoked from `make uploadsymbols`:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/Makefile.in#322

In automation, that target is controlled by `MOZ_AUTOMATION_UPLOAD_SYMBOLS`:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/build/moz-automation.mk#35

which typically gets set in mozconfigs:
https://dxr.mozilla.org/mozilla-central/search?q=MOZ_AUTOMATION_UPLOAD_SYMBOLS

It gets set to `0` by default in mozconfig.automation if it's unset:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/build/mozconfig.automation#19

Nightly Linux64 builds use this mozconfig:
https://dxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/linux64/nightly

That sources the common-opt mozconfig:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/browser/config/mozconfigs/linux64/common-opt#8

which sources build/unix/mozconfig.linux, which sets `MOZ_AUTOMATION_UPLOAD_SYMBOLS=1` if `MOZ_NIGHTLY=yes` *and* it's not already set:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/build/unix/mozconfig.linux#3

The beta mozconfig is here, it sets `MOZ_AUTOMATION_UPLOAD_SYMBOLS=1` uncondtionally if `ENABLE_RELEASE_PROMOTION` is set:
https://dxr.mozilla.org/mozilla-beta/rev/609000bcc11211d7c27ceea36fa2d2262fa0523f/browser/config/mozconfigs/linux64/beta#3

So! Taskcluster builds on mozilla-{inbound,central} aren't actually building as nightlies, so they never set `MOZ_NIGHTLY=yes` so they don't wind up turning on symbol upload. The build linked from comment 0 has `ENABLE_RELEASE_PROMOTION=1`:
```
21:00:31     INFO -  'ENABLE_RELEASE_PROMOTION': '1',
```

...so it winds up trying to upload symbols. We would have hit this same problem if we either tried to build nightlies in Taskcluster on mozilla-central, or enabled nightly promotion on m-c.

We could fix the beta mozconfig to only set `MOZ_AUTOMATION_UPLOAD_SYMBOLS` if it's not already set, and then set it to 0 in the taskcluster environment somewhere. (build-linux.sh? Docker image?)

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Thanks for the detail!!

That sounds like a job for build-linux.sh

Comment 5

[Intermittent Failures Robot]

18 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 18

Platform breakdown:
* linux64: 12
* linux32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-06-21&endday=2016-06-21&tree=all

Comment 6

[Intermittent Failures Robot]

96 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 96

Platform breakdown:
* linux64: 64
* linux32: 32

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-06-20&endday=2016-06-26&tree=all

Comment 7

[Carsten Book [:Tomcat]]

dustin, any chance we can fix this in the next weeks or so ?

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I can try, but I only half-understand what Ted's suggesting (I understand the build-linux.sh part, not the mozconfig/mozharness part).  So someone more knowledgeable than me should feel free to steal this.

Comment 9

[Intermittent Failures Robot]

49 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 49

Platform breakdown:
* linux64: 32
* linux32: 17

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-06-27&endday=2016-07-03&tree=all

Comment 10

[Intermittent Failures Robot]

27 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 27

Platform breakdown:
* linux64: 18
* linux32: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-05&endday=2016-07-05&tree=all

Comment 11

[Intermittent Failures Robot]

18 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 18

Platform breakdown:
* linux64: 12
* linux32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-08&endday=2016-07-08&tree=all

Comment 12

[Intermittent Failures Robot]

101 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 101

Platform breakdown:
* linux64: 57
* linux32: 26
* osx-10-10: 8
* windows7-32: 4
* osx-10-6: 4
* windowsxp: 1
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-04&endday=2016-07-10&tree=all

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1278445: do not try to upload symbols from beta;

Review commit: https://reviewboard.mozilla.org/r/63496/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/63496/

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

(Sorry Andrew -- Ted is cleverly not accepting r? while he's on PTO)

I don't really know how to test this..

Comment 15

[Andrew Halberstadt [:ahal]]

This is pretty simple, and looks fine to me.. but I'm not a build peer. I think Chris can help you out instead.

Comment 16

[Chris Manchester (limited bugmail, email directly)]

Sorry to bounce this around, I think mshal might have more context on this.

Comment 17

[Michael Shal [:mshal]]

Comment on attachment 8769780 [details]
Bug 1278445: do not try to upload symbols from beta;

https://reviewboard.mozilla.org/r/63496/#review60384

::: taskcluster/scripts/builder/build-linux.sh:45
(Diff revision 1)
>  
>  # use "simple" package names so that they can be hard-coded in the task's
>  # extras.locations
>  export MOZ_SIMPLE_PACKAGE_NAME=target
>  
> +# Never try to upload symbols (it doesn't work)

This will go away with bug 1164615, correct? If so, can you add that bug in the comment?

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8769780 [details]
Bug 1278445: do not try to upload symbols from beta;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/63496/diff/1-2/

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c328693e714f
do not try to upload symbols from beta; r=mshal

Comment 20

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/c328693e714f

Comment 21

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Can this be uplifted?

Comment 22

[Intermittent Failures Robot]

16 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 16

Platform breakdown:
* linux64: 8
* linux32: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-14&endday=2016-07-14&tree=all

Comment 23

[Intermittent Failures Robot]

44 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 44

Platform breakdown:
* linux64: 24
* linux32: 20

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-11&endday=2016-07-17&tree=all

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

I'm happy for it to be uplifted.  Before we spread it all around, though, perhaps :ted can verify that it does what he intended :)

Comment 25

[Carsten Book [:Tomcat]]

ted: could you take a look at this, this is a perma failure and takes some ressouces with staring and such that we could elsewhere better :)

Comment 26

[(not currently active) Ted Mielczarek]

Your patch looks sensible. It should be fine to uplift.

Comment 27

[Intermittent Failures Robot]

38 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 38

Platform breakdown:
* linux64: 27
* linux32: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-18&endday=2016-07-24&tree=all

Comment 28

[Intermittent Failures Robot]

11 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 9
* mozilla-release: 2

Platform breakdown:
* linux64: 7
* linux32: 3
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-25&endday=2016-07-31&tree=all

Comment 29

[Justin Wood (:Callek)]

self-ni so I can rebase this to apply to (current) beta, and to get a followup patch to adjust remaining mozconfigs too.

Comment 30

[Justin Wood (:Callek)]

https://hg.mozilla.org/releases/mozilla-beta/rev/3998308ce605

Comment 31

[Carsten Book [:Tomcat]]

still fails after Calleks push https://treeherder.mozilla.org/logviewer.html#?job_id=1396661&repo=mozilla-beta

Comment 32

[Dustin J. Mitchell [:dustin] (he/him)]

Let's turn to the expert..

Comment 33

[Justin Wood (:Callek)]

(In reply to Dustin J. Mitchell [:dustin] from comment #32)
> Let's turn to the expert..

I have a fix -- export'ing the var in the bash script wrapper...