mozilla-beta build failure on linux32 (and others): Error: SOCORRO_UPLOAD_TOKEN_FILE "/builds/crash-stats-api.token" does not exist

RESOLVED FIXED

Status

RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: selenamarie, Assigned: dustin)

Tracking

Details

Attachments

(1 attachment)

Some Linux builds on mozilla-beta are failing with:

INFO - Error: SOCORRO_UPLOAD_TOKEN_FILE "/builds/crash-stats-api.token" does not exist

Example: https://tools.taskcluster.net/task-inspector/#OXhKAaZAS9iWKfrIX_Hg5g/0

I looked at the scopes and both mozilla-beta and mozilla-aurora have:

docker-worker:image:taskclusterprivate/upload_symbols:*
queue:create-task:aws-provisioner-v1/symbol-upload

I'm sure I'm missing something.
Affected builds: 

Linux64 opt
Linux32 opt

Also, wasn't able to find upload_symbols.py mentioned in the TC linux64 debug build logs.
Ah, that's not the error I was thinking it was.

it sounds like something in the mozharness config on this branch is causing it to attempt to actually upload the symbols, rather than just storing them as an artifact for the symbol-upload task to process.  Maybe ted can see why that's the case?
Flags: needinfo?(ted)
So that error comes from here:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/toolkit/crashreporter/tools/upload_symbols.py#66

which gets invoked from `make uploadsymbols`:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/Makefile.in#322

In automation, that target is controlled by `MOZ_AUTOMATION_UPLOAD_SYMBOLS`:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/build/moz-automation.mk#35

which typically gets set in mozconfigs:
https://dxr.mozilla.org/mozilla-central/search?q=MOZ_AUTOMATION_UPLOAD_SYMBOLS

It gets set to `0` by default in mozconfig.automation if it's unset:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/build/mozconfig.automation#19

Nightly Linux64 builds use this mozconfig:
https://dxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/linux64/nightly

That sources the common-opt mozconfig:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/browser/config/mozconfigs/linux64/common-opt#8

which sources build/unix/mozconfig.linux, which sets `MOZ_AUTOMATION_UPLOAD_SYMBOLS=1` if `MOZ_NIGHTLY=yes` *and* it's not already set:
https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/build/unix/mozconfig.linux#3

The beta mozconfig is here, it sets `MOZ_AUTOMATION_UPLOAD_SYMBOLS=1` uncondtionally if `ENABLE_RELEASE_PROMOTION` is set:
https://dxr.mozilla.org/mozilla-beta/rev/609000bcc11211d7c27ceea36fa2d2262fa0523f/browser/config/mozconfigs/linux64/beta#3

So! Taskcluster builds on mozilla-{inbound,central} aren't actually building as nightlies, so they never set `MOZ_NIGHTLY=yes` so they don't wind up turning on symbol upload. The build linked from comment 0 has `ENABLE_RELEASE_PROMOTION=1`:
```
21:00:31     INFO -  'ENABLE_RELEASE_PROMOTION': '1',
```

...so it winds up trying to upload symbols. We would have hit this same problem if we either tried to build nightlies in Taskcluster on mozilla-central, or enabled nightly promotion on m-c.

We could fix the beta mozconfig to only set `MOZ_AUTOMATION_UPLOAD_SYMBOLS` if it's not already set, and then set it to 0 in the taskcluster environment somewhere. (build-linux.sh? Docker image?)
Flags: needinfo?(ted)
Thanks for the detail!!

That sounds like a job for build-linux.sh
18 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 18

Platform breakdown:
* linux64: 12
* linux32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-06-21&endday=2016-06-21&tree=all
96 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 96

Platform breakdown:
* linux64: 64
* linux32: 32

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-06-20&endday=2016-06-26&tree=all
dustin, any chance we can fix this in the next weeks or so ?
Flags: needinfo?(dustin)
I can try, but I only half-understand what Ted's suggesting (I understand the build-linux.sh part, not the mozconfig/mozharness part).  So someone more knowledgeable than me should feel free to steal this.
Assignee: nobody → dustin
Flags: needinfo?(dustin)
49 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 49

Platform breakdown:
* linux64: 32
* linux32: 17

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-06-27&endday=2016-07-03&tree=all
27 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 27

Platform breakdown:
* linux64: 18
* linux32: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-05&endday=2016-07-05&tree=all
18 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 18

Platform breakdown:
* linux64: 12
* linux32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-08&endday=2016-07-08&tree=all
101 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 101

Platform breakdown:
* linux64: 57
* linux32: 26
* osx-10-10: 8
* windows7-32: 4
* osx-10-6: 4
* windowsxp: 1
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-04&endday=2016-07-10&tree=all
Created attachment 8769780 [details]
Bug 1278445: do not try to upload symbols from beta;

Review commit: https://reviewboard.mozilla.org/r/63496/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/63496/
Attachment #8769780 - Flags: review?(ahalberstadt)
(Sorry Andrew -- Ted is cleverly not accepting r? while he's on PTO)

I don't really know how to test this..
Attachment #8769780 - Flags: review?(ahalberstadt) → review?(cmanchester)
This is pretty simple, and looks fine to me.. but I'm not a build peer. I think Chris can help you out instead.
Attachment #8769780 - Flags: review?(cmanchester) → review?(mshal)
Sorry to bounce this around, I think mshal might have more context on this.
Comment on attachment 8769780 [details]
Bug 1278445: do not try to upload symbols from beta;

https://reviewboard.mozilla.org/r/63496/#review60384

::: taskcluster/scripts/builder/build-linux.sh:45
(Diff revision 1)
>  
>  # use "simple" package names so that they can be hard-coded in the task's
>  # extras.locations
>  export MOZ_SIMPLE_PACKAGE_NAME=target
>  
> +# Never try to upload symbols (it doesn't work)

This will go away with bug 1164615, correct? If so, can you add that bug in the comment?
Attachment #8769780 - Flags: review?(mshal) → review+
Comment on attachment 8769780 [details]
Bug 1278445: do not try to upload symbols from beta;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/63496/diff/1-2/
Attachment #8769780 - Attachment description: Bug 1278445: do not try to upload symbols from beta; .mielczarek → Bug 1278445: do not try to upload symbols from beta;

Comment 19

2 years ago
Pushed by dmitchell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c328693e714f
do not try to upload symbols from beta; r=mshal

Comment 20

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/c328693e714f
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Can this be uplifted?
Flags: needinfo?(dustin)
16 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 16

Platform breakdown:
* linux64: 8
* linux32: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-14&endday=2016-07-14&tree=all
44 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 44

Platform breakdown:
* linux64: 24
* linux32: 20

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-11&endday=2016-07-17&tree=all
I'm happy for it to be uplifted.  Before we spread it all around, though, perhaps :ted can verify that it does what he intended :)
Flags: needinfo?(dustin) → needinfo?(ted)
ted: could you take a look at this, this is a perma failure and takes some ressouces with staring and such that we could elsewhere better :)
Your patch looks sensible. It should be fine to uplift.
Flags: needinfo?(ted)
38 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 38

Platform breakdown:
* linux64: 27
* linux32: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-18&endday=2016-07-24&tree=all
11 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 9
* mozilla-release: 2

Platform breakdown:
* linux64: 7
* linux32: 3
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1278445&startday=2016-07-25&endday=2016-07-31&tree=all
self-ni so I can rebase this to apply to (current) beta, and to get a followup patch to adjust remaining mozconfigs too.
Flags: needinfo?(bugspam.Callek)

Updated

2 years ago
Blocks: 1290984
still fails after Calleks push https://treeherder.mozilla.org/logviewer.html#?job_id=1396661&repo=mozilla-beta
Flags: needinfo?(dustin)
Let's turn to the expert..
Flags: needinfo?(dustin) → needinfo?(ted)
(In reply to Dustin J. Mitchell [:dustin] from comment #32)
> Let's turn to the expert..

I have a fix -- export'ing the var in the bash script wrapper...
Flags: needinfo?(ted)

Updated

2 years ago
Blocks: 1291337

Updated

2 years ago
Depends on: 1320762

Updated

2 years ago
See Also: → bug 1333230
You need to log in before you can comment on or make changes to this bug.