Closed Bug 1278445 Opened 6 years ago Closed 6 years ago

mozilla-beta build failure on linux32 (and others): Error: SOCORRO_UPLOAD_TOKEN_FILE "/builds/crash-stats-api.token" does not exist


(Taskcluster :: Operations and Service Requests, task)

Not set


(Not tracked)



(Reporter: selenamarie, Assigned: dustin)




(1 file)

Some Linux builds on mozilla-beta are failing with:

INFO - Error: SOCORRO_UPLOAD_TOKEN_FILE "/builds/crash-stats-api.token" does not exist


I looked at the scopes and both mozilla-beta and mozilla-aurora have:


I'm sure I'm missing something.
Affected builds: 

Linux64 opt
Linux32 opt

Also, wasn't able to find mentioned in the TC linux64 debug build logs.
Ah, that's not the error I was thinking it was.

it sounds like something in the mozharness config on this branch is causing it to attempt to actually upload the symbols, rather than just storing them as an artifact for the symbol-upload task to process.  Maybe ted can see why that's the case?
Flags: needinfo?(ted)
So that error comes from here:

which gets invoked from `make uploadsymbols`:

In automation, that target is controlled by `MOZ_AUTOMATION_UPLOAD_SYMBOLS`:

which typically gets set in mozconfigs:

It gets set to `0` by default in mozconfig.automation if it's unset:

Nightly Linux64 builds use this mozconfig:

That sources the common-opt mozconfig:

which sources build/unix/mozconfig.linux, which sets `MOZ_AUTOMATION_UPLOAD_SYMBOLS=1` if `MOZ_NIGHTLY=yes` *and* it's not already set:

The beta mozconfig is here, it sets `MOZ_AUTOMATION_UPLOAD_SYMBOLS=1` uncondtionally if `ENABLE_RELEASE_PROMOTION` is set:

So! Taskcluster builds on mozilla-{inbound,central} aren't actually building as nightlies, so they never set `MOZ_NIGHTLY=yes` so they don't wind up turning on symbol upload. The build linked from comment 0 has `ENABLE_RELEASE_PROMOTION=1`:
21:00:31     INFO -  'ENABLE_RELEASE_PROMOTION': '1',
``` it winds up trying to upload symbols. We would have hit this same problem if we either tried to build nightlies in Taskcluster on mozilla-central, or enabled nightly promotion on m-c.

We could fix the beta mozconfig to only set `MOZ_AUTOMATION_UPLOAD_SYMBOLS` if it's not already set, and then set it to 0 in the taskcluster environment somewhere. ( Docker image?)
Flags: needinfo?(ted)
Thanks for the detail!!

That sounds like a job for
dustin, any chance we can fix this in the next weeks or so ?
Flags: needinfo?(dustin)
I can try, but I only half-understand what Ted's suggesting (I understand the part, not the mozconfig/mozharness part).  So someone more knowledgeable than me should feel free to steal this.
Assignee: nobody → dustin
Flags: needinfo?(dustin)
(Sorry Andrew -- Ted is cleverly not accepting r? while he's on PTO)

I don't really know how to test this..
Attachment #8769780 - Flags: review?(ahalberstadt) → review?(cmanchester)
This is pretty simple, and looks fine to me.. but I'm not a build peer. I think Chris can help you out instead.
Attachment #8769780 - Flags: review?(cmanchester) → review?(mshal)
Sorry to bounce this around, I think mshal might have more context on this.
Comment on attachment 8769780 [details]
Bug 1278445: do not try to upload symbols from beta;

::: taskcluster/scripts/builder/
(Diff revision 1)
>  # use "simple" package names so that they can be hard-coded in the task's
>  # extras.locations
>  export MOZ_SIMPLE_PACKAGE_NAME=target
> +# Never try to upload symbols (it doesn't work)

This will go away with bug 1164615, correct? If so, can you add that bug in the comment?
Attachment #8769780 - Flags: review?(mshal) → review+
Comment on attachment 8769780 [details]
Bug 1278445: do not try to upload symbols from beta;

Review request updated; see interdiff:
Attachment #8769780 - Attachment description: Bug 1278445: do not try to upload symbols from beta; .mielczarek → Bug 1278445: do not try to upload symbols from beta;
Pushed by
do not try to upload symbols from beta; r=mshal
Closed: 6 years ago
Resolution: --- → FIXED
Can this be uplifted?
Flags: needinfo?(dustin)
I'm happy for it to be uplifted.  Before we spread it all around, though, perhaps :ted can verify that it does what he intended :)
Flags: needinfo?(dustin) → needinfo?(ted)
ted: could you take a look at this, this is a perma failure and takes some ressouces with staring and such that we could elsewhere better :)
Your patch looks sensible. It should be fine to uplift.
Flags: needinfo?(ted)
self-ni so I can rebase this to apply to (current) beta, and to get a followup patch to adjust remaining mozconfigs too.
Flags: needinfo?(bugspam.Callek)
Blocks: 1290984
Let's turn to the expert..
Flags: needinfo?(dustin) → needinfo?(ted)
(In reply to Dustin J. Mitchell [:dustin] from comment #32)
> Let's turn to the expert..

I have a fix -- export'ing the var in the bash script wrapper...
Flags: needinfo?(ted)
Blocks: 1291337
Depends on: 1320762
See Also: → 1333230
Component: Service Request → Operations and Service Requests
You need to log in before you can comment on or make changes to this bug.