1797891 - Multiple tasks fail on the toolchains project branch because of missing things in level 2

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Mike Hommey [:glandium]]

• upload-symbols tasks fail with requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-2/gecko-symbol-upload
• build-android-geckoview-docs/opt fails with [e-mail]: Permission denied (publickey). (this one fails on try too. It only works on level 3)
• source-test-doc-upload fails with botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.

In the latter two cases, the tasks are marked as run-as-projects: [mozilla-central], which doesn't prevent triggering them on other branches (which happens automatically on the toolchains branch because it fakes being mozilla-central), but the tasks are only really meant to work on level 3 branches. I feel like we should somehow eliminate those tasks from the graph on non-level 3 branches.

For the former, maybe we'd want to add symbol upload power to level 2?

Comment 1

[Intermittent Failures Robot]

30 failures in 875 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• toolchains: 30

Platform and build breakdown:

• doc: 1
  • opt: 1
• android-4-1-armv7: 1
  • opt: 1
• android-5-0-x86: 1
  • opt: 1
• windows2012-64: 3
  • debug: 2
  • opt: 1
• android-4-1-x86: 1
  • opt: 1
• android-5-0-x86_64: 4
  • debug: 1
  • debug-isolated-process: 1
  • opt: 2
• linux64-noopt: 1
  • debug: 1
• linux64: 2
  • opt: 1
  • debug: 1
• windows2012-32: 3
  • debug: 2
  • opt: 1
• osx-cross: 3
  • opt: 1
  • debug: 2
• linux32: 1
  • debug: 1
• android-5-0-armv7: 1
  • debug: 1
• linux64-shippable: 1
  • opt: 1
• linux32-shippable: 1
  • opt: 1
• android-5-0-x86-shippable: 1
  • opt: 1
• android-4-1-x86-shippable-lite: 1
  • opt: 1
• windows2012-64-shippable: 1
  • opt: 1
• windows2012-32-shippable: 1
  • opt: 1
• android-5-0-x86_64-shippable: 1
  • opt: 1
• android-5-0-x86_64-shippable-lite: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1797891&startday=2022-10-27&endday=2022-10-27&tree=all

Comment 2

[Julien Cristau [:jcristau]]

Do you think these symbols need a dedicated bucket or could we lump them in with symbols from try?

Comment 3

[Mike Hommey [:glandium]]

Lumping them with try works for me.

Comment 4

[Intermittent Failures Robot]

30 failures in 3743 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #48 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• toolchains: 30

Platform and build breakdown:

• doc: 1
  • opt: 1
• android-4-1-armv7: 1
  • opt: 1
• android-5-0-x86: 1
  • opt: 1
• windows2012-64: 3
  • debug: 2
  • opt: 1
• android-4-1-x86: 1
  • opt: 1
• android-5-0-x86_64: 4
  • debug: 1
  • debug-isolated-process: 1
  • opt: 2
• linux64-noopt: 1
  • debug: 1
• linux64: 2
  • opt: 1
  • debug: 1
• windows2012-32: 3
  • debug: 2
  • opt: 1
• osx-cross: 3
  • opt: 1
  • debug: 2
• linux32: 1
  • debug: 1
• android-5-0-armv7: 1
  • debug: 1
• linux64-shippable: 1
  • opt: 1
• linux32-shippable: 1
  • opt: 1
• android-5-0-x86-shippable: 1
  • opt: 1
• android-4-1-x86-shippable-lite: 1
  • opt: 1
• windows2012-64-shippable: 1
  • opt: 1
• windows2012-32-shippable: 1
  • opt: 1
• android-5-0-x86_64-shippable: 1
  • opt: 1
• android-5-0-x86_64-shippable-lite: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1797891&startday=2022-10-24&endday=2022-10-30&tree=all

Comment 5

[Intermittent Failures Robot]

40 failures in 733 pushes (0.055 failures/push) were associated with this bug yesterday.

Repository breakdown:

• toolchains: 40

Platform and build breakdown:

• android-4-1-armv7: 2
  • opt: 2
• android-4-1-x86: 1
  • opt: 1
• android-5-0-x86: 1
  • opt: 1
• android-5-0-armv7: 2
  • opt: 1
  • debug: 1
• android-5-0-x86_64: 4
  • opt: 2
  • debug-isolated-process: 1
  • debug: 1
• windows2012-aarch64: 3
  • opt: 2
  • debug: 1
• linux64-noopt: 1
  • debug: 1
• osx-cross: 3
  • debug: 2
  • opt: 1
• windows2012-64: 3
  • debug: 2
  • opt: 1
• windows2012-32: 3
  • debug: 2
  • opt: 1
• linux64: 2
  • debug: 1
  • opt: 1
• linux32: 1
  • debug: 1
• android-4-1-armv7-shippable-lite: 1
  • opt: 1
• linux64-aarch64: 1
  • opt: 1
• linux64-shippable: 1
  • opt: 1
• linux32-shippable: 1
  • opt: 1
• android-4-1-x86-shippable-lite: 1
  • opt: 1
• android-5-0-x86-shippable: 1
  • opt: 1
• android-5-0-aarch64-shippable: 1
  • opt: 1
• android-5-0-x86_64-shippable-lite: 1
  • opt: 1
• osx-aarch64-shippable: 1
  • opt: 1
• android-5-0-aarch64-shippable-lite: 1
  • opt: 1
• osx-shippable: 1
  • opt: 1
• android-5-0-x86_64-shippable: 1
  • opt: 1
• windows2012-32-shippable: 1
  • opt: 1
• windows2012-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1797891&startday=2022-11-02&endday=2022-11-02&tree=all

Comment 6

[Intermittent Failures Robot]

40 failures in 3626 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #39 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• toolchains: 40

Platform and build breakdown:

• android-4-1-armv7: 2
  • opt: 2
• android-4-1-x86: 1
  • opt: 1
• android-5-0-x86: 1
  • opt: 1
• android-5-0-armv7: 2
  • opt: 1
  • debug: 1
• android-5-0-x86_64: 4
  • opt: 2
  • debug-isolated-process: 1
  • debug: 1
• windows2012-aarch64: 3
  • opt: 2
  • debug: 1
• linux64-noopt: 1
  • debug: 1
• osx-cross: 3
  • debug: 2
  • opt: 1
• windows2012-64: 3
  • debug: 2
  • opt: 1
• windows2012-32: 3
  • debug: 2
  • opt: 1
• linux64: 2
  • debug: 1
  • opt: 1
• linux32: 1
  • debug: 1
• android-4-1-armv7-shippable-lite: 1
  • opt: 1
• linux64-aarch64: 1
  • opt: 1
• linux64-shippable: 1
  • opt: 1
• linux32-shippable: 1
  • opt: 1
• android-4-1-x86-shippable-lite: 1
  • opt: 1
• android-5-0-x86-shippable: 1
  • opt: 1
• android-5-0-aarch64-shippable: 1
  • opt: 1
• android-5-0-x86_64-shippable-lite: 1
  • opt: 1
• osx-aarch64-shippable: 1
  • opt: 1
• android-5-0-aarch64-shippable-lite: 1
  • opt: 1
• osx-shippable: 1
  • opt: 1
• android-5-0-x86_64-shippable: 1
  • opt: 1
• windows2012-32-shippable: 1
  • opt: 1
• windows2012-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1797891&startday=2022-10-31&endday=2022-11-06&tree=all

Comment 7

[Julien Cristau [:jcristau]]

project/releng/gecko/build/level-2/gecko-symbol-upload secret created.

Comment 8

[Mike Hommey [:glandium]]

Thanks, the symbol tasks now don't fail. That leaves us with build-android-geckoview-docs/opt and source-test-doc-upload.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:ahal, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 10

[Mike Hommey [:glandium]]

Attachment: Bug 1797891 - Make some tasks really only run on m-c or autoland.

As of bug 1791454 m-c and autoland both also imply the toolchains
branch, which doesn't work for every kind of task, some of which are
really meant to only run on those branches.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/aa124b39d3e5
Make some tasks really only run on m-c or autoland. r=jcristau

Comment 12

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/aa124b39d3e5