1681267 - Permanent 'controlled' task: Exception: WEBNETEM_KEY not set

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=323858176&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Wv_J_p98SJSyLbcA9T-EwQ/runs/0/artifacts/public/logs/live_backing.log

---

The task fails since it got added in bug 1672142.

[task 2020-12-08T05:38:51.728Z] + python3.exe python/mozperftest/mozperftest/runner.py netwerk/test/perf/perftest_http3_controlled.js --browsertime-binary C:/Users/task_1607393965/fetches/firefox/firefox.exe --browsertime-iterations 1 --browsertime-cycles 96 --hooks netwerk/test/perf/hooks_throttling.py --flavor desktop-browser --perfherder --perfherder-metrics name:navigationTiming,unit:ms name:pageTimings,unit:ms name:resources,unit:ms name:firstPaint,unit:ms name:timeToContentfulPaint,unit:ms --perfherder-simplify-names --browsertime-geckodriver C:/Users/task_1607393965/fetches/geckodriver.exe --browsertime-node C:/Users/task_1607393965/fetches/node/node.exe --output C:/Users/task_1607393965/fetches/../artifacts
[task 2020-12-08T05:38:53.186Z] Traceback (most recent call last):
[task 2020-12-08T05:38:53.186Z]   File "python/mozperftest/mozperftest/runner.py", line 220, in <module>
[task 2020-12-08T05:38:53.186Z]     sys.exit(main())
[task 2020-12-08T05:38:53.186Z]   File "python/mozperftest/mozperftest/runner.py", line 216, in main
[task 2020-12-08T05:38:53.186Z]     run_tests(mach_cmd, args, user_args)
[task 2020-12-08T05:38:53.186Z]   File "python/mozperftest/mozperftest/runner.py", line 114, in run_tests
[task 2020-12-08T05:38:53.186Z]     hooks = Hooks(mach_cmd, hooks_file)
[task 2020-12-08T05:38:53.186Z]   File "C:\Users\task_1607393965\fetches\python\mozperftest\mozperftest\hooks.py", line 36, in __init__
[task 2020-12-08T05:38:53.186Z]     spec.loader.exec_module(hook_module)
[task 2020-12-08T05:38:53.186Z]   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
[task 2020-12-08T05:38:53.186Z]   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[task 2020-12-08T05:38:53.186Z]   File "netwerk\test\perf\hooks_throttling.py", line 34, in <module>
[task 2020-12-08T05:38:53.186Z]     raise Exception("WEBNETEM_KEY not set")
[task 2020-12-08T05:38:53.186Z] Exception: WEBNETEM_KEY not set
[taskcluster 2020-12-08T05:38:53.301Z]    Exit Code: 1```

Comment 1

[Tarek Ziadé (:tarek)]

Are you sure it's permanent? I was able to make it work on try. This bug happens if the Task Cluster environment can't get the secrets

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Yes, it's permanent.

Comment 3

[Tarek Ziadé (:tarek)]

https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=350e8dd638af500f8429bbf379bd9301039f9170

Comment 4

[Intermittent Failures Robot]

1 failures in 4060 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• win64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1681267&startday=2020-12-07&endday=2020-12-13&tree=all

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Tarek Ziadé (:tarek) from comment #1)

Are you sure it's permanent? I was able to make it work on try. This bug happens if the Task Cluster environment can't get the secrets

It does still appear to be perma-failing.

Comment 6

[Tarek Ziadé (:tarek)]

but it works on try. I don't know how to reproduce/fix this. maybe we need to get some help from the CI team to understand

Comment 7

[Tarek Ziadé (:tarek)]

Attachment: Bug 1681267 - Improve error message when the hook can't get the webnetem secret.

Improves the error message when the hook can't get the secret

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:jmaher can you help with troubleshooting this failure?

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

I have no idea where to begin, I will start with :aki, and if that fails, check with :dustin

Comment 10

[Aki Sasaki (not active)]

The level 1 secret, project/releng/gecko/build/level-1/conditioned-profiles, contains 12 key/value pairs including throttler_key. This is the secret we use on level 1 branches, e.g. Try.

The level 3 secret, project/releng/gecko/build/level-3/conditioned-profiles, contains 2 key/value pairs. Just username and password. This is the secret we use on level 3 branches, e.g. Autoland and Mozilla-Central.

We need to populate the level 3 secret.

Comment 11

[Tarek Ziadé (:tarek)]

oh thanks Aki !

Doing this righht now

Comment 12

[Tarek Ziadé (:tarek)]

Done, hopefully the next run will be OK Thanks Aki and Joel

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:tarek, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 14

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Release mgmt bot [:sylvestre / :calixte / :marco for bugbug] from comment #13)

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:tarek, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b81848b20b1e
Improve error message when the hook can't get the webnetem secret. r=davehunt

Comment 16

[Greg Mierzwinski [:sparky]]

:aki, I'm not able to check or modify the level-3 secrets. Would you be able to check if throttler_key is in the level-3 one and if not could you add it to there?

Comment 17

[Aki Sasaki (not active)]

Both secrets appear to have the same information in them.
Is this still broken? I was thinking this bug was fixed as of comment 12.

Comment 18

[Greg Mierzwinski [:sparky]]

Oh! It's failing for a different reason now - something about the throttling server itself. Thanks for double-checking, we can close this bug now.

Comment 19

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/b81848b20b1e

Comment 20

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Which bug tracks the permanent task failures now?

Comment 21

[Greg Mierzwinski [:sparky]]

I've restarted the 3 services today (windows, mac, and linux) just now and the tests are working locally for me now - it looks like they were stuck. They should be green in tomorrow's run.

Comment 22

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

All 'controlled' tasks are failing. Windows executes tests but times out after 3h, Linux and macOS fail to acquire the test server:

Comment 23

[Greg Mierzwinski [:sparky]]

I've filed another bug for these ones.