1800814 - Perma [tier 2] mozperftest.test.webpagetest.WPTExpiredAPIKeyError: The API key has expired | KeyError: 'statusText'

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nfay [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=396687259&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Mvl33NzASKGGnidintR3_Q/runs/0/artifacts/public/logs/live_backing.log

---

[task 2022-11-16T05:34:19.341Z] executing ['bash', '-cx', 'mkdir -p $MOZ_FETCHES_DIR/../artifacts && cd $MOZ_FETCHES_DIR && python3.8 python/mozperftest/mozperftest/runner.py testing/performance/perftest_WPT_firefox_init_file.js --flavor=webpagetest --webpagetest --perfherder --perfherder-metrics median --perfherder-app=firefox --output $MOZ_FETCHES_DIR/../artifacts']
[task 2022-11-16T05:34:19.342Z] + mkdir -p /home/cltbld/tasks/task_166857684440160/fetches/../artifacts
[task 2022-11-16T05:34:19.343Z] + cd /home/cltbld/tasks/task_166857684440160/fetches
[task 2022-11-16T05:34:19.343Z] + python3.8 python/mozperftest/mozperftest/runner.py testing/performance/perftest_WPT_firefox_init_file.js --flavor=webpagetest --webpagetest --perfherder --perfherder-metrics median --perfherder-app=firefox --output /home/cltbld/tasks/task_166857684440160/fetches/../artifacts
[task 2022-11-16T05:34:20.861Z]  0:00.75 Created profile at /tmp/tmpe9huv9i2.mozrunner
[task 2022-11-16T05:34:21.413Z]  0:01.30 Test queue at ec2-us-east-1(Virginia - EC2) is 61
[task 2022-11-16T05:34:21.414Z]  0:01.30 Testing: google.com
[task 2022-11-16T05:34:21.415Z]  0:01.30 Testing: youtube.com
[task 2022-11-16T05:34:21.417Z]  0:01.30 Testing: facebook.com
[task 2022-11-16T05:34:21.418Z]  0:01.30 Testing: qq.com
[task 2022-11-16T05:34:21.418Z]  0:01.30 Testing: baidu.com
[task 2022-11-16T05:34:21.420Z]  0:01.31 Testing: sohu.com
[task 2022-11-16T05:34:21.423Z]  0:01.31 Testing: 360.cn
[task 2022-11-16T05:34:21.423Z]  0:01.31 Testing: jd.com
[task 2022-11-16T05:34:21.424Z]  0:01.31 Testing: amazon.com
[task 2022-11-16T05:34:21.427Z]  0:01.31 Testing: yahoo.com
[task 2022-11-16T05:34:21.432Z]  0:01.32 Testing: zoom.us
[task 2022-11-16T05:34:21.436Z]  0:01.32 Testing: weibo.com
[task 2022-11-16T05:34:21.438Z]  0:01.32 Testing: sina.com.cn
[task 2022-11-16T05:34:21.439Z]  0:01.32 Testing: live.com
[task 2022-11-16T05:34:21.441Z]  0:01.33 Testing: reddit.com
[task 2022-11-16T05:34:21.443Z]  0:01.33 Testing: netflix.com
[task 2022-11-16T05:34:21.446Z]  0:01.33 Testing: microsoft.com
[task 2022-11-16T05:34:21.450Z]  0:01.33 Testing: instagram.com
[task 2022-11-16T05:34:21.451Z]  0:01.34 Testing: google.com.hk
[task 2022-11-16T05:34:21.454Z]  0:01.34 Testing: csdn.net
[task 2022-11-16T05:34:21.455Z]  0:01.34 Testing: bing.com
[task 2022-11-16T05:34:21.455Z]  0:01.34 Testing: vk.com
[task 2022-11-16T05:34:21.458Z]  0:01.34 Testing: yahoo.co.jp
[task 2022-11-16T05:34:21.463Z]  0:01.35 Testing: twitter.com
[task 2022-11-16T05:34:21.465Z]  0:01.35 Testing: naver.com
[task 2022-11-16T05:34:21.468Z]  0:01.35 Testing: canva.com
[task 2022-11-16T05:34:21.470Z]  0:01.36 Testing: ebay.com
[task 2022-11-16T05:34:21.473Z]  0:01.36 Testing: force.com
[task 2022-11-16T05:34:21.474Z]  0:01.36 Testing: amazon.in
[task 2022-11-16T05:34:21.477Z]  0:01.36 Testing: adobe.com
[task 2022-11-16T05:34:21.480Z]  0:01.37 Testing: aliexpress.com
[task 2022-11-16T05:34:21.482Z]  0:01.37 Testing: linkedin.com
[task 2022-11-16T05:34:21.484Z]  0:01.37 Testing: tianya.cn
[task 2022-11-16T05:34:21.486Z]  0:01.37 Testing: yy.com
[task 2022-11-16T05:34:21.489Z]  0:01.37 Testing: huanqiu.com
[task 2022-11-16T05:34:21.491Z]  0:01.38 Testing: amazon.co.jp
[task 2022-11-16T05:34:21.493Z]  0:01.38 Testing: okezone.com
[task 2022-11-16T05:34:22.045Z] Site not up-to-date reason: "/home/cltbld/tasks/task_166857684440160/fetches/obj-x86_64-pc-linux-gnu/_virtualenvs/common" does not exist
[task 2022-11-16T05:34:22.089Z] Traceback (most recent call last):
[task 2022-11-16T05:34:22.089Z]   File "python/mozperftest/mozperftest/runner.py", line 280, in <module>
[task 2022-11-16T05:34:22.089Z]     sys.exit(main())
[task 2022-11-16T05:34:22.089Z]   File "python/mozperftest/mozperftest/runner.py", line 276, in main
[task 2022-11-16T05:34:22.089Z]     run_tests(mach_cmd, args, user_args)
[task 2022-11-16T05:34:22.089Z]   File "python/mozperftest/mozperftest/runner.py", line 168, in run_tests
[task 2022-11-16T05:34:22.089Z]     e.run(metadata)
[task 2022-11-16T05:34:22.089Z]   File "/home/cltbld/tasks/task_166857684440160/fetches/python/mozperftest/mozperftest/environment.py", line 93, in run
[task 2022-11-16T05:34:22.089Z]     metadata = testlayer(syslayer(metadata))
[task 2022-11-16T05:34:22.089Z]   File "/home/cltbld/tasks/task_166857684440160/fetches/python/mozperftest/mozperftest/layers.py", line 156, in __call__
[task 2022-11-16T05:34:22.089Z]     metadata = layer(metadata)
[task 2022-11-16T05:34:22.089Z]   File "/home/cltbld/tasks/task_166857684440160/fetches/python/mozperftest/mozperftest/layers.py", line 68, in __call__
[task 2022-11-16T05:34:22.089Z]     metadata = self.run(metadata)
[task 2022-11-16T05:34:22.089Z]   File "/home/cltbld/tasks/task_166857684440160/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 193, in run
[task 2022-11-16T05:34:22.089Z]     self.create_and_run_wpt_threaded_tests(test_list, metadata)
[task 2022-11-16T05:34:22.089Z]   File "/home/cltbld/tasks/task_166857684440160/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 289, in create_and_run_wpt_threaded_tests
[task 2022-11-16T05:34:22.089Z]     thread.join()
[task 2022-11-16T05:34:22.089Z]   File "/home/cltbld/tasks/task_166857684440160/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 135, in join
[task 2022-11-16T05:34:22.089Z]     raise self.exc
[task 2022-11-16T05:34:22.089Z]   File "/home/cltbld/tasks/task_166857684440160/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 128, in run
[task 2022-11-16T05:34:22.089Z]     self._target(*self._args, **self._kwargs)
[task 2022-11-16T05:34:22.089Z]   File "/home/cltbld/tasks/task_166857684440160/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 292, in create_and_run_wpt_tests
[task 2022-11-16T05:34:22.089Z]     wpt_run = self.get_WPT_results(
[task 2022-11-16T05:34:22.089Z]   File "/home/cltbld/tasks/task_166857684440160/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 303, in get_WPT_results
[task 2022-11-16T05:34:22.089Z]     send_wpt_test_request = self.request_with_timeout(wpt_test_request_link)[
[task 2022-11-16T05:34:22.089Z]   File "/home/cltbld/tasks/task_166857684440160/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 257, in request_with_timeout
[task 2022-11-16T05:34:22.089Z]     raise WPTExpiredAPIKeyError("The API key has expired")
[task 2022-11-16T05:34:22.089Z] mozperftest.test.webpagetest.WPTExpiredAPIKeyError: The API key has expired
[taskcluster 2022-11-16T05:34:23.115Z]    Exit Code: 1
[taskcluster 2022-11-16T05:34:23.115Z]    User Time: 15.096396s
[taskcluster 2022-11-16T05:34:23.115Z]  Kernel Time: 2.208934s
[taskcluster 2022-11-16T05:34:23.115Z]    Wall Time: 17.163929112s
[taskcluster 2022-11-16T05:34:23.115Z]       Result: FAILED
[taskcluster 2022-11-16T05:34:23.115Z] === Task Finished ===
[taskcluster 2022-11-16T05:34:23.115Z] Task Duration: 17.165182937s
[taskcluster 2022-11-16T05:34:23.180Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Mvl33NzASKGGnidintR3_Q/runs/0/artifacts/public%2Flogs%2Flive_backing.log with mime type "text/plain; charset=utf-8" and expiry 2023-02-14T04:06:21.129Z
[taskcluster:error] exit status 1

Comment 1

[Norisz Fay [:noriszfay]]

Hello, this seems to be permafailing as tier 2 on central and it might be regressed by Bug 1796780. Could you take a look? Thank you!

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1796780

Comment 3

[Andrej (:aglavic)]

Hi Norisz,
Thank you for letting me know about the failure, Bug 1796780 gave us a more descriptive error message so we know what is going on when we have a failure, this job was failing before because the API key has expired. We are in talks with the vendor to get an updated API key, I'll assign the issue to myself until we stop seeing these errors.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Andrej (:andrej) from comment #3)

Hi Norisz,
Thank you for letting me know about the failure, Bug 1796780 gave us a more descriptive error message so we know what is going on when we have a failure, this job was failing before because the API key has expired. We are in talks with the vendor to get an updated API key, I'll assign the issue to myself until we stop seeing these errors.

It also now takes 1 minute to fail, rather than 180 minutes! \o/

@andrej I'm told this has all gone through, but I'm having an issue logging into the account, and the API key still appears to be expired. I'll reach out to Catchpoint for assistance.

Comment 5

[Andrej (:aglavic)]

I am able to login but I can't seem to go to the account page where the API key was previously:

Comment 6

[Cristian Tuns]

It looks like the first line got fixed somehow, but there is still the second one: https://treeherder.mozilla.org/logviewer?job_id=397045295&repo=mozilla-central. Can someone please take a look?
https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&searchStr=linux%2C18.04%2Cx64%2Cshippable%2Copt%2Cperformance%2Ctests%2Cusing%2Cchrone%2Cperftest-linux-webpagetest-chrome%2Clinux-webpagetest&revision=3b5a8f67189bd6549f0da19ea5da4a53f7e5c79a&selectedTaskRun=BYlgNfyYQ0KWz3bWpHpSYg.0

Comment 7

[Cosmin Sabou [:CosminS]]

This is green on latest run of these tests. Let's call this fixed and we'll reopen if we hit it again.

Comment 8

[Intermittent Failures Robot]

28 failures in 3589 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 28

Platform and build breakdown:

• linux1804-64-shippable: 28
  • opt: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800814&startday=2022-11-14&endday=2022-11-20&tree=all