1816042 - Perma perftest-chrome [tier 2] json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Status: RESOLVED INVALID

(Testing :: mozperftest, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ctuns [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=405229792&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PGONFl2JS4aHdusPxfFaYA/runs/0/artifacts/public/logs/live_backing.log

---

[fetches 2023-02-10T05:31:34.407Z] /home/cltbld/tasks/task_167600703465090/fetches/target.perftests.tests.tar.gz extracted in 1.219s
[fetches 2023-02-10T05:31:34.407Z] Removing /home/cltbld/tasks/task_167600703465090/fetches/target.perftests.tests.tar.gz
[fetches 2023-02-10T05:31:34.776Z] /home/cltbld/tasks/task_167600703465090/fetches/target.common.tests.tar.gz extracted in 1.769s
[fetches 2023-02-10T05:31:34.776Z] Removing /home/cltbld/tasks/task_167600703465090/fetches/target.common.tests.tar.gz
[fetches 2023-02-10T05:31:42.937Z] /home/cltbld/tasks/task_167600703465090/fetches/target.tar.bz2 extracted in 9.644s
[fetches 2023-02-10T05:31:42.937Z] Removing /home/cltbld/tasks/task_167600703465090/fetches/target.tar.bz2
[fetches 2023-02-10T05:31:42.948Z] PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 13.615748928000016, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2023-02-10T05:31:42.971Z] finished fetching artifacts
[task 2023-02-10T05:31:42.971Z] executing ['bash', '-cx', 'mkdir -p $MOZ_FETCHES_DIR/../artifacts && cd $MOZ_FETCHES_DIR && python3.8 python/mozperftest/mozperftest/runner.py testing/performance/perftest_WPT_chrome_init_file.js --flavor=webpagetest --webpagetest --perfherder --perfherder-metrics median --perfherder-app=chrome --output $MOZ_FETCHES_DIR/../artifacts']
[task 2023-02-10T05:31:42.973Z] + mkdir -p /home/cltbld/tasks/task_167600703465090/fetches/../artifacts
[task 2023-02-10T05:31:42.974Z] + cd /home/cltbld/tasks/task_167600703465090/fetches
[task 2023-02-10T05:31:42.974Z] + python3.8 python/mozperftest/mozperftest/runner.py testing/performance/perftest_WPT_chrome_init_file.js --flavor=webpagetest --webpagetest --perfherder --perfherder-metrics median --perfherder-app=chrome --output /home/cltbld/tasks/task_167600703465090/fetches/../artifacts
[task 2023-02-10T05:31:45.697Z]  0:01.64 Created profile at /tmp/tmpnj6uskki.mozrunner
[task 2023-02-10T05:31:46.205Z]  0:02.15 Test queue at ec2-us-east-1(Virginia - EC2) is 36
[task 2023-02-10T05:31:46.205Z]  0:02.15 Testing: google.com
[task 2023-02-10T05:31:46.208Z]  0:02.15 Testing: youtube.com
[task 2023-02-10T05:31:46.208Z]  0:02.15 Testing: facebook.com
[task 2023-02-10T05:31:46.211Z]  0:02.16 Testing: qq.com
[task 2023-02-10T05:31:46.218Z]  0:02.16 Testing: baidu.com
[task 2023-02-10T05:31:46.221Z]  0:02.17 Testing: sohu.com
[task 2023-02-10T05:31:46.226Z]  0:02.17 Testing: 360.cn
[task 2023-02-10T05:31:46.229Z]  0:02.17 Testing: jd.com
[task 2023-02-10T05:31:46.232Z]  0:02.18 Testing: amazon.com
[task 2023-02-10T05:31:46.233Z]  0:02.18 Testing: yahoo.com
[task 2023-02-10T05:31:46.234Z]  0:02.18 Testing: zoom.us
[task 2023-02-10T05:31:46.236Z]  0:02.18 Testing: weibo.com
[task 2023-02-10T05:31:46.245Z]  0:02.19 Testing: sina.com.cn
[task 2023-02-10T05:31:46.248Z]  0:02.19 Testing: live.com
[task 2023-02-10T05:31:46.251Z]  0:02.20 Testing: reddit.com
[task 2023-02-10T05:31:46.256Z]  0:02.20 Testing: netflix.com
[task 2023-02-10T05:31:46.258Z]  0:02.20 Testing: microsoft.com
[task 2023-02-10T05:31:46.259Z]  0:02.20 Testing: instagram.com
[task 2023-02-10T05:31:46.262Z]  0:02.21 Testing: google.com.hk
[task 2023-02-10T05:31:46.267Z]  0:02.21 Testing: csdn.net
[task 2023-02-10T05:31:46.269Z]  0:02.21 Testing: bing.com
[task 2023-02-10T05:31:46.271Z]  0:02.22 Testing: vk.com
[task 2023-02-10T05:31:46.275Z]  0:02.22 Testing: yahoo.co.jp
[task 2023-02-10T05:31:46.278Z]  0:02.22 Testing: twitter.com
[task 2023-02-10T05:31:46.281Z]  0:02.23 Testing: naver.com
[task 2023-02-10T05:31:46.287Z]  0:02.23 Testing: canva.com
[task 2023-02-10T05:31:46.290Z]  0:02.23 Testing: ebay.com
[task 2023-02-10T05:31:46.295Z]  0:02.24 Testing: force.com
[task 2023-02-10T05:31:46.299Z]  0:02.24 Testing: amazon.in
[task 2023-02-10T05:31:46.302Z]  0:02.25 Testing: adobe.com
[task 2023-02-10T05:31:46.305Z]  0:02.25 Testing: aliexpress.com
[task 2023-02-10T05:31:46.309Z]  0:02.25 Testing: linkedin.com
[task 2023-02-10T05:31:46.313Z]  0:02.26 Testing: tianya.cn
[task 2023-02-10T05:31:46.318Z]  0:02.26 Testing: yy.com
[task 2023-02-10T05:31:46.319Z]  0:02.26 Testing: huanqiu.com
[task 2023-02-10T05:31:46.321Z]  0:02.27 Testing: amazon.co.jp
[task 2023-02-10T05:31:46.323Z]  0:02.27 Testing: okezone.com
[task 2023-02-10T05:31:49.422Z] Site not up-to-date reason: "/home/cltbld/tasks/task_167600703465090/fetches/obj-x86_64-pc-linux-gnu/_virtualenvs/common" does not exist
[task 2023-02-10T05:31:49.426Z] Traceback (most recent call last):
[task 2023-02-10T05:31:49.426Z]   File "python/mozperftest/mozperftest/runner.py", line 283, in <module>
[task 2023-02-10T05:31:49.426Z]     sys.exit(main())
[task 2023-02-10T05:31:49.426Z]   File "python/mozperftest/mozperftest/runner.py", line 279, in main
[task 2023-02-10T05:31:49.426Z]     run_tests(mach_cmd, args, user_args)
[task 2023-02-10T05:31:49.426Z]   File "python/mozperftest/mozperftest/runner.py", line 168, in run_tests
[task 2023-02-10T05:31:49.426Z]     e.run(metadata)
[task 2023-02-10T05:31:49.426Z]   File "/home/cltbld/tasks/task_167600703465090/fetches/python/mozperftest/mozperftest/environment.py", line 92, in run
[task 2023-02-10T05:31:49.426Z]     metadata = testlayer(syslayer(metadata))
[task 2023-02-10T05:31:49.426Z]   File "/home/cltbld/tasks/task_167600703465090/fetches/python/mozperftest/mozperftest/layers.py", line 157, in __call__
[task 2023-02-10T05:31:49.426Z]     metadata = layer(metadata)
[task 2023-02-10T05:31:49.426Z]   File "/home/cltbld/tasks/task_167600703465090/fetches/python/mozperftest/mozperftest/layers.py", line 69, in __call__
[task 2023-02-10T05:31:49.426Z]     metadata = self.run(metadata)
[task 2023-02-10T05:31:49.426Z]   File "/home/cltbld/tasks/task_167600703465090/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 195, in run
[task 2023-02-10T05:31:49.426Z]     self.create_and_run_wpt_threaded_tests(test_list, metadata)
[task 2023-02-10T05:31:49.426Z]   File "/home/cltbld/tasks/task_167600703465090/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 294, in create_and_run_wpt_threaded_tests
[task 2023-02-10T05:31:49.426Z]     thread.join()
[task 2023-02-10T05:31:49.426Z]   File "/home/cltbld/tasks/task_167600703465090/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 137, in join
[task 2023-02-10T05:31:49.426Z]     raise self.exc
[task 2023-02-10T05:31:49.426Z]   File "/home/cltbld/tasks/task_167600703465090/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 130, in run
[task 2023-02-10T05:31:49.426Z]     self._target(*self._args, **self._kwargs)
[task 2023-02-10T05:31:49.426Z]   File "/home/cltbld/tasks/task_167600703465090/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 297, in create_and_run_wpt_tests
[task 2023-02-10T05:31:49.426Z]     wpt_run = self.get_WPT_results(
[task 2023-02-10T05:31:49.426Z]   File "/home/cltbld/tasks/task_167600703465090/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 311, in get_WPT_results
[task 2023-02-10T05:31:49.426Z]     results_of_test = self.request_with_timeout(send_wpt_test_request)
[task 2023-02-10T05:31:49.426Z]   File "/home/cltbld/tasks/task_167600703465090/fetches/python/mozperftest/mozperftest/test/webpagetest.py", line 256, in request_with_timeout
[task 2023-02-10T05:31:49.426Z]     results_of_request = json.loads(requested_results.text)
[task 2023-02-10T05:31:49.426Z]   File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
[task 2023-02-10T05:31:49.426Z]     return _default_decoder.decode(s)
[task 2023-02-10T05:31:49.426Z]   File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
[task 2023-02-10T05:31:49.426Z]     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
[task 2023-02-10T05:31:49.426Z]   File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
[task 2023-02-10T05:31:49.426Z]     raise JSONDecodeError("Expecting value", s, err.value) from None
[task 2023-02-10T05:31:49.426Z] json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[taskcluster 2023-02-10T05:31:50.018Z]    Exit Code: 1
[taskcluster 2023-02-10T05:31:50.018Z]    User Time: 18.229344s
[taskcluster 2023-02-10T05:31:50.018Z]  Kernel Time: 2.837587s
[taskcluster 2023-02-10T05:31:50.018Z]    Wall Time: 20.901875693s
[taskcluster 2023-02-10T05:31:50.018Z]       Result: FAILED
[taskcluster 2023-02-10T05:31:50.018Z] === Task Finished ===
[taskcluster 2023-02-10T05:31:50.018Z] Task Duration: 20.905143715s
[taskcluster 2023-02-10T05:31:50.083Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2023-05-11T04:02:36.614Z
[taskcluster:error] exit status 1

Comment 1

[Cristian Tuns]

Hi, can you please take a look at this? I think it might be caused by Bug 1678629 but not sure. Thanks!

Comment 2

[Daisuke Akatsuka (:daisuke)]

Hi Cristian! Thank you very much for the report.

However, unfortunately, I threw to the try without my patches, it seems that the failure still happens.
https://treeherder.mozilla.org/jobs?repo=try&revision=bbdf40330db71a42ceac2a09af888bbe32647671
And its changeset is https://hg.mozilla.org/try/rev/1f471cdfbd2342f7e82936f2a1f64331ec0a524c
(That changeset is pushed as the parent of my patches.)
So, I think that there is another cause. Could you confirm whether I am not wrong?

Comment 3

[Natalia Csoregi [:nataliaCs]]

The most recent changeset related to WebPageTest is this: https://hg.mozilla.org/mozilla-central/rev/e2325123dc9724fefc8184a4538280670f3f2500
At retriggers and backfills, it fails even where the job has previously passed.
Alex, is there anyone you can assign this to?
We could not find an exact cause.
Thank you.

Comment 4

[Cosmin Sabou [:CosminS]]

This looks to be an off-tree issue, it happens when trying to reach okezone.com - Site not up-to-date reason: "/home/cltbld/tasks/task_167609136627231/fetches/obj-x86_64-pc-linux-gnu/_virtualenvs/common" does not exist
Andrej, please have a look over this one. Thank you.

Comment 5

[Intermittent Failures Robot]

16 failures in 3766 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 16

Platform and build breakdown:

• linux1804-64-shippable: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1816042&startday=2023-02-06&endday=2023-02-12&tree=all

Comment 6

[Andrej (:aglavic)]

Attachment: Bug 1816042 - WPT permafail JSONDecodeError. r?#perftest

Comment 7

[Intermittent Failures Robot]

15 failures in 3856 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 15

Platform and build breakdown:

• linux1804-64-shippable: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1816042&startday=2023-02-13&endday=2023-02-19&tree=all

Comment 8

[Andrej (:aglavic)]

The error appears to be a result of WPT introducing a private variable and the code checking the remaining tests left this month feature failing, patch is in place but am debugging the latter

Comment 9

[Intermittent Failures Robot]

16 failures in 3643 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 16

Platform and build breakdown:

• linux1804-64-shippable: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1816042&startday=2023-02-20&endday=2023-02-26&tree=all

Comment 10

[Intermittent Failures Robot]

14 failures in 3752 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 14

Platform and build breakdown:

• linux1804-64-shippable: 14
  • opt: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1816042&startday=2023-02-27&endday=2023-03-05&tree=all

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0ffdb58e01c8
WPT permafail JSONDecodeError. r=perftest-reviewers,AlexandruIonescu,sparky

Comment 12

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/0ffdb58e01c8

Comment 13

[Cosmin Sabou [:CosminS]]

This is still failing on the latest merge here.
Failure log: https://treeherder.mozilla.org/logviewer?job_id=408245124&repo=mozilla-central

Comment 14

[Intermittent Failures Robot]

4 failures in 3966 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform and build breakdown:

• linux1804-64-shippable: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1816042&startday=2023-03-06&endday=2023-03-12&tree=all

Comment 15

[Andrej (:aglavic)]

Latest merge did not include the change made to resolve the issue, fails have gone to 0 since