1810037 - Frequent [tier 2] test-info all json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

Status: RESOLVED FIXED

(Testing :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nfay [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=402235619&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/c9QB3_FrTm-19lwOD5QJ0A/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-01-12T21:40:25.700Z] executing ['bash', '-cx', 'source taskcluster/scripts/misc/source-test-common.sh && ./mach test-info testrun-report --output-file /builds/worker/artifacts/test-run-info.json && ./mach test-info report --show-tests --show-summary --show-testruns --verbose --output-file /builds/worker/artifacts/test-info-all-tests.json && ./mach test-info report --show-annotations --output-file /builds/worker/artifacts/test-info-manifest-conditions.json']in /builds/worker/checkouts/gecko
[task 2023-01-12T21:40:25.704Z] + source taskcluster/scripts/misc/source-test-common.sh
[task 2023-01-12T21:40:25.704Z] ++ set -x -e
[task 2023-01-12T21:40:25.704Z] ++ export MOZBUILD_STATE_PATH=/builds/worker/workspace
[task 2023-01-12T21:40:25.704Z] ++ MOZBUILD_STATE_PATH=/builds/worker/workspace
[task 2023-01-12T21:40:25.705Z] ++ export PATH=/builds/worker/fetches/clang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2023-01-12T21:40:25.705Z] ++ PATH=/builds/worker/fetches/clang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2023-01-12T21:40:25.705Z] ++ export PATH=/builds/worker/fetches/rustc/bin:/builds/worker/fetches/clang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2023-01-12T21:40:25.705Z] ++ PATH=/builds/worker/fetches/rustc/bin:/builds/worker/fetches/clang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2023-01-12T21:40:25.705Z] ++ export PATH=/builds/worker/fetches/cbindgen:/builds/worker/fetches/rustc/bin:/builds/worker/fetches/clang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2023-01-12T21:40:25.705Z] ++ PATH=/builds/worker/fetches/cbindgen:/builds/worker/fetches/rustc/bin:/builds/worker/fetches/clang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2023-01-12T21:40:25.705Z] ++ export PATH=/builds/worker/fetches/nasm:/builds/worker/fetches/cbindgen:/builds/worker/fetches/rustc/bin:/builds/worker/fetches/clang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2023-01-12T21:40:25.706Z] ++ PATH=/builds/worker/fetches/nasm:/builds/worker/fetches/cbindgen:/builds/worker/fetches/rustc/bin:/builds/worker/fetches/clang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2023-01-12T21:40:25.706Z] ++ export PATH=/builds/worker/fetches/node/bin:/builds/worker/fetches/nasm:/builds/worker/fetches/cbindgen:/builds/worker/fetches/rustc/bin:/builds/worker/fetches/clang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2023-01-12T21:40:25.706Z] ++ PATH=/builds/worker/fetches/node/bin:/builds/worker/fetches/nasm:/builds/worker/fetches/cbindgen:/builds/worker/fetches/rustc/bin:/builds/worker/fetches/clang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2023-01-12T21:40:25.706Z] ++ export CC=/builds/worker/fetches/clang/bin/clang
[task 2023-01-12T21:40:25.706Z] ++ CC=/builds/worker/fetches/clang/bin/clang
[task 2023-01-12T21:40:25.706Z] ++ export CXX=/builds/worker/fetches/clang/bin/clang++
[task 2023-01-12T21:40:25.706Z] ++ CXX=/builds/worker/fetches/clang/bin/clang++
[task 2023-01-12T21:40:25.706Z] + ./mach test-info testrun-report --output-file /builds/worker/artifacts/test-run-info.json
[task 2023-01-12T21:41:29.036Z] Error running mach:
[task 2023-01-12T21:41:29.036Z] 
[task 2023-01-12T21:41:29.036Z]     ['test-info', 'testrun-report', '--output-file', '/builds/worker/artifacts/test-run-info.json']
[task 2023-01-12T21:41:29.036Z] 
[task 2023-01-12T21:41:29.036Z] The error occurred in code that was called by the mach command. This is either
[task 2023-01-12T21:41:29.036Z] a bug in the called code itself or in the way that mach is calling it.
[task 2023-01-12T21:41:29.036Z] You can invoke |./mach busted| to check if this issue is already on file. If it
[task 2023-01-12T21:41:29.036Z] isn't, please use |./mach busted file test-info| to report it. If |./mach busted| is
[task 2023-01-12T21:41:29.036Z] misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2023-01-12T21:41:29.036Z] 
[task 2023-01-12T21:41:29.036Z] If filing a bug, please include the full output of mach, including this error
[task 2023-01-12T21:41:29.036Z] message.
[task 2023-01-12T21:41:29.036Z] 
[task 2023-01-12T21:41:29.036Z] The details of the failure are as follows:
[task 2023-01-12T21:41:29.036Z] 
[task 2023-01-12T21:41:29.036Z] json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
[task 2023-01-12T21:41:29.036Z] 
[task 2023-01-12T21:41:29.036Z]   File "/builds/worker/checkouts/gecko/testing/mach_commands.py", line 965, in test_info_testrun_report
[task 2023-01-12T21:41:29.036Z]     runcounts = ti.get_runcounts()
[task 2023-01-12T21:41:29.036Z]   File "/builds/worker/checkouts/gecko/testing/testinfo.py", line 423, in get_runcounts
[task 2023-01-12T21:41:29.036Z]     testrundata[str(nextday.date())] = r.json()
[task 2023-01-12T21:41:29.036Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/models.py", line 900, in json
[task 2023-01-12T21:41:29.036Z]     return complexjson.loads(self.text, **kwargs)
[task 2023-01-12T21:41:29.036Z]   File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
[task 2023-01-12T21:41:29.036Z]     return _default_decoder.decode(s)
[task 2023-01-12T21:41:29.036Z]   File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
[task 2023-01-12T21:41:29.036Z]     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
[task 2023-01-12T21:41:29.036Z]   File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
[task 2023-01-12T21:41:29.036Z]     raise JSONDecodeError("Expecting value", s, err.value) from None
[taskcluster 2023-01-12 21:41:29.430Z] === Task Finished ===
[taskcluster 2023-01-12 21:41:29.545Z] Artifact "public" not found at "/builds/worker/artifacts"
[taskcluster 2023-01-12 21:41:29.622Z] Unsuccessful task run with exit code: 1 completed in 234.194 seconds

Comment 1

[Cosmin Sabou [:CosminS]]

Green on the next merge: https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&searchStr=linting%2Copt%2Ctest%2Cmanifest%2Cskip%2Ffail%2Cinformation%2Csource-test-file-metadata-test-info-all%2Call&fromchange=e5ed236608191461d74f0b5a291d57d74657ddab&selectedTaskRun=CEryvxymRb-AuHslcLqPXA.0
Let's see if it pops up again.

Comment 2

[Cosmin Sabou [:CosminS]]

Could you have a look over this mach issue? It also failed on the following merge.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I need to look into this a bit more, it has been on my list all day, still trying to resolve all other pings, etc. I am afraid this might be a Tuesday thing for me.

Comment 4

[Intermittent Failures Robot]

16 failures in 3530 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 15
• toolchains: 1

Platform and build breakdown:

• lint: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1810037&startday=2023-01-09&endday=2023-01-15&tree=all

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

my solution for a fix will be to catch the JSONDecodeError and ensure no data is recorded for that requested date- this will allow us to move forward with what we have collected, and then downstream consumers will have partial data and can determine if they fill in the missing data on their end, or skip action.

The good news is next time this is run, any missing data will be queried again and any lack of data should be short lived and minimized.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1810037 - account for invalid or missing data while running ./mach test-info and gathering testrundata. r=bxbrenden!

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8f1df2381919
account for invalid or missing data while running ./mach test-info and gathering testrundata. r=bxbrenden

Comment 9

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/8f1df2381919

Comment 10

[Intermittent Failures Robot]

3 failures in 3717 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• mozilla-beta: 2

Platform and build breakdown:

• lint: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1810037&startday=2023-01-16&endday=2023-01-22&tree=all

Comment 11

[Intermittent Failures Robot]

1 failures in 3606 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• lint: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1810037&startday=2023-01-30&endday=2023-02-05&tree=all