1986824 - Ensure PERFHERDER_DATA is also uploaded as perfherder-data.json artifacts across all test harnesses

Status: RESOLVED FIXED

(Tree Management :: Perfherder, task)

Description

[Myeongjun Go]

Currently, several test harnesses (for example, Talos, Raptor) write PERFHERDER_DATA lines into the task log.
However, not all of them also create a corresponding perfherder-data.json artifact.

Having consistent JSON artifacts makes it easier and more reliable for downstream tools to consume performance data, instead of relying only on log parsing.
We should update each harness that outputs PERFHERDER_DATA to also generate and upload a perfherder-data.json artifact for consistency.

Comment 1

[Myeongjun Go]

I checked all PERFHERDER_DATA on Searchfox (https://searchfox.org/firefox-main/search?q=PERFHERDER_DATA&path=&case=true&regexp=false) and documented any exceptions that create a perfherder-data.json file in the history.

1. https://searchfox.org/firefox-main/source/build/valgrind/mach_commands.py#197

• This test script is intended for local use only, so generating a perfherder-data.json artifact is unnecessary.

Comment 2

[Myeongjun Go]

Attachment: Bug 1986824 - Ensure PERFHERDER_DATA is also uploaded as perfherder-data.json artifacts across all test harnesses r=sparky

Comment 3

[Greg Mierzwinski [:sparky]]

I wonder if (In reply to Myeongjun Go from comment #1)

I checked all PERFHERDER_DATA on Searchfox (https://searchfox.org/firefox-main/search?q=PERFHERDER_DATA&path=&case=true&regexp=false) and documented any exceptions that create a perfherder-data.json file in the history.

1. https://searchfox.org/firefox-main/source/build/valgrind/mach_commands.py#197

• This test script is intended for local use only, so generating a perfherder-data.json artifact is unnecessary.
• For the production: https://searchfox.org/firefox-main/source/taskcluster/kinds/valgrind/kind.yml

It seems odd to me that there are local-only PERFHERDER_DATA outputs there. :glandium, can you confirm that this valgrind perfherder data is only output locally or is there a CI task that outputs it?

Comment 4

[Mike Hommey [:glandium]]

It's under if "MOZ_AUTOMATION" in os.environ: so it only shows on CI. And yes it's collected: https://treeherder.mozilla.org/perfherder/graphs?series=autoland,5078829,1,2

Comment 5

[Myeongjun Go]

Thanks so much for the patient explanation!
I was misunderstanding it, but I’ve now checked the CI tests and confirmed the logs. Appreciate your time and help :)

Comment 6

[Greg Mierzwinski [:sparky]]

We're considering removing the fetch-content perfherder data. :ahal will be checking with his team to see if there is any need/want to keep it around.

:ahal also mentioned to check with you :glandium. Do you use/want/need that fetch-content data for anything? It's not a problem if you want to keep it around.

Comment 7

[Mike Hommey [:glandium]]

(In reply to Greg Mierzwinski [:sparky] from comment #6)

:ahal also mentioned to check with you :glandium. Do you use/want/need that fetch-content data for anything? It's not a problem if you want to keep it around.

Yes, it's useful to have as a separate metric.